$30 off During Our Annual Pro Sale. View Details »

Mobile Developer Productivity at Uber Scale [Droidcon NYC 2022]

Ty Smith
September 02, 2022

Mobile Developer Productivity at Uber Scale [Droidcon NYC 2022]

Building large mobile apps is hard. Keeping developers productive while working on large mobile apps is harder. Most Android developer tools target smaller apps, so how do we think about building some of the largest apps in the world? In this talk, you’ll learn how Uber enables hundreds of Android developers to contribute on a handful of app in a single monorepo, while keeping developers feeling productive and shipping reliable apps quickly. We’ll cover how the mobile teams are structured, how developer productivity is measured and monitored, custom tools like Bazel & Buck, our CI/CD diff pipeline, our remote dev server environment, our custom IDE plugins and tools, how we modularize and use common app architecture and frameworks to keep consistency in all our apps, how we prioritize open-source engagement, and some of our best practices to continually evolve and modernize our apps at scale along to keep up with the fast moving pace of Android. Whether you're planning for your own app's scale, or just curious how it's done in one of the largest apps, you won't want to miss this talk.

Ty Smith

September 02, 2022
Tweet

More Decks by Ty Smith

Other Decks in Programming

Transcript

  1. Mobile Developer
    Productivity at Uber Scale
    Ty Smith
    Android Platform Tech Lead
    1
    @tsmith

    View Slide

  2. Agenda
    01 Uber’s Scale
    02 How Uber Organizes
    03 Development Workflow
    03 Measuring Devx
    04 Making Builds Faster
    05 IDE

    View Slide

  3. Uber’s Scale
    3

    View Slide

  4. 2008 2012 2016
    MVC
    100k LOC
    RIBs
    1m LOC
    Startup
    10k LOC
    2021
    RIBs++
    10m LOC

    View Slide

  5. Development at Uber Scale
    Thousands
    Microservices
    Tens of
    Thousands
    commits per
    month
    Millions
    Config changes
    per month
    Trillions
    Mobile analytics
    events per month
    Hundreds of
    Millions
    Monorepo LOC
    Hundreds of
    Thousands
    Deployments per
    month
    4K+
    Engineers
    7
    Programming
    languages
    Tens
    Mobile apps
    Billions
    User sessions per
    month
    5
    Platform
    Monorepo’s

    View Slide

  6. Mobile Scale
    600+
    Mobile Engineers
    Tens of
    Thousands
    Build Modules
    Tens of
    Millions
    Mobile LOC
    Thousands
    RIBs - Our Mobile
    Architecture
    And 100’s of internal
    apps

    View Slide

  7. Development Workflow
    7

    View Slide

  8. Developers move
    quickly using
    standardized tools
    and frameworks
    Standardization
    Each platform lives in
    it’s own Monorepo. 5
    Total.
    Monorepos
    Monorepo teams
    centrally support all
    engineers
    Centralized Teams
    Development Stack

    View Slide

  9. Apps (Rider, Driver, Eats, Freight, etc)
    Observability & XP
    (Services, Tools, Libraries)
    Networking
    (Protocols , Libraries, Edge)
    Architecture
    Programming
    Languages
    Platforms
    Open
    Source
    Libraries
    Build & Tools
    IDEs
    Release &
    Compiler
    Extensions
    Device
    Lab
    Automated
    Testing
    Docs CI/CD
    Mobile Tech Stack

    View Slide

  10. Android Tech Stack
    Architecture
    ● RIBS - Converged Architecture
    ● Kotlin, Compose, Coroutines
    ● Motif - Code generated DI
    Education
    ● Centralized documentation, Codelabs, developer
    onboarding, and support
    UI
    ● Stylist/Artist - View & Style generators
    ● Base Mobile - Compose Based UI Components
    ● Base SDUI - Server Driven UI Rendering
    Frameworks
    ● Citrus - Parameterized XP and Plugins
    ● Simple-store - Async KVS Storage
    Build
    ● Buck/Bazel - Hermetic Build system*
    ● Kaptish - Fast Kotlin Annotation Processing*
    ● Buildkite - Sharded containerized CI/mergequeue
    IDE and Tools
    ● Intellij Plugins*
    ● Flipper + Custom Plugins*
    ● Devpods - Cloud IDEs*
    ● QuickUI - UI Hotloading*
    ● LDA - Local Developer Analytics*
    ● Mobile Studio - Plugin based debug drawer
    Reliability
    ● Detekt/Errorprone/Lint/Shellcheck - Static Analysis
    ● Ktfmt/GJF - Code Formatter
    ● Piranha - automated XP code cleanup
    ● Nanoscope - Hyper performant profiler
    Data and Observability
    ● Unified Reporter - Analytics pipeline
    ● Jenga - GRPC Model generation pipeline
    ● Healthline - Crash & Observability
    ● Blackswan - Crash Recovery

    View Slide

  11. Organization
    11

    View Slide

  12. Engineering Teams at Uber

    View Slide

  13. Mobile Org
    ELT
    Platform Eng
    Mobility
    Delivery
    Developer
    Platform
    Rider
    Driver
    Eater
    Eater
    Features
    *Org structure simplified
    Driver
    Features
    Rider
    Features
    Rider
    Platform
    Driver
    Platform
    Eater
    Platform
    Mobile
    Platform

    View Slide

  14. Mobile Platform
    Mobile
    Networking
    Android
    Foundations
    Mobile Data
    iOS Devx
    Mobile Platform Teams
    Devtools
    Mobile
    Observability
    Testing
    iOS
    Foundations
    Bug
    Reporting
    Android
    Devx
    Mobile UI

    View Slide

  15. Mobile Platform Vision
    Be the industry leader
    for how developers build,
    deploy, and manage high-quality
    software productively
    and at scale.

    View Slide

  16. Fast, modern tools
    Powerful frameworks
    Safety nets
    Standardization
    VELOCITY
    QUALITY
    EFFICIENCY
    Developer
    sentiment (NPS)
    Developer metrics
    Strategy

    View Slide

  17. Measuring Devx
    17

    View Slide

  18. Build, test or
    merge failure
    Diff create to commit time (20-60 hr)
    Land time
    (10-45 min)
    Local build time
    (30-120 sec)
    Diff review time (10-40 hr)
    Delivery time (1
    hr)
    Local Development Submit Queue
    Commit
    Code Review
    Diff create Diff land Delivery
    uCD

    CI Build
    ❌ ❌
    Build or
    test failure
    Changes
    requested
    Diff create to AppStore (p50: 15 days) or Diff create to production
    Diff CI time
    Wait for
    build cut
    Mobile Test
    Submission to
    App/Play Stores
    App Store
    Rollouts
    Mobile
    build cut

    Mobile
    Hotfix
    Mobile Release (9-10 days)
    ❌Burnt
    release
    A SWE’s
    inner loop
    Proguard or
    Redex Failure

    Mobile Code Development

    View Slide

  19. What We Measure
    ● Net Promoter Score (NPS)
    ● Developer Throughput
    ● Build Time
    ● Failure Rate
    ● Git time
    ● IDE Performance
    ● CI/CD Time
    ● Tooling uptime
    ● App Performance
    ● App Reliability

    View Slide

  20. Developer Satisfaction (NPS)

    View Slide

  21. Local Environment
    Local Developer Analytics (LDA)
    Data Collectors
    Data Collector Data Collector Data Collector
    Analytics Daemon
    Kafka
    Kibana/search
    Hadoop/Hive/Presto
    Log
    Collection
    m3/monitoring

    View Slide

  22. Local Developer Analytics
    Data Collectors
    Git Arc Buck Bazel IDE
    Gradle
    Custom
    CLIs and
    tools

    View Slide

  23. View Slide

  24. Apdex

    View Slide

  25. Making Builds Faster
    25

    View Slide

  26. Uber’s Android Build Systems
    2021
    Evaluate Bazel
    Worked with Bazel and Gradle teams
    to evaluate both and decide to
    converge with Uber on Bazel.
    2022
    Bazel Migration
    ● Dynamically generate Bazel
    BUILD files and run in
    parallel with BUCK.
    ● Coordinate with Community
    ● Invest into performance
    and compatibility
    2016
    Introduced Buck with OkBuck
    ● Use Gradle to generate
    BUCK files
    ● < 1K Modules
    ● 3min to 30s p75 build time
    2019
    Pure Buck
    Stopped generating Buck files
    dynamically from Gradle
    2023
    Deprecate Buck
    ● Remove Buck as
    Shadow
    ● >10K Modules
    ● Remote Execution

    View Slide

  27. Bazel Benefits
    ● Hermetic/reproducible Builds
    ● Highly Parallelizable
    ● Aligned with Industry
    ● Dedicated build language (Starlark)
    ● No side effects
    ● Cacheable by default
    ● Remote Build support
    ● Easily Extensible
    ● Converged with other platforms

    View Slide

  28. Bazel Downsides
    ● Lacking out of the box experience
    ● Requires dedicated engineers
    ● Slower for new Android features

    View Slide

  29. Mobile vs Backend Builds

    View Slide

  30. Rider App Dependency Graph
    *Reduced via transitive reduction
    Over 12,000 module dependencies

    View Slide

  31. View Slide

  32. View Slide

  33. FooModule BarModule
    dependsOn
    FooBarModule
    dependsOn
    BazModule
    dependsOn
    1
    2
    3
    4

    View Slide

  34. Kotlin Builds 🐢
    Kotlin is up to 2x slower than Java (with Errorprone)

    View Slide

  35. Faster Compilation
    ● Ins and Outs
    ● Cache Keys
    ● ABI Jars

    View Slide

  36. Faster Compilation
    Build Target Inputs
    ● Toolchain
    ● Dependencies
    ● Configuration
    FooModule
    Build Target Outputs
    ● JARs/AARs
    Cache Key: 9e92c406b5efc492a714d29c5e26516d

    View Slide

  37. ABI Jars
    FooModule BarModule
    dependsOn
    ?

    View Slide

  38. ABI Jars
    compile_against_abis = false
    Bar Module
    src/foo.kt
    fun bar() = “Foo”
    Bar.jar
    Foo.class
    Cache-key: 9e92c406b5efc492a714d29c5e26516d
    public final class FooKt {
    @NotNull
    public static final String bar() {
    return "Foo";
    }
    }
    FooModule BarModule
    dependsOn

    View Slide

  39. ABI Jars
    compile_against_abis = false
    Bar Module
    src/foo.kt
    fun bar() = “FooBar”
    Bar.jar
    Foo.class
    Cache-key: a28d01554c0f541c04ec3a2fe28396ed
    public final class FooKt {
    @NotNull
    public static final String bar() {
    return "FooBar";
    }
    }
    FooModule BarModule
    dependsOn

    View Slide

  40. ABI Jars
    compile_against_abis = true
    Bar Module
    src/foo.kt
    fun bar() = “Foo”
    Bar.abi.jar
    Foo.class
    Cache-key: d3b7138441694bea985e32ef219697b6
    public final class FooKt {
    @NotNull
    public static final String bar() { }
    }
    FooModule BarModule
    dependsOn

    View Slide

  41. ABI Jars
    compile_against_abis = true
    Bar Module
    src/foo.kt
    fun bar() = “FooBar”
    Bar.abi.jar
    Foo.class
    Cache-key: d3b7138441694bea985e32ef219697b6
    public final class FooKt {
    @NotNull
    public static final String bar() { }
    }
    FooModule BarModule
    dependsOn

    View Slide

  42. Faster Compilation
    ● Ins and Outs
    ● Cache Keys
    ● ABI Jars
    ○ Class ABI jars
    ○ Src ABI Jars

    View Slide

  43. Bar Module
    src/foo.kt
    fun bar() = “Foo”
    Bar.abi.jar
    Foo.class
    Cache-key:
    d3b7138441694bea985e32ef219697b6
    public final class FooKt {
    @NotNull
    public static final String bar() { }
    }
    Class ABI Jars
    Bar.jar
    Foo.class
    Cache-key: a28d01554c0f541c04ec3a2fe28396ed
    public final class FooKt {
    @NotNull
    public static final String bar() {
    return "FooBar";
    }
    }
    ABI Generator (ASM)
    javac/kotlinc

    View Slide

  44. Bar Module
    src/foo.kt
    fun bar() = “Foo”
    Bar.abi.jar
    Foo.class
    Cache-key:
    d3b7138441694bea985e32ef219697b6
    public final class FooKt {
    @NotNull
    public static final String bar() { }
    }
    Source ABI Jars
    Bar.jar
    Foo.class
    Cache-key: a28d01554c0f541c04ec3a2fe28396ed
    public final class FooKt {
    @NotNull
    public static final String bar() {
    return "FooBar";
    }
    }
    javac/kotlinc

    View Slide

  45. Javac
    Bar Module
    src/foo.kt
    fun bar() = “Foo”
    Bar.abi.jar
    Foo.class
    Cache-key:
    d3b7138441694bea985e32ef219697b6
    public final class FooKt {
    @NotNull
    public static final String bar() { }
    }
    Source ABI Jars (Java)
    Bar.jar
    Foo.class
    Cache-key: a28d01554c0f541c04ec3a2fe28396ed
    public final class FooKt {
    @NotNull
    public static final String bar() {
    return "FooBar";
    }
    }
    parse+enter analyze+generate

    View Slide

  46. Rule Pipelining (Java)
    Bar.abi.jar
    BarModule
    Javac parse+enter analyze+generate
    FooModule BarModule
    dependsOn
    1
    FooModule
    Javac parse+enter analyze+generate
    2

    View Slide

  47. Kotlinc
    Bar Module
    src/foo.kt
    fun bar() = “Foo”
    Bar.abi.jar
    Foo.class
    Cache-key:
    d3b7138441694bea985e32ef219697b6
    public final class FooKt {
    @NotNull
    public static final String bar() { }
    }
    Source ABI Jars (Kotlin)
    Bar.jar
    Foo.class
    Cache-key: a28d01554c0f541c04ec3a2fe28396ed
    public final class FooKt {
    @NotNull
    public static final String bar() {
    return "FooBar";
    }
    }
    Jvm-abi-gen

    View Slide

  48. Mixed Java and Kotlin Sources
    Kotlinc Jvm-abi-gen
    Javac parse+enter analyze+generate
    kapt ksp
    FooBarModule
    foo.kt
    bar.java
    foo.class
    bar.class
    Compress foobar.jar
    1
    2
    3

    View Slide

  49. Kotlin Rule Pipelining
    Making Mixed Java/Kt Sources Faster
    Kotlinc Jvm-abi-gen
    Javac parse+enter analyze+generate
    kapt ksp
    FooBarModule
    foo.kt
    bar.java
    foo.class
    bar.class
    Compress foobar.abi.jar
    1
    2
    4
    Compress foobar.jar
    foo.abi.class
    bar.abi.class
    3
    5

    View Slide

  50. Kotlin Rule Pipelining
    Making Mixed Java/Kt Sources Faster
    FooModule BarModule
    dependsOn
    BarModule
    Javac
    analyze+generate
    parse+enter
    Jar
    Kotlinc
    ABI Jar
    FooModule
    Jvm-Abi-Gen
    Javac
    analyze+generate
    parse+enter
    Jar
    Kotlinc
    ABI Jar
    Jvm-Abi-Gen

    View Slide

  51. Faster Compilation
    ● Ins and Outs
    ● Cache Keys
    ● ABI Jars
    ○ Class ABI jars
    ○ Src ABI Jars
    ● Per class compiler avoidance

    View Slide

  52. Per Class Compiler Avoidance
    FooModule
    src/foo.kt
    fun foo() {
    bar()
    }
    BarModule
    src/bar.kt
    fun bar()
    dependsOn
    FooBarModule
    src/foobar.kt
    fun foobar()

    View Slide

  53. Per Class Compiler Avoidance
    FooBarModule
    src/foobar.kt
    fun foobar()
    FooModule
    src/foo.kt
    fun foo() {
    bar()
    }
    BarModule
    src/bar.kt
    fun bar()
    dependsOn
    Class
    in jar is
    used?
    Class
    in jar is
    used?

    View Slide

  54. Kotlin Class Usage Tracking Compiler Plugin
    ClassUsageCompilerPlugin
    : AnalysisHandlerExtension,
    StorageComponentContainerExtension
    fun analysisCompleted() {
    collectClasses(...)
    }
    ClassCollector : CallChecker, DeclarationChecker
    fun check(ResolvedCall) {
    registerClass(...)
    }
    fun check(KtDeclaration) {
    registerClass(...)
    }
    Used Classes Storage
    - JDeps
    - used-classes.json

    View Slide

  55. Faster Compilation
    ● Ins and Outs
    ● Cache Keys
    ● ABI Jars
    ○ Class ABI jars
    ○ Src ABI Jars
    ● Per class compiler avoidance
    ● Annotation Processing

    View Slide

  56. Annotation Processing
    ● Kotlin Annotation Processing Tool (KAPT)
    ○ Slow
    ● Java Annotation Processing (AP)
    ○ Fast
    ○ Doesn’t support Kotlin
    ● Kotlin Symbol Processing (KSP)
    ○ Fast
    ○ Doesn’t have wide support

    View Slide

  57. Kotlin Annotation Processing
    FooBarModule
    Foo.kt
    @Foo
    Foo.class
    Foo_Impl.class
    KAPT slows build time by more than 150% compared to Kotlin without AP or Javac AP
    Kotlinc
    KAPT
    Stubs
    Annotation
    Processing
    Compilation
    Foo_Impl.kt

    View Slide

  58. Kotlin Annotation Processing
    FooBarModule
    Foo.kt
    @Foo
    Foo.class
    Foo_Impl.class
    Javac
    Bar.java
    @Bar
    Bar.class
    Bar_Impl.class
    Kotlinc
    KAPT
    Stubs
    Annotation
    Processing
    Compilation
    Foo_Impl.kt

    View Slide

  59. Kotlin Annotation Processing
    FooBarModule
    Foo.kt
    @Foo
    Foo.class
    @Foo
    Javac
    Bar.java
    @Bar
    Bar.class
    Bar_Impl.class
    Kotlinc
    Kaptish
    Foo_Impl.class

    View Slide

  60. Kaptish
    class Kaptish : Plugin {
    override fun getName() = "Kaptish"
    override fun init(task: JavacTask, vararg args: String?) {
    val classes = getClasses(task.context)
    Arguments.instance(task.context).classNames.addAll(classes)
    }
    “The javac command can also process annotations in Java source files and classes.” - Javac docs

    View Slide

  61. Kaptish Downsides
    ● Cannot reference generated code directly in Kotlin.
    ● Must have class retained annotations
    ● Cannot generate Kotlin code

    View Slide

  62. Faster Compilation
    ● Ins and Outs
    ● Cache Keys
    ● ABI Jars
    ○ Class ABI jars
    ○ Src ABI Jars
    ● Per class compiler avoidance
    ● Annotation Processing
    ● Future:
    ○ KSP
    ○ K2 Compiler
    ○ Graph Flattening

    View Slide

  63. IDEs and Devtools
    63

    View Slide

  64. Local development
    ● M1 Max hardware running Mac OS
    ● Intellij 2022
    ● Uber IDE Plugins
    ● Automatic IDE settings (Vmoptions, default settings, code style)
    ● Integrate LDA
    ● Managed IT and standard applications via Chef
    ● Developer setup Android environment ❌

    View Slide

  65. Custom Intellij Plugins
    ● Local Developer Analytics
    ● Scaffold new Module
    ● Scaffold new RIB with Compose Views
    ● Scaffold new Sandbox app
    ● Motif Dependency explorer
    ● Realtime RIB explorer
    ● Index new apps/modules
    ● Manage third party plugins
    ● Analytics line markers
    ● Live Templates
    ● Realtime UI updates

    View Slide

  66. QuickUI
    Realtime XML Updates
    Phone
    Uber App
    Custom Resource
    Wrapper
    Resource Only APK
    Resource V1
    Resource V2
    Resource V2
    Build Command
    Resource
    Only
    Change?
    Compile
    Full APK
    and ADB
    push
    Compile
    Resources
    and ADB
    push
    No
    Yes

    View Slide

  67. QuickUI Demo

    View Slide

  68. Devpod (Cloud IDEs)
    ● Remote Linux Dev Servers (400gb Ram, 96 core)
    ● Automatically Provisioned for Uber’s Android development
    ● All packages are fully managed
    ● Batteries Included! Prebuilt IDE indexes, build snapshots, 3rd party deps, and more.
    ● IDE always running and available to connect
    ● Running Jetbrains Gateway, Projector, and VSCode
    ● Automatic machine suspension
    ● Allows multiple machines
    ● Instant context Switching
    “I used to barely being able to make large scale migrations on my macbook w/o steam coming out of it, and now im
    changing 20K LOC in 2K files w/ all modules in Intellij indexed on a cloud machine over airplane wifi. Crazy times!”

    View Slide

  69. compute node 0
    devpod
    nvme
    devpod
    nvme
    devpod
    nvme
    devpod
    nvme
    ● Devpod is a docker container
    ● Running inside kubernetes
    ● With nvme disk
    ● Sharing powerful compute nodes with other
    devpods
    compute node 1
    Devpod

    View Slide

  70. Devpod Regions

    View Slide

  71. ● Base flavor contains all common software between other flavors
    ● Each monorepo build their own tailored experience on top of base
    ● Addon workflows, user dotfiles, custom packages/IDE plugins, and more customizations
    +
    Flavors

    View Slide

  72. Devpod
    IDE
    ADB
    Build
    Laptop
    ADB
    SSH
    SSHD
    Gateway
    Emulator /
    Phone
    gateway://…
    devpod init-adb mydevpod
    (ssh -r 5037:localhost:5037)
    Devpods + Android Emulators & Devices

    View Slide

  73. Snapshots
    24
    Hours
    Boot Devpod
    Build Project
    Clone Repo
    Archive outputs
    ● ~/android/buck-out
    ● ~/.idea/* folders
    ● ~/.gradle/* folders
    ● ~/.buck-cache
    ● ~/.bazel-cache
    Upload Archives
    Boot IDE
    > devpod create
    Provision Devpod
    Boot Devpod
    Clone Repo
    Download Artifacts
    Extract Archives
    ~/android/buck-out
    ~/.idea/* folders
    ~/.gradle/* folders
    ~/.buck-cache
    ~/.bazel-cache
    Boot IDE
    Storage

    View Slide

  74. Indexing Project target: a project in Android monorepo
    * provided time corresponds to Intellij IDEA “Updating Index” stage
    Regular Indexing
    38 minutes
    Jetbrains Shared Index Plugin
    32 minutes
    Cached Index
    seconds
    Jetbrains Cached Index

    View Slide

  75. Devpod Demo

    View Slide

  76. Resources
    ● RIBs - github.com/uber/ribs
    ● Motif - github.com/uber/motif
    ● Nanoscope - github.com/uber/nanoscope
    ● Nullaway - github.com/uber/nullaway
    ● Piranha - github.com/uber/piranha
    ● Gradle/Buck/Bazel evaluation - github.com/uber-common/android-build-eval
    ● Napt - github.com/sergei-lapin/napt
    ● Flipper - github.com/facebook/flipper
    ● Detekt - github.com/detekt/detekt
    ● KTFMT - github.com/facebookincubator/ktfmt
    ● Devpod like product - coder.com, Gitpod.io, & Github Code Spaces

    View Slide

  77. Mobile Developer
    Productivity at Uber Scale
    Ty Smith
    Android Platform Tech Lead
    77
    @tsmith

    View Slide