Mobile Developer Productivity at Uber Scale [Droidcon NYC 2022]

Building large mobile apps is hard. Keeping developers productive while working on large mobile apps is harder. Most Android developer tools target smaller apps, so how do we think about building some of the largest apps in the world? In this talk, you’ll learn how Uber enables hundreds of Android developers to contribute on a handful of app in a single monorepo, while keeping developers feeling productive and shipping reliable apps quickly. We’ll cover how the mobile teams are structured, how developer productivity is measured and monitored, custom tools like Bazel & Buck, our CI/CD diff pipeline, our remote dev server environment, our custom IDE plugins and tools, how we modularize and use common app architecture and frameworks to keep consistency in all our apps, how we prioritize open-source engagement, and some of our best practices to continually evolve and modernize our apps at scale along to keep up with the fast moving pace of Android. Whether you're planning for your own app's scale, or just curious how it's done in one of the largest apps, you won't want to miss this talk.

Transcript

1. Mobile Developer
   Productivity at Uber Scale
   Ty Smith
   Android Platform Tech Lead
   1
   @tsmith
   

   View Slide

2. Agenda
   01 Uber’s Scale
   02 How Uber Organizes
   03 Development Workflow
   03 Measuring Devx
   04 Making Builds Faster
   05 IDE
   

   View Slide

3. Uber’s Scale
   3
   

   View Slide

4. 2008 2012 2016
   MVC
   100k LOC
   RIBs
   1m LOC
   Startup
   10k LOC
   2021
   RIBs++
   10m LOC
   

   View Slide

5. Development at Uber Scale
   Thousands
   Microservices
   Tens of
   Thousands
   commits per
   month
   Millions
   Config changes
   per month
   Trillions
   Mobile analytics
   events per month
   Hundreds of
   Millions
   Monorepo LOC
   Hundreds of
   Thousands
   Deployments per
   month
   4K+
   Engineers
   7
   Programming
   languages
   Tens
   Mobile apps
   Billions
   User sessions per
   month
   5
   Platform
   Monorepo’s
   

   View Slide

6. Mobile Scale
   600+
   Mobile Engineers
   Tens of
   Thousands
   Build Modules
   Tens of
   Millions
   Mobile LOC
   Thousands
   RIBs - Our Mobile
   Architecture
   And 100’s of internal
   apps
   

   View Slide

7. Development Workflow
   7
   

   View Slide

8. Developers move
   quickly using
   standardized tools
   and frameworks
   Standardization
   Each platform lives in
   it’s own Monorepo. 5
   Total.
   Monorepos
   Monorepo teams
   centrally support all
   engineers
   Centralized Teams
   Development Stack
   

   View Slide

9. Apps (Rider, Driver, Eats, Freight, etc)
   Observability & XP
   (Services, Tools, Libraries)
   Networking
   (Protocols , Libraries, Edge)
   Architecture
   Programming
   Languages
   Platforms
   Open
   Source
   Libraries
   Build & Tools
   IDEs
   Release &
   Compiler
   Extensions
   Device
   Lab
   Automated
   Testing
   Docs CI/CD
   Mobile Tech Stack
   

   View Slide

10. Android Tech Stack
    Architecture
    ● RIBS - Converged Architecture
    ● Kotlin, Compose, Coroutines
    ● Motif - Code generated DI
    Education
    ● Centralized documentation, Codelabs, developer
    onboarding, and support
    UI
    ● Stylist/Artist - View & Style generators
    ● Base Mobile - Compose Based UI Components
    ● Base SDUI - Server Driven UI Rendering
    Frameworks
    ● Citrus - Parameterized XP and Plugins
    ● Simple-store - Async KVS Storage
    Build
    ● Buck/Bazel - Hermetic Build system*
    ● Kaptish - Fast Kotlin Annotation Processing*
    ● Buildkite - Sharded containerized CI/mergequeue
    IDE and Tools
    ● Intellij Plugins*
    ● Flipper + Custom Plugins*
    ● Devpods - Cloud IDEs*
    ● QuickUI - UI Hotloading*
    ● LDA - Local Developer Analytics*
    ● Mobile Studio - Plugin based debug drawer
    Reliability
    ● Detekt/Errorprone/Lint/Shellcheck - Static Analysis
    ● Ktfmt/GJF - Code Formatter
    ● Piranha - automated XP code cleanup
    ● Nanoscope - Hyper performant profiler
    Data and Observability
    ● Unified Reporter - Analytics pipeline
    ● Jenga - GRPC Model generation pipeline
    ● Healthline - Crash & Observability
    ● Blackswan - Crash Recovery
    

    View Slide

11. Organization
    11
    

    View Slide

12. Engineering Teams at Uber
    

    View Slide

13. Mobile Org
    ELT
    Platform Eng
    Mobility
    Delivery
    Developer
    Platform
    Rider
    Driver
    Eater
    Eater
    Features
    *Org structure simplified
    Driver
    Features
    Rider
    Features
    Rider
    Platform
    Driver
    Platform
    Eater
    Platform
    Mobile
    Platform
    

    View Slide

14. Mobile Platform
    Mobile
    Networking
    Android
    Foundations
    Mobile Data
    iOS Devx
    Mobile Platform Teams
    Devtools
    Mobile
    Observability
    Testing
    iOS
    Foundations
    Bug
    Reporting
    Android
    Devx
    Mobile UI
    

    View Slide

15. Mobile Platform Vision
    Be the industry leader
    for how developers build,
    deploy, and manage high-quality
    software productively
    and at scale.
    

    View Slide

16. Fast, modern tools
    Powerful frameworks
    Safety nets
    Standardization
    VELOCITY
    QUALITY
    EFFICIENCY
    Developer
    sentiment (NPS)
    Developer metrics
    Strategy
    

    View Slide

17. Measuring Devx
    17
    

    View Slide

18. Build, test or
    merge failure
    Diff create to commit time (20-60 hr)
    Land time
    (10-45 min)
    Local build time
    (30-120 sec)
    Diff review time (10-40 hr)
    Delivery time (1
    hr)
    Local Development Submit Queue
    Commit
    Code Review
    Diff create Diff land Delivery
    uCD
    ❌
    CI Build
    ❌ ❌
    Build or
    test failure
    Changes
    requested
    Diff create to AppStore (p50: 15 days) or Diff create to production
    Diff CI time
    Wait for
    build cut
    Mobile Test
    Submission to
    App/Play Stores
    App Store
    Rollouts
    Mobile
    build cut
    ❌
    Mobile
    Hotfix
    Mobile Release (9-10 days)
    ❌Burnt
    release
    A SWE’s
    inner loop
    Proguard or
    Redex Failure
    ❌
    Mobile Code Development
    

    View Slide

19. What We Measure
    ● Net Promoter Score (NPS)
    ● Developer Throughput
    ● Build Time
    ● Failure Rate
    ● Git time
    ● IDE Performance
    ● CI/CD Time
    ● Tooling uptime
    ● App Performance
    ● App Reliability
    

    View Slide

20. Developer Satisfaction (NPS)
    

    View Slide

21. Local Environment
    Local Developer Analytics (LDA)
    Data Collectors
    Data Collector Data Collector Data Collector
    Analytics Daemon
    Kafka
    Kibana/search
    Hadoop/Hive/Presto
    Log
    Collection
    m3/monitoring
    

    View Slide

22. Local Developer Analytics
    Data Collectors
    Git Arc Buck Bazel IDE
    Gradle
    Custom
    CLIs and
    tools
    

    View Slide

23. View Slide

24. Apdex
    

    View Slide

25. Making Builds Faster
    25
    

    View Slide

26. Uber’s Android Build Systems
    2021
    Evaluate Bazel
    Worked with Bazel and Gradle teams
    to evaluate both and decide to
    converge with Uber on Bazel.
    2022
    Bazel Migration
    ● Dynamically generate Bazel
    BUILD files and run in
    parallel with BUCK.
    ● Coordinate with Community
    ● Invest into performance
    and compatibility
    2016
    Introduced Buck with OkBuck
    ● Use Gradle to generate
    BUCK files
    ● < 1K Modules
    ● 3min to 30s p75 build time
    2019
    Pure Buck
    Stopped generating Buck files
    dynamically from Gradle
    2023
    Deprecate Buck
    ● Remove Buck as
    Shadow
    ● >10K Modules
    ● Remote Execution
    

    View Slide

27. Bazel Benefits
    ● Hermetic/reproducible Builds
    ● Highly Parallelizable
    ● Aligned with Industry
    ● Dedicated build language (Starlark)
    ● No side effects
    ● Cacheable by default
    ● Remote Build support
    ● Easily Extensible
    ● Converged with other platforms
    

    View Slide

28. Bazel Downsides
    ● Lacking out of the box experience
    ● Requires dedicated engineers
    ● Slower for new Android features
    

    View Slide

29. Mobile vs Backend Builds
    

    View Slide

30. Rider App Dependency Graph
    *Reduced via transitive reduction
    Over 12,000 module dependencies
    

    View Slide

31. View Slide

32. View Slide

33. FooModule BarModule
    dependsOn
    FooBarModule
    dependsOn
    BazModule
    dependsOn
    1
    2
    3
    4
    

    View Slide

34. Kotlin Builds 🐢
    Kotlin is up to 2x slower than Java (with Errorprone)
    

    View Slide

35. Faster Compilation
    ● Ins and Outs
    ● Cache Keys
    ● ABI Jars
    

    View Slide

36. Faster Compilation
    Build Target Inputs
    ● Toolchain
    ● Dependencies
    ● Configuration
    FooModule
    Build Target Outputs
    ● JARs/AARs
    Cache Key: 9e92c406b5efc492a714d29c5e26516d
    

    View Slide

37. ABI Jars
    FooModule BarModule
    dependsOn
    ?
    

    View Slide

38. ABI Jars
    compile_against_abis = false
    Bar Module
    src/foo.kt
    fun bar() = “Foo”
    Bar.jar
    Foo.class
    Cache-key: 9e92c406b5efc492a714d29c5e26516d
    public final class FooKt {
    @NotNull
    public static final String bar() {
    return "Foo";
    }
    }
    FooModule BarModule
    dependsOn
    

    View Slide

39. ABI Jars
    compile_against_abis = false
    Bar Module
    src/foo.kt
    fun bar() = “FooBar”
    Bar.jar
    Foo.class
    Cache-key: a28d01554c0f541c04ec3a2fe28396ed
    public final class FooKt {
    @NotNull
    public static final String bar() {
    return "FooBar";
    }
    }
    FooModule BarModule
    dependsOn
    

    View Slide

40. ABI Jars
    compile_against_abis = true
    Bar Module
    src/foo.kt
    fun bar() = “Foo”
    Bar.abi.jar
    Foo.class
    Cache-key: d3b7138441694bea985e32ef219697b6
    public final class FooKt {
    @NotNull
    public static final String bar() { }
    }
    FooModule BarModule
    dependsOn
    

    View Slide

41. ABI Jars
    compile_against_abis = true
    Bar Module
    src/foo.kt
    fun bar() = “FooBar”
    Bar.abi.jar
    Foo.class
    Cache-key: d3b7138441694bea985e32ef219697b6
    public final class FooKt {
    @NotNull
    public static final String bar() { }
    }
    FooModule BarModule
    dependsOn
    

    View Slide

42. Faster Compilation
    ● Ins and Outs
    ● Cache Keys
    ● ABI Jars
    ○ Class ABI jars
    ○ Src ABI Jars
    

    View Slide

43. Bar Module
    src/foo.kt
    fun bar() = “Foo”
    Bar.abi.jar
    Foo.class
    Cache-key:
    d3b7138441694bea985e32ef219697b6
    public final class FooKt {
    @NotNull
    public static final String bar() { }
    }
    Class ABI Jars
    Bar.jar
    Foo.class
    Cache-key: a28d01554c0f541c04ec3a2fe28396ed
    public final class FooKt {
    @NotNull
    public static final String bar() {
    return "FooBar";
    }
    }
    ABI Generator (ASM)
    javac/kotlinc
    

    View Slide

44. Bar Module
    src/foo.kt
    fun bar() = “Foo”
    Bar.abi.jar
    Foo.class
    Cache-key:
    d3b7138441694bea985e32ef219697b6
    public final class FooKt {
    @NotNull
    public static final String bar() { }
    }
    Source ABI Jars
    Bar.jar
    Foo.class
    Cache-key: a28d01554c0f541c04ec3a2fe28396ed
    public final class FooKt {
    @NotNull
    public static final String bar() {
    return "FooBar";
    }
    }
    javac/kotlinc
    

    View Slide

45. Javac
    Bar Module
    src/foo.kt
    fun bar() = “Foo”
    Bar.abi.jar
    Foo.class
    Cache-key:
    d3b7138441694bea985e32ef219697b6
    public final class FooKt {
    @NotNull
    public static final String bar() { }
    }
    Source ABI Jars (Java)
    Bar.jar
    Foo.class
    Cache-key: a28d01554c0f541c04ec3a2fe28396ed
    public final class FooKt {
    @NotNull
    public static final String bar() {
    return "FooBar";
    }
    }
    parse+enter analyze+generate
    

    View Slide

46. Rule Pipelining (Java)
    Bar.abi.jar
    BarModule
    Javac parse+enter analyze+generate
    FooModule BarModule
    dependsOn
    1
    FooModule
    Javac parse+enter analyze+generate
    2
    

    View Slide

47. Kotlinc
    Bar Module
    src/foo.kt
    fun bar() = “Foo”
    Bar.abi.jar
    Foo.class
    Cache-key:
    d3b7138441694bea985e32ef219697b6
    public final class FooKt {
    @NotNull
    public static final String bar() { }
    }
    Source ABI Jars (Kotlin)
    Bar.jar
    Foo.class
    Cache-key: a28d01554c0f541c04ec3a2fe28396ed
    public final class FooKt {
    @NotNull
    public static final String bar() {
    return "FooBar";
    }
    }
    Jvm-abi-gen
    

    View Slide

48. Mixed Java and Kotlin Sources
    Kotlinc Jvm-abi-gen
    Javac parse+enter analyze+generate
    kapt ksp
    FooBarModule
    foo.kt
    bar.java
    foo.class
    bar.class
    Compress foobar.jar
    1
    2
    3
    

    View Slide

49. Kotlin Rule Pipelining
    Making Mixed Java/Kt Sources Faster
    Kotlinc Jvm-abi-gen
    Javac parse+enter analyze+generate
    kapt ksp
    FooBarModule
    foo.kt
    bar.java
    foo.class
    bar.class
    Compress foobar.abi.jar
    1
    2
    4
    Compress foobar.jar
    foo.abi.class
    bar.abi.class
    3
    5
    

    View Slide

50. Kotlin Rule Pipelining
    Making Mixed Java/Kt Sources Faster
    FooModule BarModule
    dependsOn
    BarModule
    Javac
    analyze+generate
    parse+enter
    Jar
    Kotlinc
    ABI Jar
    FooModule
    Jvm-Abi-Gen
    Javac
    analyze+generate
    parse+enter
    Jar
    Kotlinc
    ABI Jar
    Jvm-Abi-Gen
    

    View Slide

51. Faster Compilation
    ● Ins and Outs
    ● Cache Keys
    ● ABI Jars
    ○ Class ABI jars
    ○ Src ABI Jars
    ● Per class compiler avoidance
    

    View Slide

52. Per Class Compiler Avoidance
    FooModule
    src/foo.kt
    fun foo() {
    bar()
    }
    BarModule
    src/bar.kt
    fun bar()
    dependsOn
    FooBarModule
    src/foobar.kt
    fun foobar()
    

    View Slide

53. Per Class Compiler Avoidance
    FooBarModule
    src/foobar.kt
    fun foobar()
    FooModule
    src/foo.kt
    fun foo() {
    bar()
    }
    BarModule
    src/bar.kt
    fun bar()
    dependsOn
    Class
    in jar is
    used?
    Class
    in jar is
    used?
    

    View Slide

54. Kotlin Class Usage Tracking Compiler Plugin
    ClassUsageCompilerPlugin
    : AnalysisHandlerExtension,
    StorageComponentContainerExtension
    fun analysisCompleted() {
    collectClasses(...)
    }
    ClassCollector : CallChecker, DeclarationChecker
    fun check(ResolvedCall) {
    registerClass(...)
    }
    fun check(KtDeclaration) {
    registerClass(...)
    }
    Used Classes Storage
    - JDeps
    - used-classes.json
    

    View Slide

55. Faster Compilation
    ● Ins and Outs
    ● Cache Keys
    ● ABI Jars
    ○ Class ABI jars
    ○ Src ABI Jars
    ● Per class compiler avoidance
    ● Annotation Processing
    

    View Slide

56. Annotation Processing
    ● Kotlin Annotation Processing Tool (KAPT)
    ○ Slow
    ● Java Annotation Processing (AP)
    ○ Fast
    ○ Doesn’t support Kotlin
    ● Kotlin Symbol Processing (KSP)
    ○ Fast
    ○ Doesn’t have wide support
    

    View Slide

57. Kotlin Annotation Processing
    FooBarModule
    Foo.kt
    @Foo
    Foo.class
    Foo_Impl.class
    KAPT slows build time by more than 150% compared to Kotlin without AP or Javac AP
    Kotlinc
    KAPT
    Stubs
    Annotation
    Processing
    Compilation
    Foo_Impl.kt
    

    View Slide

58. Kotlin Annotation Processing
    FooBarModule
    Foo.kt
    @Foo
    Foo.class
    Foo_Impl.class
    Javac
    Bar.java
    @Bar
    Bar.class
    Bar_Impl.class
    Kotlinc
    KAPT
    Stubs
    Annotation
    Processing
    Compilation
    Foo_Impl.kt
    

    View Slide

59. Kotlin Annotation Processing
    FooBarModule
    Foo.kt
    @Foo
    Foo.class
    @Foo
    Javac
    Bar.java
    @Bar
    Bar.class
    Bar_Impl.class
    Kotlinc
    Kaptish
    Foo_Impl.class
    

    View Slide

60. Kaptish
    class Kaptish : Plugin {
    override fun getName() = "Kaptish"
    override fun init(task: JavacTask, vararg args: String?) {
    val classes = getClasses(task.context)
    Arguments.instance(task.context).classNames.addAll(classes)
    }
    “The javac command can also process annotations in Java source files and classes.” - Javac docs
    

    View Slide

61. Kaptish Downsides
    ● Cannot reference generated code directly in Kotlin.
    ● Must have class retained annotations
    ● Cannot generate Kotlin code
    

    View Slide

62. Faster Compilation
    ● Ins and Outs
    ● Cache Keys
    ● ABI Jars
    ○ Class ABI jars
    ○ Src ABI Jars
    ● Per class compiler avoidance
    ● Annotation Processing
    ● Future:
    ○ KSP
    ○ K2 Compiler
    ○ Graph Flattening
    

    View Slide

63. IDEs and Devtools
    63
    

    View Slide

64. Local development
    ● M1 Max hardware running Mac OS
    ● Intellij 2022
    ● Uber IDE Plugins
    ● Automatic IDE settings (Vmoptions, default settings, code style)
    ● Integrate LDA
    ● Managed IT and standard applications via Chef
    ● Developer setup Android environment ❌
    

    View Slide

65. Custom Intellij Plugins
    ● Local Developer Analytics
    ● Scaffold new Module
    ● Scaffold new RIB with Compose Views
    ● Scaffold new Sandbox app
    ● Motif Dependency explorer
    ● Realtime RIB explorer
    ● Index new apps/modules
    ● Manage third party plugins
    ● Analytics line markers
    ● Live Templates
    ● Realtime UI updates
    

    View Slide

66. QuickUI
    Realtime XML Updates
    Phone
    Uber App
    Custom Resource
    Wrapper
    Resource Only APK
    Resource V1
    Resource V2
    Resource V2
    Build Command
    Resource
    Only
    Change?
    Compile
    Full APK
    and ADB
    push
    Compile
    Resources
    and ADB
    push
    No
    Yes
    

    View Slide

67. QuickUI Demo
    

    View Slide

68. Devpod (Cloud IDEs)
    ● Remote Linux Dev Servers (400gb Ram, 96 core)
    ● Automatically Provisioned for Uber’s Android development
    ● All packages are fully managed
    ● Batteries Included! Prebuilt IDE indexes, build snapshots, 3rd party deps, and more.
    ● IDE always running and available to connect
    ● Running Jetbrains Gateway, Projector, and VSCode
    ● Automatic machine suspension
    ● Allows multiple machines
    ● Instant context Switching
    “I used to barely being able to make large scale migrations on my macbook w/o steam coming out of it, and now im
    changing 20K LOC in 2K files w/ all modules in Intellij indexed on a cloud machine over airplane wifi. Crazy times!”
    

    View Slide

69. compute node 0
    devpod
    nvme
    devpod
    nvme
    devpod
    nvme
    devpod
    nvme
    ● Devpod is a docker container
    ● Running inside kubernetes
    ● With nvme disk
    ● Sharing powerful compute nodes with other
    devpods
    compute node 1
    Devpod
    

    View Slide

70. Devpod Regions
    

    View Slide

71. ● Base flavor contains all common software between other flavors
    ● Each monorepo build their own tailored experience on top of base
    ● Addon workflows, user dotfiles, custom packages/IDE plugins, and more customizations
    +
    Flavors
    

    View Slide

72. Devpod
    IDE
    ADB
    Build
    Laptop
    ADB
    SSH
    SSHD
    Gateway
    Emulator /
    Phone
    gateway://…
    devpod init-adb mydevpod
    (ssh -r 5037:localhost:5037)
    Devpods + Android Emulators & Devices
    

    View Slide

73. Snapshots
    24
    Hours
    Boot Devpod
    Build Project
    Clone Repo
    Archive outputs
    ● ~/android/buck-out
    ● ~/.idea/* folders
    ● ~/.gradle/* folders
    ● ~/.buck-cache
    ● ~/.bazel-cache
    Upload Archives
    Boot IDE
    > devpod create
    Provision Devpod
    Boot Devpod
    Clone Repo
    Download Artifacts
    Extract Archives
    ~/android/buck-out
    ~/.idea/* folders
    ~/.gradle/* folders
    ~/.buck-cache
    ~/.bazel-cache
    Boot IDE
    Storage
    

    View Slide

74. Indexing Project target: a project in Android monorepo
    * provided time corresponds to Intellij IDEA “Updating Index” stage
    Regular Indexing
    38 minutes
    Jetbrains Shared Index Plugin
    32 minutes
    Cached Index
    seconds
    Jetbrains Cached Index
    

    View Slide

75. Devpod Demo
    

    View Slide

76. Resources
    ● RIBs - github.com/uber/ribs
    ● Motif - github.com/uber/motif
    ● Nanoscope - github.com/uber/nanoscope
    ● Nullaway - github.com/uber/nullaway
    ● Piranha - github.com/uber/piranha
    ● Gradle/Buck/Bazel evaluation - github.com/uber-common/android-build-eval
    ● Napt - github.com/sergei-lapin/napt
    ● Flipper - github.com/facebook/flipper
    ● Detekt - github.com/detekt/detekt
    ● KTFMT - github.com/facebookincubator/ktfmt
    ● Devpod like product - coder.com, Gitpod.io, & Github Code Spaces
    

    View Slide

77. Mobile Developer
    Productivity at Uber Scale
    Ty Smith
    Android Platform Tech Lead
    77
    @tsmith
    

    View Slide