Mobile Developer Productivity at Uber Scale [Droidcon NYC 2022]

Building large mobile apps is hard. Keeping developers productive while working on large mobile apps is harder. Most Android developer tools target smaller apps, so how do we think about building some of the largest apps in the world? In this talk, you’ll learn how Uber enables hundreds of Android developers to contribute on a handful of app in a single monorepo, while keeping developers feeling productive and shipping reliable apps quickly. We’ll cover how the mobile teams are structured, how developer productivity is measured and monitored, custom tools like Bazel & Buck, our CI/CD diff pipeline, our remote dev server environment, our custom IDE plugins and tools, how we modularize and use common app architecture and frameworks to keep consistency in all our apps, how we prioritize open-source engagement, and some of our best practices to continually evolve and modernize our apps at scale along to keep up with the fast moving pace of Android. Whether you're planning for your own app's scale, or just curious how it's done in one of the largest apps, you won't want to miss this talk.

Transcript

1. Mobile Developer Productivity at Uber Scale Ty Smith Android Platform Tech Lead 1 @tsmith
2. Agenda 01 Uber’s Scale 02 How Uber Organizes 03 Development Workflow 03 Measuring Devx 04 Making Builds Faster 05 IDE
3. Uber’s Scale 3
4. 2008 2012 2016 MVC 100k LOC RIBs 1m LOC Startup 10k LOC 2021 RIBs++ 10m LOC
5. Development at Uber Scale Thousands Microservices Tens of Thousands commits per month Millions Config changes per month Trillions Mobile analytics events per month Hundreds of Millions Monorepo LOC Hundreds of Thousands Deployments per month 4K+ Engineers 7 Programming languages Tens Mobile apps Billions User sessions per month 5 Platform Monorepo’s
6. Mobile Scale 600+ Mobile Engineers Tens of Thousands Build Modules Tens of Millions Mobile LOC Thousands RIBs - Our Mobile Architecture And 100’s of internal apps
7. Development Workflow 7
8. Developers move quickly using standardized tools and frameworks Standardization Each platform lives in it’s own Monorepo. 5 Total. Monorepos Monorepo teams centrally support all engineers Centralized Teams Development Stack
9. Apps (Rider, Driver, Eats, Freight, etc) Observability & XP (Services, Tools, Libraries) Networking (Protocols , Libraries, Edge) Architecture Programming Languages Platforms Open Source Libraries Build & Tools IDEs Release & Compiler Extensions Device Lab Automated Testing Docs CI/CD Mobile Tech Stack
10. Android Tech Stack Architecture • RIBS - Converged Architecture • Kotlin, Compose, Coroutines • Motif - Code generated DI Education • Centralized documentation, Codelabs, developer onboarding, and support UI • Stylist/Artist - View & Style generators • Base Mobile - Compose Based UI Components • Base SDUI - Server Driven UI Rendering Frameworks • Citrus - Parameterized XP and Plugins • Simple-store - Async KVS Storage Build • Buck/Bazel - Hermetic Build system* • Kaptish - Fast Kotlin Annotation Processing* • Buildkite - Sharded containerized CI/mergequeue IDE and Tools • Intellij Plugins* • Flipper + Custom Plugins* • Devpods - Cloud IDEs* • QuickUI - UI Hotloading* • LDA - Local Developer Analytics* • Mobile Studio - Plugin based debug drawer Reliability • Detekt/Errorprone/Lint/Shellcheck - Static Analysis • Ktfmt/GJF - Code Formatter • Piranha - automated XP code cleanup • Nanoscope - Hyper performant profiler Data and Observability • Unified Reporter - Analytics pipeline • Jenga - GRPC Model generation pipeline • Healthline - Crash & Observability • Blackswan - Crash Recovery
11. Organization 11
12. Engineering Teams at Uber
13. Mobile Org ELT Platform Eng Mobility Delivery Developer Platform Rider Driver Eater Eater Features *Org structure simplified Driver Features Rider Features Rider Platform Driver Platform Eater Platform Mobile Platform
14. Mobile Platform Mobile Networking Android Foundations Mobile Data iOS Devx Mobile Platform Teams Devtools Mobile Observability Testing iOS Foundations Bug Reporting Android Devx Mobile UI
15. Mobile Platform Vision Be the industry leader for how developers build, deploy, and manage high-quality software productively and at scale.
16. Fast, modern tools Powerful frameworks Safety nets Standardization VELOCITY QUALITY EFFICIENCY Developer sentiment (NPS) Developer metrics Strategy
17. Measuring Devx 17
18. Build, test or merge failure Diff create to commit time (20-60 hr) Land time (10-45 min) Local build time (30-120 sec) Diff review time (10-40 hr) Delivery time (1 hr) Local Development Submit Queue Commit Code Review Diff create Diff land Delivery uCD ❌ CI Build ❌ ❌ Build or test failure Changes requested Diff create to AppStore (p50: 15 days) or Diff create to production Diff CI time Wait for build cut Mobile Test Submission to App/Play Stores App Store Rollouts Mobile build cut ❌ Mobile Hotfix Mobile Release (9-10 days) ❌Burnt release A SWE’s inner loop Proguard or Redex Failure ❌ Mobile Code Development
19. What We Measure • Net Promoter Score (NPS) • Developer Throughput • Build Time • Failure Rate • Git time • IDE Performance • CI/CD Time • Tooling uptime • App Performance • App Reliability
20. Developer Satisfaction (NPS)
21. Local Environment Local Developer Analytics (LDA) Data Collectors Data Collector Data Collector Data Collector Analytics Daemon Kafka Kibana/search Hadoop/Hive/Presto Log Collection m3/monitoring
22. Local Developer Analytics Data Collectors Git Arc Buck Bazel IDE Gradle Custom CLIs and tools
24. Apdex
25. Making Builds Faster 25
26. Uber’s Android Build Systems 2021 Evaluate Bazel Worked with Bazel and Gradle teams to evaluate both and decide to converge with Uber on Bazel. 2022 Bazel Migration • Dynamically generate Bazel BUILD files and run in parallel with BUCK. • Coordinate with Community • Invest into performance and compatibility 2016 Introduced Buck with OkBuck • Use Gradle to generate BUCK files • < 1K Modules • 3min to 30s p75 build time 2019 Pure Buck Stopped generating Buck files dynamically from Gradle 2023 Deprecate Buck • Remove Buck as Shadow • >10K Modules • Remote Execution
27. Bazel Benefits • Hermetic/reproducible Builds • Highly Parallelizable • Aligned with Industry • Dedicated build language (Starlark) • No side effects • Cacheable by default • Remote Build support • Easily Extensible • Converged with other platforms
28. Bazel Downsides • Lacking out of the box experience • Requires dedicated engineers • Slower for new Android features
29. Mobile vs Backend Builds
30. Rider App Dependency Graph *Reduced via transitive reduction Over 12,000 module dependencies
33. FooModule BarModule dependsOn FooBarModule dependsOn BazModule dependsOn 1 2 3 4
34. Kotlin Builds 🐢 Kotlin is up to 2x slower than Java (with Errorprone)
35. Faster Compilation • Ins and Outs • Cache Keys • ABI Jars
36. Faster Compilation Build Target Inputs • Toolchain • Dependencies • Configuration FooModule Build Target Outputs • JARs/AARs Cache Key: 9e92c406b5efc492a714d29c5e26516d
37. ABI Jars FooModule BarModule dependsOn ?
38. ABI Jars compile_against_abis = false Bar Module src/foo.kt fun bar() = “Foo” Bar.jar Foo.class Cache-key: 9e92c406b5efc492a714d29c5e26516d public final class FooKt { @NotNull public static final String bar() { return "Foo"; } } FooModule BarModule dependsOn
39. ABI Jars compile_against_abis = false Bar Module src/foo.kt fun bar() = “FooBar” Bar.jar Foo.class Cache-key: a28d01554c0f541c04ec3a2fe28396ed public final class FooKt { @NotNull public static final String bar() { return "FooBar"; } } FooModule BarModule dependsOn
40. ABI Jars compile_against_abis = true Bar Module src/foo.kt fun bar() = “Foo” Bar.abi.jar Foo.class Cache-key: d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } FooModule BarModule dependsOn
41. ABI Jars compile_against_abis = true Bar Module src/foo.kt fun bar() = “FooBar” Bar.abi.jar Foo.class Cache-key: d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } FooModule BarModule dependsOn
42. Faster Compilation • Ins and Outs • Cache Keys • ABI Jars ◦ Class ABI jars ◦ Src ABI Jars
43. Bar Module src/foo.kt fun bar() = “Foo” Bar.abi.jar Foo.class Cache-key: d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } Class ABI Jars Bar.jar Foo.class Cache-key: a28d01554c0f541c04ec3a2fe28396ed public final class FooKt { @NotNull public static final String bar() { return "FooBar"; } } ABI Generator (ASM) javac/kotlinc
44. Bar Module src/foo.kt fun bar() = “Foo” Bar.abi.jar Foo.class Cache-key: d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } Source ABI Jars Bar.jar Foo.class Cache-key: a28d01554c0f541c04ec3a2fe28396ed public final class FooKt { @NotNull public static final String bar() { return "FooBar"; } } javac/kotlinc
45. Javac Bar Module src/foo.kt fun bar() = “Foo” Bar.abi.jar Foo.class Cache-key: d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } Source ABI Jars (Java) Bar.jar Foo.class Cache-key: a28d01554c0f541c04ec3a2fe28396ed public final class FooKt { @NotNull public static final String bar() { return "FooBar"; } } parse+enter analyze+generate
46. Rule Pipelining (Java) Bar.abi.jar BarModule Javac parse+enter analyze+generate FooModule BarModule dependsOn 1 FooModule Javac parse+enter analyze+generate 2
47. Kotlinc Bar Module src/foo.kt fun bar() = “Foo” Bar.abi.jar Foo.class Cache-key: d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } Source ABI Jars (Kotlin) Bar.jar Foo.class Cache-key: a28d01554c0f541c04ec3a2fe28396ed public final class FooKt { @NotNull public static final String bar() { return "FooBar"; } } Jvm-abi-gen
48. Mixed Java and Kotlin Sources Kotlinc Jvm-abi-gen Javac parse+enter analyze+generate kapt ksp FooBarModule foo.kt bar.java foo.class bar.class Compress foobar.jar 1 2 3
49. Kotlin Rule Pipelining Making Mixed Java/Kt Sources Faster Kotlinc Jvm-abi-gen Javac parse+enter analyze+generate kapt ksp FooBarModule foo.kt bar.java foo.class bar.class Compress foobar.abi.jar 1 2 4 Compress foobar.jar foo.abi.class bar.abi.class 3 5
50. Kotlin Rule Pipelining Making Mixed Java/Kt Sources Faster FooModule BarModule dependsOn BarModule Javac analyze+generate parse+enter Jar Kotlinc ABI Jar FooModule Jvm-Abi-Gen Javac analyze+generate parse+enter Jar Kotlinc ABI Jar Jvm-Abi-Gen
51. Faster Compilation • Ins and Outs • Cache Keys • ABI Jars ◦ Class ABI jars ◦ Src ABI Jars • Per class compiler avoidance
52. Per Class Compiler Avoidance FooModule src/foo.kt fun foo() { bar() } BarModule src/bar.kt fun bar() dependsOn FooBarModule src/foobar.kt fun foobar()
53. Per Class Compiler Avoidance FooBarModule src/foobar.kt fun foobar() FooModule src/foo.kt fun foo() { bar() } BarModule src/bar.kt fun bar() dependsOn Class in jar is used? Class in jar is used?
54. Kotlin Class Usage Tracking Compiler Plugin ClassUsageCompilerPlugin : AnalysisHandlerExtension, StorageComponentContainerExtension fun analysisCompleted() { collectClasses(...) } ClassCollector : CallChecker, DeclarationChecker fun check(ResolvedCall) { registerClass(...) } fun check(KtDeclaration) { registerClass(...) } Used Classes Storage - JDeps - used-classes.json
55. Faster Compilation • Ins and Outs • Cache Keys • ABI Jars ◦ Class ABI jars ◦ Src ABI Jars • Per class compiler avoidance • Annotation Processing
56. Annotation Processing • Kotlin Annotation Processing Tool (KAPT) ◦ Slow • Java Annotation Processing (AP) ◦ Fast ◦ Doesn’t support Kotlin • Kotlin Symbol Processing (KSP) ◦ Fast ◦ Doesn’t have wide support
57. Kotlin Annotation Processing FooBarModule Foo.kt @Foo Foo.class Foo_Impl.class KAPT slows build time by more than 150% compared to Kotlin without AP or Javac AP Kotlinc KAPT Stubs Annotation Processing Compilation Foo_Impl.kt
58. Kotlin Annotation Processing FooBarModule Foo.kt @Foo Foo.class Foo_Impl.class Javac Bar.java @Bar Bar.class Bar_Impl.class Kotlinc KAPT Stubs Annotation Processing Compilation Foo_Impl.kt
59. Kotlin Annotation Processing FooBarModule Foo.kt @Foo Foo.class @Foo Javac Bar.java @Bar Bar.class Bar_Impl.class Kotlinc Kaptish Foo_Impl.class
60. Kaptish class Kaptish : Plugin { override fun getName() = "Kaptish" override fun init(task: JavacTask, vararg args: String?) { val classes = getClasses(task.context) Arguments.instance(task.context).classNames.addAll(classes) } “The javac command can also process annotations in Java source files and classes.” - Javac docs
61. Kaptish Downsides • Cannot reference generated code directly in Kotlin. • Must have class retained annotations • Cannot generate Kotlin code
62. Faster Compilation • Ins and Outs • Cache Keys • ABI Jars ◦ Class ABI jars ◦ Src ABI Jars • Per class compiler avoidance • Annotation Processing • Future: ◦ KSP ◦ K2 Compiler ◦ Graph Flattening
63. IDEs and Devtools 63
64. Local development • M1 Max hardware running Mac OS • Intellij 2022 • Uber IDE Plugins • Automatic IDE settings (Vmoptions, default settings, code style) • Integrate LDA • Managed IT and standard applications via Chef • Developer setup Android environment ❌
65. Custom Intellij Plugins • Local Developer Analytics • Scaffold new Module • Scaffold new RIB with Compose Views • Scaffold new Sandbox app • Motif Dependency explorer • Realtime RIB explorer • Index new apps/modules • Manage third party plugins • Analytics line markers • Live Templates • Realtime UI updates
66. QuickUI Realtime XML Updates Phone Uber App Custom Resource Wrapper Resource Only APK Resource V1 Resource V2 Resource V2 Build Command Resource Only Change? Compile Full APK and ADB push Compile Resources and ADB push No Yes
67. QuickUI Demo
68. Devpod (Cloud IDEs) • Remote Linux Dev Servers (400gb Ram, 96 core) • Automatically Provisioned for Uber’s Android development • All packages are fully managed • Batteries Included! Prebuilt IDE indexes, build snapshots, 3rd party deps, and more. • IDE always running and available to connect • Running Jetbrains Gateway, Projector, and VSCode • Automatic machine suspension • Allows multiple machines • Instant context Switching “I used to barely being able to make large scale migrations on my macbook w/o steam coming out of it, and now im changing 20K LOC in 2K files w/ all modules in Intellij indexed on a cloud machine over airplane wifi. Crazy times!”
69. compute node 0 devpod nvme devpod nvme devpod nvme devpod nvme • Devpod is a docker container • Running inside kubernetes • With nvme disk • Sharing powerful compute nodes with other devpods compute node 1 Devpod
70. Devpod Regions
71. • Base flavor contains all common software between other flavors • Each monorepo build their own tailored experience on top of base • Addon workflows, user dotfiles, custom packages/IDE plugins, and more customizations + Flavors
72. Devpod IDE ADB Build Laptop ADB SSH SSHD Gateway Emulator / Phone gateway://… devpod init-adb mydevpod (ssh -r 5037:localhost:5037) Devpods + Android Emulators & Devices
73. Snapshots 24 Hours Boot Devpod Build Project Clone Repo Archive outputs • ~/android/buck-out • ~/.idea/* folders • ~/.gradle/* folders • ~/.buck-cache • ~/.bazel-cache Upload Archives Boot IDE > devpod create Provision Devpod Boot Devpod Clone Repo Download Artifacts Extract Archives ~/android/buck-out ~/.idea/* folders ~/.gradle/* folders ~/.buck-cache ~/.bazel-cache Boot IDE Storage
74. Indexing Project target: a project in Android monorepo * provided time corresponds to Intellij IDEA “Updating Index” stage Regular Indexing 38 minutes Jetbrains Shared Index Plugin 32 minutes Cached Index seconds Jetbrains Cached Index
75. Devpod Demo
76. Resources • RIBs - github.com/uber/ribs • Motif - github.com/uber/motif • Nanoscope - github.com/uber/nanoscope • Nullaway - github.com/uber/nullaway • Piranha - github.com/uber/piranha • Gradle/Buck/Bazel evaluation - github.com/uber-common/android-build-eval • Napt - github.com/sergei-lapin/napt • Flipper - github.com/facebook/flipper • Detekt - github.com/detekt/detekt • KTFMT - github.com/facebookincubator/ktfmt • Devpod like product - coder.com, Gitpod.io, & Github Code Spaces
77. Mobile Developer Productivity at Uber Scale Ty Smith Android Platform Tech Lead 77 @tsmith