Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mobile Developer Productivity at Uber Scale [Droidcon NYC 2022]

Ty Smith
September 02, 2022

Mobile Developer Productivity at Uber Scale [Droidcon NYC 2022]

Building large mobile apps is hard. Keeping developers productive while working on large mobile apps is harder. Most Android developer tools target smaller apps, so how do we think about building some of the largest apps in the world? In this talk, you’ll learn how Uber enables hundreds of Android developers to contribute on a handful of app in a single monorepo, while keeping developers feeling productive and shipping reliable apps quickly. We’ll cover how the mobile teams are structured, how developer productivity is measured and monitored, custom tools like Bazel & Buck, our CI/CD diff pipeline, our remote dev server environment, our custom IDE plugins and tools, how we modularize and use common app architecture and frameworks to keep consistency in all our apps, how we prioritize open-source engagement, and some of our best practices to continually evolve and modernize our apps at scale along to keep up with the fast moving pace of Android. Whether you're planning for your own app's scale, or just curious how it's done in one of the largest apps, you won't want to miss this talk.

Ty Smith

September 02, 2022
Tweet

More Decks by Ty Smith

Other Decks in Programming

Transcript

  1. Mobile Developer Productivity at Uber Scale Ty Smith Android Platform

    Tech Lead 1 @tsmith
  2. Agenda 01 Uber’s Scale 02 How Uber Organizes 03 Development

    Workflow 03 Measuring Devx 04 Making Builds Faster 05 IDE
  3. Uber’s Scale 3

  4. 2008 2012 2016 MVC 100k LOC RIBs 1m LOC Startup

    10k LOC 2021 RIBs++ 10m LOC
  5. Development at Uber Scale Thousands Microservices Tens of Thousands commits

    per month Millions Config changes per month Trillions Mobile analytics events per month Hundreds of Millions Monorepo LOC Hundreds of Thousands Deployments per month 4K+ Engineers 7 Programming languages Tens Mobile apps Billions User sessions per month 5 Platform Monorepo’s
  6. Mobile Scale 600+ Mobile Engineers Tens of Thousands Build Modules

    Tens of Millions Mobile LOC Thousands RIBs - Our Mobile Architecture And 100’s of internal apps
  7. Development Workflow 7

  8. Developers move quickly using standardized tools and frameworks Standardization Each

    platform lives in it’s own Monorepo. 5 Total. Monorepos Monorepo teams centrally support all engineers Centralized Teams Development Stack
  9. Apps (Rider, Driver, Eats, Freight, etc) Observability & XP (Services,

    Tools, Libraries) Networking (Protocols , Libraries, Edge) Architecture Programming Languages Platforms Open Source Libraries Build & Tools IDEs Release & Compiler Extensions Device Lab Automated Testing Docs CI/CD Mobile Tech Stack
  10. Android Tech Stack Architecture • RIBS - Converged Architecture •

    Kotlin, Compose, Coroutines • Motif - Code generated DI Education • Centralized documentation, Codelabs, developer onboarding, and support UI • Stylist/Artist - View & Style generators • Base Mobile - Compose Based UI Components • Base SDUI - Server Driven UI Rendering Frameworks • Citrus - Parameterized XP and Plugins • Simple-store - Async KVS Storage Build • Buck/Bazel - Hermetic Build system* • Kaptish - Fast Kotlin Annotation Processing* • Buildkite - Sharded containerized CI/mergequeue IDE and Tools • Intellij Plugins* • Flipper + Custom Plugins* • Devpods - Cloud IDEs* • QuickUI - UI Hotloading* • LDA - Local Developer Analytics* • Mobile Studio - Plugin based debug drawer Reliability • Detekt/Errorprone/Lint/Shellcheck - Static Analysis • Ktfmt/GJF - Code Formatter • Piranha - automated XP code cleanup • Nanoscope - Hyper performant profiler Data and Observability • Unified Reporter - Analytics pipeline • Jenga - GRPC Model generation pipeline • Healthline - Crash & Observability • Blackswan - Crash Recovery
  11. Organization 11

  12. Engineering Teams at Uber

  13. Mobile Org ELT Platform Eng Mobility Delivery Developer Platform Rider

    Driver Eater Eater Features *Org structure simplified Driver Features Rider Features Rider Platform Driver Platform Eater Platform Mobile Platform
  14. Mobile Platform Mobile Networking Android Foundations Mobile Data iOS Devx

    Mobile Platform Teams Devtools Mobile Observability Testing iOS Foundations Bug Reporting Android Devx Mobile UI
  15. Mobile Platform Vision Be the industry leader for how developers

    build, deploy, and manage high-quality software productively and at scale.
  16. Fast, modern tools Powerful frameworks Safety nets Standardization VELOCITY QUALITY

    EFFICIENCY Developer sentiment (NPS) Developer metrics Strategy
  17. Measuring Devx 17

  18. Build, test or merge failure Diff create to commit time

    (20-60 hr) Land time (10-45 min) Local build time (30-120 sec) Diff review time (10-40 hr) Delivery time (1 hr) Local Development Submit Queue Commit Code Review Diff create Diff land Delivery uCD ❌ CI Build ❌ ❌ Build or test failure Changes requested Diff create to AppStore (p50: 15 days) or Diff create to production Diff CI time Wait for build cut Mobile Test Submission to App/Play Stores App Store Rollouts Mobile build cut ❌ Mobile Hotfix Mobile Release (9-10 days) ❌Burnt release A SWE’s inner loop Proguard or Redex Failure ❌ Mobile Code Development
  19. What We Measure • Net Promoter Score (NPS) • Developer

    Throughput • Build Time • Failure Rate • Git time • IDE Performance • CI/CD Time • Tooling uptime • App Performance • App Reliability
  20. Developer Satisfaction (NPS)

  21. Local Environment Local Developer Analytics (LDA) Data Collectors Data Collector

    Data Collector Data Collector Analytics Daemon Kafka Kibana/search Hadoop/Hive/Presto Log Collection m3/monitoring
  22. Local Developer Analytics Data Collectors Git Arc Buck Bazel IDE

    Gradle Custom CLIs and tools
  23. None
  24. Apdex

  25. Making Builds Faster 25

  26. Uber’s Android Build Systems 2021 Evaluate Bazel Worked with Bazel

    and Gradle teams to evaluate both and decide to converge with Uber on Bazel. 2022 Bazel Migration • Dynamically generate Bazel BUILD files and run in parallel with BUCK. • Coordinate with Community • Invest into performance and compatibility 2016 Introduced Buck with OkBuck • Use Gradle to generate BUCK files • < 1K Modules • 3min to 30s p75 build time 2019 Pure Buck Stopped generating Buck files dynamically from Gradle 2023 Deprecate Buck • Remove Buck as Shadow • >10K Modules • Remote Execution
  27. Bazel Benefits • Hermetic/reproducible Builds • Highly Parallelizable • Aligned

    with Industry • Dedicated build language (Starlark) • No side effects • Cacheable by default • Remote Build support • Easily Extensible • Converged with other platforms
  28. Bazel Downsides • Lacking out of the box experience •

    Requires dedicated engineers • Slower for new Android features
  29. Mobile vs Backend Builds

  30. Rider App Dependency Graph *Reduced via transitive reduction Over 12,000

    module dependencies
  31. None
  32. None
  33. FooModule BarModule dependsOn FooBarModule dependsOn BazModule dependsOn 1 2 3

    4
  34. Kotlin Builds 🐢 Kotlin is up to 2x slower than

    Java (with Errorprone)
  35. Faster Compilation • Ins and Outs • Cache Keys •

    ABI Jars
  36. Faster Compilation Build Target Inputs • Toolchain • Dependencies •

    Configuration FooModule Build Target Outputs • JARs/AARs Cache Key: 9e92c406b5efc492a714d29c5e26516d
  37. ABI Jars FooModule BarModule dependsOn ?

  38. ABI Jars compile_against_abis = false Bar Module src/foo.kt fun bar()

    = “Foo” Bar.jar Foo.class Cache-key: 9e92c406b5efc492a714d29c5e26516d public final class FooKt { @NotNull public static final String bar() { return "Foo"; } } FooModule BarModule dependsOn
  39. ABI Jars compile_against_abis = false Bar Module src/foo.kt fun bar()

    = “FooBar” Bar.jar Foo.class Cache-key: a28d01554c0f541c04ec3a2fe28396ed public final class FooKt { @NotNull public static final String bar() { return "FooBar"; } } FooModule BarModule dependsOn
  40. ABI Jars compile_against_abis = true Bar Module src/foo.kt fun bar()

    = “Foo” Bar.abi.jar Foo.class Cache-key: d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } FooModule BarModule dependsOn
  41. ABI Jars compile_against_abis = true Bar Module src/foo.kt fun bar()

    = “FooBar” Bar.abi.jar Foo.class Cache-key: d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } FooModule BarModule dependsOn
  42. Faster Compilation • Ins and Outs • Cache Keys •

    ABI Jars ◦ Class ABI jars ◦ Src ABI Jars
  43. Bar Module src/foo.kt fun bar() = “Foo” Bar.abi.jar Foo.class Cache-key:

    d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } Class ABI Jars Bar.jar Foo.class Cache-key: a28d01554c0f541c04ec3a2fe28396ed public final class FooKt { @NotNull public static final String bar() { return "FooBar"; } } ABI Generator (ASM) javac/kotlinc
  44. Bar Module src/foo.kt fun bar() = “Foo” Bar.abi.jar Foo.class Cache-key:

    d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } Source ABI Jars Bar.jar Foo.class Cache-key: a28d01554c0f541c04ec3a2fe28396ed public final class FooKt { @NotNull public static final String bar() { return "FooBar"; } } javac/kotlinc
  45. Javac Bar Module src/foo.kt fun bar() = “Foo” Bar.abi.jar Foo.class

    Cache-key: d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } Source ABI Jars (Java) Bar.jar Foo.class Cache-key: a28d01554c0f541c04ec3a2fe28396ed public final class FooKt { @NotNull public static final String bar() { return "FooBar"; } } parse+enter analyze+generate
  46. Rule Pipelining (Java) Bar.abi.jar BarModule Javac parse+enter analyze+generate FooModule BarModule

    dependsOn 1 FooModule Javac parse+enter analyze+generate 2
  47. Kotlinc Bar Module src/foo.kt fun bar() = “Foo” Bar.abi.jar Foo.class

    Cache-key: d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } Source ABI Jars (Kotlin) Bar.jar Foo.class Cache-key: a28d01554c0f541c04ec3a2fe28396ed public final class FooKt { @NotNull public static final String bar() { return "FooBar"; } } Jvm-abi-gen
  48. Mixed Java and Kotlin Sources Kotlinc Jvm-abi-gen Javac parse+enter analyze+generate

    kapt ksp FooBarModule foo.kt bar.java foo.class bar.class Compress foobar.jar 1 2 3
  49. Kotlin Rule Pipelining Making Mixed Java/Kt Sources Faster Kotlinc Jvm-abi-gen

    Javac parse+enter analyze+generate kapt ksp FooBarModule foo.kt bar.java foo.class bar.class Compress foobar.abi.jar 1 2 4 Compress foobar.jar foo.abi.class bar.abi.class 3 5
  50. Kotlin Rule Pipelining Making Mixed Java/Kt Sources Faster FooModule BarModule

    dependsOn BarModule Javac analyze+generate parse+enter Jar Kotlinc ABI Jar FooModule Jvm-Abi-Gen Javac analyze+generate parse+enter Jar Kotlinc ABI Jar Jvm-Abi-Gen
  51. Faster Compilation • Ins and Outs • Cache Keys •

    ABI Jars ◦ Class ABI jars ◦ Src ABI Jars • Per class compiler avoidance
  52. Per Class Compiler Avoidance FooModule src/foo.kt fun foo() { bar()

    } BarModule src/bar.kt fun bar() dependsOn FooBarModule src/foobar.kt fun foobar()
  53. Per Class Compiler Avoidance FooBarModule src/foobar.kt fun foobar() FooModule src/foo.kt

    fun foo() { bar() } BarModule src/bar.kt fun bar() dependsOn Class in jar is used? Class in jar is used?
  54. Kotlin Class Usage Tracking Compiler Plugin ClassUsageCompilerPlugin : AnalysisHandlerExtension, StorageComponentContainerExtension

    fun analysisCompleted() { collectClasses(...) } ClassCollector : CallChecker, DeclarationChecker fun check(ResolvedCall) { registerClass(...) } fun check(KtDeclaration) { registerClass(...) } Used Classes Storage - JDeps - used-classes.json
  55. Faster Compilation • Ins and Outs • Cache Keys •

    ABI Jars ◦ Class ABI jars ◦ Src ABI Jars • Per class compiler avoidance • Annotation Processing
  56. Annotation Processing • Kotlin Annotation Processing Tool (KAPT) ◦ Slow

    • Java Annotation Processing (AP) ◦ Fast ◦ Doesn’t support Kotlin • Kotlin Symbol Processing (KSP) ◦ Fast ◦ Doesn’t have wide support
  57. Kotlin Annotation Processing FooBarModule Foo.kt @Foo Foo.class Foo_Impl.class KAPT slows

    build time by more than 150% compared to Kotlin without AP or Javac AP Kotlinc KAPT Stubs Annotation Processing Compilation Foo_Impl.kt
  58. Kotlin Annotation Processing FooBarModule Foo.kt @Foo Foo.class Foo_Impl.class Javac Bar.java

    @Bar Bar.class Bar_Impl.class Kotlinc KAPT Stubs Annotation Processing Compilation Foo_Impl.kt
  59. Kotlin Annotation Processing FooBarModule Foo.kt @Foo Foo.class @Foo Javac Bar.java

    @Bar Bar.class Bar_Impl.class Kotlinc Kaptish Foo_Impl.class
  60. Kaptish class Kaptish : Plugin { override fun getName() =

    "Kaptish" override fun init(task: JavacTask, vararg args: String?) { val classes = getClasses(task.context) Arguments.instance(task.context).classNames.addAll(classes) } “The javac command can also process annotations in Java source files and classes.” - Javac docs
  61. Kaptish Downsides • Cannot reference generated code directly in Kotlin.

    • Must have class retained annotations • Cannot generate Kotlin code
  62. Faster Compilation • Ins and Outs • Cache Keys •

    ABI Jars ◦ Class ABI jars ◦ Src ABI Jars • Per class compiler avoidance • Annotation Processing • Future: ◦ KSP ◦ K2 Compiler ◦ Graph Flattening
  63. IDEs and Devtools 63

  64. Local development • M1 Max hardware running Mac OS •

    Intellij 2022 • Uber IDE Plugins • Automatic IDE settings (Vmoptions, default settings, code style) • Integrate LDA • Managed IT and standard applications via Chef • Developer setup Android environment ❌
  65. Custom Intellij Plugins • Local Developer Analytics • Scaffold new

    Module • Scaffold new RIB with Compose Views • Scaffold new Sandbox app • Motif Dependency explorer • Realtime RIB explorer • Index new apps/modules • Manage third party plugins • Analytics line markers • Live Templates • Realtime UI updates
  66. QuickUI Realtime XML Updates Phone Uber App Custom Resource Wrapper

    Resource Only APK Resource V1 Resource V2 Resource V2 Build Command Resource Only Change? Compile Full APK and ADB push Compile Resources and ADB push No Yes
  67. QuickUI Demo

  68. Devpod (Cloud IDEs) • Remote Linux Dev Servers (400gb Ram,

    96 core) • Automatically Provisioned for Uber’s Android development • All packages are fully managed • Batteries Included! Prebuilt IDE indexes, build snapshots, 3rd party deps, and more. • IDE always running and available to connect • Running Jetbrains Gateway, Projector, and VSCode • Automatic machine suspension • Allows multiple machines • Instant context Switching “I used to barely being able to make large scale migrations on my macbook w/o steam coming out of it, and now im changing 20K LOC in 2K files w/ all modules in Intellij indexed on a cloud machine over airplane wifi. Crazy times!”
  69. compute node 0 devpod nvme devpod nvme devpod nvme devpod

    nvme • Devpod is a docker container • Running inside kubernetes • With nvme disk • Sharing powerful compute nodes with other devpods compute node 1 Devpod
  70. Devpod Regions

  71. • Base flavor contains all common software between other flavors

    • Each monorepo build their own tailored experience on top of base • Addon workflows, user dotfiles, custom packages/IDE plugins, and more customizations + Flavors
  72. Devpod IDE ADB Build Laptop ADB SSH SSHD Gateway Emulator

    / Phone gateway://… devpod init-adb mydevpod (ssh -r 5037:localhost:5037) Devpods + Android Emulators & Devices
  73. Snapshots 24 Hours Boot Devpod Build Project Clone Repo Archive

    outputs • ~/android/buck-out • ~/.idea/* folders • ~/.gradle/* folders • ~/.buck-cache • ~/.bazel-cache Upload Archives Boot IDE > devpod create Provision Devpod Boot Devpod Clone Repo Download Artifacts Extract Archives ~/android/buck-out ~/.idea/* folders ~/.gradle/* folders ~/.buck-cache ~/.bazel-cache Boot IDE Storage
  74. Indexing Project target: a project in Android monorepo * provided

    time corresponds to Intellij IDEA “Updating Index” stage Regular Indexing 38 minutes Jetbrains Shared Index Plugin 32 minutes Cached Index seconds Jetbrains Cached Index
  75. Devpod Demo

  76. Resources • RIBs - github.com/uber/ribs • Motif - github.com/uber/motif •

    Nanoscope - github.com/uber/nanoscope • Nullaway - github.com/uber/nullaway • Piranha - github.com/uber/piranha • Gradle/Buck/Bazel evaluation - github.com/uber-common/android-build-eval • Napt - github.com/sergei-lapin/napt • Flipper - github.com/facebook/flipper • Detekt - github.com/detekt/detekt • KTFMT - github.com/facebookincubator/ktfmt • Devpod like product - coder.com, Gitpod.io, & Github Code Spaces
  77. Mobile Developer Productivity at Uber Scale Ty Smith Android Platform

    Tech Lead 77 @tsmith