Slide 1

Slide 1 text

Mobile Developer Productivity at Uber Scale Ty Smith Android Platform Tech Lead 1 @tsmith

Slide 2

Slide 2 text

Agenda 01 Uber’s Scale 02 How Uber Organizes 03 Development Workflow 03 Measuring Devx 04 Making Builds Faster 05 IDE

Slide 3

Slide 3 text

Uber’s Scale 3

Slide 4

Slide 4 text

2008 2012 2016 MVC 100k LOC RIBs 1m LOC Startup 10k LOC 2021 RIBs++ 10m LOC

Slide 5

Slide 5 text

Development at Uber Scale Thousands Microservices Tens of Thousands commits per month Millions Config changes per month Trillions Mobile analytics events per month Hundreds of Millions Monorepo LOC Hundreds of Thousands Deployments per month 4K+ Engineers 7 Programming languages Tens Mobile apps Billions User sessions per month 5 Platform Monorepo’s

Slide 6

Slide 6 text

Mobile Scale 600+ Mobile Engineers Tens of Thousands Build Modules Tens of Millions Mobile LOC Thousands RIBs - Our Mobile Architecture And 100’s of internal apps

Slide 7

Slide 7 text

Development Workflow 7

Slide 8

Slide 8 text

Developers move quickly using standardized tools and frameworks Standardization Each platform lives in it’s own Monorepo. 5 Total. Monorepos Monorepo teams centrally support all engineers Centralized Teams Development Stack

Slide 9

Slide 9 text

Apps (Rider, Driver, Eats, Freight, etc) Observability & XP (Services, Tools, Libraries) Networking (Protocols , Libraries, Edge) Architecture Programming Languages Platforms Open Source Libraries Build & Tools IDEs Release & Compiler Extensions Device Lab Automated Testing Docs CI/CD Mobile Tech Stack

Slide 10

Slide 10 text

Android Tech Stack Architecture ● RIBS - Converged Architecture ● Kotlin, Compose, Coroutines ● Motif - Code generated DI Education ● Centralized documentation, Codelabs, developer onboarding, and support UI ● Stylist/Artist - View & Style generators ● Base Mobile - Compose Based UI Components ● Base SDUI - Server Driven UI Rendering Frameworks ● Citrus - Parameterized XP and Plugins ● Simple-store - Async KVS Storage Build ● Buck/Bazel - Hermetic Build system* ● Kaptish - Fast Kotlin Annotation Processing* ● Buildkite - Sharded containerized CI/mergequeue IDE and Tools ● Intellij Plugins* ● Flipper + Custom Plugins* ● Devpods - Cloud IDEs* ● QuickUI - UI Hotloading* ● LDA - Local Developer Analytics* ● Mobile Studio - Plugin based debug drawer Reliability ● Detekt/Errorprone/Lint/Shellcheck - Static Analysis ● Ktfmt/GJF - Code Formatter ● Piranha - automated XP code cleanup ● Nanoscope - Hyper performant profiler Data and Observability ● Unified Reporter - Analytics pipeline ● Jenga - GRPC Model generation pipeline ● Healthline - Crash & Observability ● Blackswan - Crash Recovery

Slide 11

Slide 11 text

Organization 11

Slide 12

Slide 12 text

Engineering Teams at Uber

Slide 13

Slide 13 text

Mobile Org ELT Platform Eng Mobility Delivery Developer Platform Rider Driver Eater Eater Features *Org structure simplified Driver Features Rider Features Rider Platform Driver Platform Eater Platform Mobile Platform

Slide 14

Slide 14 text

Mobile Platform Mobile Networking Android Foundations Mobile Data iOS Devx Mobile Platform Teams Devtools Mobile Observability Testing iOS Foundations Bug Reporting Android Devx Mobile UI

Slide 15

Slide 15 text

Mobile Platform Vision Be the industry leader for how developers build, deploy, and manage high-quality software productively and at scale.

Slide 16

Slide 16 text

Fast, modern tools Powerful frameworks Safety nets Standardization VELOCITY QUALITY EFFICIENCY Developer sentiment (NPS) Developer metrics Strategy

Slide 17

Slide 17 text

Measuring Devx 17

Slide 18

Slide 18 text

Build, test or merge failure Diff create to commit time (20-60 hr) Land time (10-45 min) Local build time (30-120 sec) Diff review time (10-40 hr) Delivery time (1 hr) Local Development Submit Queue Commit Code Review Diff create Diff land Delivery uCD ❌ CI Build ❌ ❌ Build or test failure Changes requested Diff create to AppStore (p50: 15 days) or Diff create to production Diff CI time Wait for build cut Mobile Test Submission to App/Play Stores App Store Rollouts Mobile build cut ❌ Mobile Hotfix Mobile Release (9-10 days) ❌Burnt release A SWE’s inner loop Proguard or Redex Failure ❌ Mobile Code Development

Slide 19

Slide 19 text

What We Measure ● Net Promoter Score (NPS) ● Developer Throughput ● Build Time ● Failure Rate ● Git time ● IDE Performance ● CI/CD Time ● Tooling uptime ● App Performance ● App Reliability

Slide 20

Slide 20 text

Developer Satisfaction (NPS)

Slide 21

Slide 21 text

Local Environment Local Developer Analytics (LDA) Data Collectors Data Collector Data Collector Data Collector Analytics Daemon Kafka Kibana/search Hadoop/Hive/Presto Log Collection m3/monitoring

Slide 22

Slide 22 text

Local Developer Analytics Data Collectors Git Arc Buck Bazel IDE Gradle Custom CLIs and tools

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

Apdex

Slide 25

Slide 25 text

Making Builds Faster 25

Slide 26

Slide 26 text

Uber’s Android Build Systems 2021 Evaluate Bazel Worked with Bazel and Gradle teams to evaluate both and decide to converge with Uber on Bazel. 2022 Bazel Migration ● Dynamically generate Bazel BUILD files and run in parallel with BUCK. ● Coordinate with Community ● Invest into performance and compatibility 2016 Introduced Buck with OkBuck ● Use Gradle to generate BUCK files ● < 1K Modules ● 3min to 30s p75 build time 2019 Pure Buck Stopped generating Buck files dynamically from Gradle 2023 Deprecate Buck ● Remove Buck as Shadow ● >10K Modules ● Remote Execution

Slide 27

Slide 27 text

Bazel Benefits ● Hermetic/reproducible Builds ● Highly Parallelizable ● Aligned with Industry ● Dedicated build language (Starlark) ● No side effects ● Cacheable by default ● Remote Build support ● Easily Extensible ● Converged with other platforms

Slide 28

Slide 28 text

Bazel Downsides ● Lacking out of the box experience ● Requires dedicated engineers ● Slower for new Android features

Slide 29

Slide 29 text

Mobile vs Backend Builds

Slide 30

Slide 30 text

Rider App Dependency Graph *Reduced via transitive reduction Over 12,000 module dependencies

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

FooModule BarModule dependsOn FooBarModule dependsOn BazModule dependsOn 1 2 3 4

Slide 34

Slide 34 text

Kotlin Builds 🐢 Kotlin is up to 2x slower than Java (with Errorprone)

Slide 35

Slide 35 text

Faster Compilation ● Ins and Outs ● Cache Keys ● ABI Jars

Slide 36

Slide 36 text

Faster Compilation Build Target Inputs ● Toolchain ● Dependencies ● Configuration FooModule Build Target Outputs ● JARs/AARs Cache Key: 9e92c406b5efc492a714d29c5e26516d

Slide 37

Slide 37 text

ABI Jars FooModule BarModule dependsOn ?

Slide 38

Slide 38 text

ABI Jars compile_against_abis = false Bar Module src/foo.kt fun bar() = “Foo” Bar.jar Foo.class Cache-key: 9e92c406b5efc492a714d29c5e26516d public final class FooKt { @NotNull public static final String bar() { return "Foo"; } } FooModule BarModule dependsOn

Slide 39

Slide 39 text

ABI Jars compile_against_abis = false Bar Module src/foo.kt fun bar() = “FooBar” Bar.jar Foo.class Cache-key: a28d01554c0f541c04ec3a2fe28396ed public final class FooKt { @NotNull public static final String bar() { return "FooBar"; } } FooModule BarModule dependsOn

Slide 40

Slide 40 text

ABI Jars compile_against_abis = true Bar Module src/foo.kt fun bar() = “Foo” Bar.abi.jar Foo.class Cache-key: d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } FooModule BarModule dependsOn

Slide 41

Slide 41 text

ABI Jars compile_against_abis = true Bar Module src/foo.kt fun bar() = “FooBar” Bar.abi.jar Foo.class Cache-key: d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } FooModule BarModule dependsOn

Slide 42

Slide 42 text

Faster Compilation ● Ins and Outs ● Cache Keys ● ABI Jars ○ Class ABI jars ○ Src ABI Jars

Slide 43

Slide 43 text

Bar Module src/foo.kt fun bar() = “Foo” Bar.abi.jar Foo.class Cache-key: d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } Class ABI Jars Bar.jar Foo.class Cache-key: a28d01554c0f541c04ec3a2fe28396ed public final class FooKt { @NotNull public static final String bar() { return "FooBar"; } } ABI Generator (ASM) javac/kotlinc

Slide 44

Slide 44 text

Bar Module src/foo.kt fun bar() = “Foo” Bar.abi.jar Foo.class Cache-key: d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } Source ABI Jars Bar.jar Foo.class Cache-key: a28d01554c0f541c04ec3a2fe28396ed public final class FooKt { @NotNull public static final String bar() { return "FooBar"; } } javac/kotlinc

Slide 45

Slide 45 text

Javac Bar Module src/foo.kt fun bar() = “Foo” Bar.abi.jar Foo.class Cache-key: d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } Source ABI Jars (Java) Bar.jar Foo.class Cache-key: a28d01554c0f541c04ec3a2fe28396ed public final class FooKt { @NotNull public static final String bar() { return "FooBar"; } } parse+enter analyze+generate

Slide 46

Slide 46 text

Rule Pipelining (Java) Bar.abi.jar BarModule Javac parse+enter analyze+generate FooModule BarModule dependsOn 1 FooModule Javac parse+enter analyze+generate 2

Slide 47

Slide 47 text

Kotlinc Bar Module src/foo.kt fun bar() = “Foo” Bar.abi.jar Foo.class Cache-key: d3b7138441694bea985e32ef219697b6 public final class FooKt { @NotNull public static final String bar() { } } Source ABI Jars (Kotlin) Bar.jar Foo.class Cache-key: a28d01554c0f541c04ec3a2fe28396ed public final class FooKt { @NotNull public static final String bar() { return "FooBar"; } } Jvm-abi-gen

Slide 48

Slide 48 text

Mixed Java and Kotlin Sources Kotlinc Jvm-abi-gen Javac parse+enter analyze+generate kapt ksp FooBarModule foo.kt bar.java foo.class bar.class Compress foobar.jar 1 2 3

Slide 49

Slide 49 text

Kotlin Rule Pipelining Making Mixed Java/Kt Sources Faster Kotlinc Jvm-abi-gen Javac parse+enter analyze+generate kapt ksp FooBarModule foo.kt bar.java foo.class bar.class Compress foobar.abi.jar 1 2 4 Compress foobar.jar foo.abi.class bar.abi.class 3 5

Slide 50

Slide 50 text

Kotlin Rule Pipelining Making Mixed Java/Kt Sources Faster FooModule BarModule dependsOn BarModule Javac analyze+generate parse+enter Jar Kotlinc ABI Jar FooModule Jvm-Abi-Gen Javac analyze+generate parse+enter Jar Kotlinc ABI Jar Jvm-Abi-Gen

Slide 51

Slide 51 text

Faster Compilation ● Ins and Outs ● Cache Keys ● ABI Jars ○ Class ABI jars ○ Src ABI Jars ● Per class compiler avoidance

Slide 52

Slide 52 text

Per Class Compiler Avoidance FooModule src/foo.kt fun foo() { bar() } BarModule src/bar.kt fun bar() dependsOn FooBarModule src/foobar.kt fun foobar()

Slide 53

Slide 53 text

Per Class Compiler Avoidance FooBarModule src/foobar.kt fun foobar() FooModule src/foo.kt fun foo() { bar() } BarModule src/bar.kt fun bar() dependsOn Class in jar is used? Class in jar is used?

Slide 54

Slide 54 text

Kotlin Class Usage Tracking Compiler Plugin ClassUsageCompilerPlugin : AnalysisHandlerExtension, StorageComponentContainerExtension fun analysisCompleted() { collectClasses(...) } ClassCollector : CallChecker, DeclarationChecker fun check(ResolvedCall) { registerClass(...) } fun check(KtDeclaration) { registerClass(...) } Used Classes Storage - JDeps - used-classes.json

Slide 55

Slide 55 text

Faster Compilation ● Ins and Outs ● Cache Keys ● ABI Jars ○ Class ABI jars ○ Src ABI Jars ● Per class compiler avoidance ● Annotation Processing

Slide 56

Slide 56 text

Annotation Processing ● Kotlin Annotation Processing Tool (KAPT) ○ Slow ● Java Annotation Processing (AP) ○ Fast ○ Doesn’t support Kotlin ● Kotlin Symbol Processing (KSP) ○ Fast ○ Doesn’t have wide support

Slide 57

Slide 57 text

Kotlin Annotation Processing FooBarModule Foo.kt @Foo Foo.class Foo_Impl.class KAPT slows build time by more than 150% compared to Kotlin without AP or Javac AP Kotlinc KAPT Stubs Annotation Processing Compilation Foo_Impl.kt

Slide 58

Slide 58 text

Kotlin Annotation Processing FooBarModule Foo.kt @Foo Foo.class Foo_Impl.class Javac Bar.java @Bar Bar.class Bar_Impl.class Kotlinc KAPT Stubs Annotation Processing Compilation Foo_Impl.kt

Slide 59

Slide 59 text

Kotlin Annotation Processing FooBarModule Foo.kt @Foo Foo.class @Foo Javac Bar.java @Bar Bar.class Bar_Impl.class Kotlinc Kaptish Foo_Impl.class

Slide 60

Slide 60 text

Kaptish class Kaptish : Plugin { override fun getName() = "Kaptish" override fun init(task: JavacTask, vararg args: String?) { val classes = getClasses(task.context) Arguments.instance(task.context).classNames.addAll(classes) } “The javac command can also process annotations in Java source files and classes.” - Javac docs

Slide 61

Slide 61 text

Kaptish Downsides ● Cannot reference generated code directly in Kotlin. ● Must have class retained annotations ● Cannot generate Kotlin code

Slide 62

Slide 62 text

Faster Compilation ● Ins and Outs ● Cache Keys ● ABI Jars ○ Class ABI jars ○ Src ABI Jars ● Per class compiler avoidance ● Annotation Processing ● Future: ○ KSP ○ K2 Compiler ○ Graph Flattening

Slide 63

Slide 63 text

IDEs and Devtools 63

Slide 64

Slide 64 text

Local development ● M1 Max hardware running Mac OS ● Intellij 2022 ● Uber IDE Plugins ● Automatic IDE settings (Vmoptions, default settings, code style) ● Integrate LDA ● Managed IT and standard applications via Chef ● Developer setup Android environment ❌

Slide 65

Slide 65 text

Custom Intellij Plugins ● Local Developer Analytics ● Scaffold new Module ● Scaffold new RIB with Compose Views ● Scaffold new Sandbox app ● Motif Dependency explorer ● Realtime RIB explorer ● Index new apps/modules ● Manage third party plugins ● Analytics line markers ● Live Templates ● Realtime UI updates

Slide 66

Slide 66 text

QuickUI Realtime XML Updates Phone Uber App Custom Resource Wrapper Resource Only APK Resource V1 Resource V2 Resource V2 Build Command Resource Only Change? Compile Full APK and ADB push Compile Resources and ADB push No Yes

Slide 67

Slide 67 text

QuickUI Demo

Slide 68

Slide 68 text

Devpod (Cloud IDEs) ● Remote Linux Dev Servers (400gb Ram, 96 core) ● Automatically Provisioned for Uber’s Android development ● All packages are fully managed ● Batteries Included! Prebuilt IDE indexes, build snapshots, 3rd party deps, and more. ● IDE always running and available to connect ● Running Jetbrains Gateway, Projector, and VSCode ● Automatic machine suspension ● Allows multiple machines ● Instant context Switching “I used to barely being able to make large scale migrations on my macbook w/o steam coming out of it, and now im changing 20K LOC in 2K files w/ all modules in Intellij indexed on a cloud machine over airplane wifi. Crazy times!”

Slide 69

Slide 69 text

compute node 0 devpod nvme devpod nvme devpod nvme devpod nvme ● Devpod is a docker container ● Running inside kubernetes ● With nvme disk ● Sharing powerful compute nodes with other devpods compute node 1 Devpod

Slide 70

Slide 70 text

Devpod Regions

Slide 71

Slide 71 text

● Base flavor contains all common software between other flavors ● Each monorepo build their own tailored experience on top of base ● Addon workflows, user dotfiles, custom packages/IDE plugins, and more customizations + Flavors

Slide 72

Slide 72 text

Devpod IDE ADB Build Laptop ADB SSH SSHD Gateway Emulator / Phone gateway://… devpod init-adb mydevpod (ssh -r 5037:localhost:5037) Devpods + Android Emulators & Devices

Slide 73

Slide 73 text

Snapshots 24 Hours Boot Devpod Build Project Clone Repo Archive outputs ● ~/android/buck-out ● ~/.idea/* folders ● ~/.gradle/* folders ● ~/.buck-cache ● ~/.bazel-cache Upload Archives Boot IDE > devpod create Provision Devpod Boot Devpod Clone Repo Download Artifacts Extract Archives ~/android/buck-out ~/.idea/* folders ~/.gradle/* folders ~/.buck-cache ~/.bazel-cache Boot IDE Storage

Slide 74

Slide 74 text

Indexing Project target: a project in Android monorepo * provided time corresponds to Intellij IDEA “Updating Index” stage Regular Indexing 38 minutes Jetbrains Shared Index Plugin 32 minutes Cached Index seconds Jetbrains Cached Index

Slide 75

Slide 75 text

Devpod Demo

Slide 76

Slide 76 text

Resources ● RIBs - github.com/uber/ribs ● Motif - github.com/uber/motif ● Nanoscope - github.com/uber/nanoscope ● Nullaway - github.com/uber/nullaway ● Piranha - github.com/uber/piranha ● Gradle/Buck/Bazel evaluation - github.com/uber-common/android-build-eval ● Napt - github.com/sergei-lapin/napt ● Flipper - github.com/facebook/flipper ● Detekt - github.com/detekt/detekt ● KTFMT - github.com/facebookincubator/ktfmt ● Devpod like product - coder.com, Gitpod.io, & Github Code Spaces

Slide 77

Slide 77 text

Mobile Developer Productivity at Uber Scale Ty Smith Android Platform Tech Lead 77 @tsmith