Efficient Android Builds at Scale: Lessons from Pinterest

Transcript

1. Efficient Android Builds at Scale: Lessons from Pinterest David Chang 1
2. Agenda 1. Pinterest Scale 2. Gradle • Basics • Caching 3. Project Setup 4. CI 2
3. Pinterest Scale 3
4. Pinterest Scale • 458 Gradle Modules • 30k hours of CI builds per month • 720 hours of local builds per month • 100+ Montly Active Android developers • > 1 million lines of code 4
5. ! Gradle 5
6. Gradle 1. Initialization • Determines which projects (modules) are participating in the build 2. Configuration • Evaluates build.gradle and creates the tasks for all the projects and generates the task graph 3. Execution • Runs the tasks 6
8. Remote cache Develocity Gradle Local cache Incremental build 1 2 3 Read/Write Read/Write Read/Write 8
9. ! Project Setup 9
10. ! settings.gradle def subDirsList = [ "codelab", "gql", "library", "test-library", "fakes-library", "feature-library", "feature", "tests" ] for(path in projectPaths.keySet()){ include path project(path).buildFileName = projectPaths.get(path) } 10
11. Gradle Plugins - Pinterest Build Tools • Android Manifest Validation • Firebase Test Lab • Metro • GraphQL Registry • Gradle Build Scan • Module Score 11
12. Developer Tools • Android Studio Plugin: State Based Architecture • Module scripts 12
13. Convention Plugin Plugin that encapsulates the build logic common to several subprojects in a project 13
14. Convention Plugin // AndroidFeatureLibraryConventionPlugin.kt class AndroidFeatureLibraryConventionPlugin : Plugin<Project> { override fun apply(target: Project) { with(target) { with(pluginManager) { apply("com.android.library") apply(AndroidBasicConventionPlugin::class.java) apply("pinterest.android.hilt") apply("pinterest.android.dagger") } extensions.configure(LibraryExtension::class.java) { t -> configureApolloDependencies() // Also seems unnecessary but we see various build failures without it with(dependencies) { add("implementation", project(":library:navigation")) } } enforceModuleDependencies(NoFeatureModuleDependencies) } } } 14
15. Convention Plugin // convention.gradle register("androidFeatureLibrary") { id = "pinterest.android.feature-library" implementationClass = "AndroidFeatureLibraryConventionPlugin" } 15
17. Local Build Metrics Metric Value H2 Goal P50 0:41m 0:33m P95 5:33m 6:40m 17
18. Local Build Improvements • P50 • Gradle Configuration Cache • P95 • Gradle Cache Prefetcher (internal) 18
19. Local Development Flow • Main local variant - fastdevDebug Flavor Build Type fastdev debug alpha release ota staging production benchmark amazon 19
20. Local Development Flow • Unnecessary plugins gated with enableAllTasksAndPlugins • Unnecessary variants gated with enableExtraVariants 20
21. ! CI 21
22. Workflow 22
23. CI Build Metrics Metric Value H2 Goal Master SR 98.40% 97.00% PR P50 13:35m 15:00m PR P95 26:25m 30:00m MQ P50 9:06m 10:00m MQ P95 17:09m 18:00m 23
24. Infra • prod-light • c7i.xlarge - 4 cores 3.2 ghz, 8 gb • prod • r6i.2xlarge - 8 cores 3.5 ghz, 64 gb • prod-highcpu • r6i.4xlarge - 16 cores 3.5 ghz, 128 gb 24
25. Infra • prod-light • 75 nodes • prod • 190 nodes • prod-highcpu • 24 nodes 25
26. Docker 26
27. Github 27
28. CI • Pipeline code is in repo .buildkite/ • IS_CI=true automatically sets other properties 28
29. CI 29
30. Monitoring 30
31. Ground Control Data Aspect Baseline H2 Goal H2 9/16/24 9/9/24 9/3/24 Documentation Contributor - - Person A Person B Person C Local Build Time P50 (s) 63 30 47 44 46 Local Build Time P95 (s) 626 400 480 492 470 PR Pipeline Time P50 (s) 900 900 750 856 905 PR Pipeline Time P95 (s) 1920 1920 1860 1979 1962 Master Success Rate 98.9% 97.0% 99.81% 93.2% 94.4% 31
33. We're Hiring! https://www.pinterestcareers.com 33
34. Q&A @changusmc (X) @changd24 (Linkedin) 34