Performant Android CI

This talk will cover a breadth of topics to make Android CI pipelines as fast as possible: caching every bit of Gradle possible, custom git checkouts, a look at relevant JVM args, how we can apply layered Docker images and design a supporting CI pipeline.

Transcript

1. Performant Android CI Mar 20th, 2025 Jason Pearson
2. Performant Android CI Mar 20th, 2025 Jason Pearson
3. Performant Android CI Jason Pearson
4. Performant Android CI Jason Pearson
5. Performant Android CI Jason Pearson The Purpose of CI 💻 The Journey to Performant Android CI
6. Performant Android CI Jason Pearson The Purpose of CI 💻 The Journey to Performant Android CI
7. Performant Android CI Build & Cache Techniques 📦 CI Pipeline Design Layered Docker Images Git Custom Checkout & Fetch
8. Performant Android CI Build & Cache Techniques 📦 CI Pipeline Design Layered Docker Images Git Custom Checkout & Fetch
9. The Purpose of CI
10. The Purpose of CI 💻
11. The Purpose of CI 💻
12. The Purpose of CI 💻
13. 💻
14. 💻
15. 💻
16. 💻
17. 💻
18. 💻
19. 💻
20. 💻
21. 💻
22. 💻
23. 💻
24. 💻 🧪
25. 💻 🧪
26. 💻 🧪
27. 💻 🧪
28. 💻 🧪 🐉
29. 💻 🧪 🐉
30. 💻 🧪 🐉
31. 💻 🧪 🐉
32. 💻 🧪 🐉
33. 💻 🧪 🐉
34. 💻 🧪 🐉
35. ~900 Modules 💻 ~1M lines of code ~100 contributors Codebase Context Dagger Anvil (KAPT) 🗡 Gradle Kotlin Scripts Kotlin 1.9
36. CI Context Single Docker image Datadog, Umbra Firebase Test Lab Gradle Develocity Self hosted EC2 Runners Artifactory GitLab CI
37. Very long MR pipeline times
38. • 3-4 minutes pipeline veri fi cation jobs • 30 minutes in Lint • 50+ minutes in Test • ~30 minutes Building APKs • ~15 minutes running integration tests that are unreliable Very long MR pipeline times
39. The Journey to Performant Android CI
40. The Journey to Performant Android CI Some caching via Develocity 📦 All artifacts all the time Full git clone via GitLab Slow and flaky UI tests
41. The Journey to Performant Android CI Some caching via Develocity 📦 All artifacts all the time Full git clone via GitLab Slow and flaky UI tests
42. The Journey to Performant Android CI Everything cached 📦 All artifacts all the time Full git clone via GitLab Slow and flaky UI tests
43. The Journey to Performant Android CI Everything cached 📦 Minimize git overhead All artifacts all the time Slow and flaky UI tests
44. The Journey to Performant Android CI Everything cached 📦 Minimize git overhead Fast & Reliable UI Tests All artifacts all the time
45. The Journey to Performant Android CI Everything cached 📦 Only artifacts as needed Fast & Reliable UI Tests Minimize git overhead
46. Fast & Reliable Later… Minimize git overhead The Journey to Performant Android CI Everything cached 📦 Only artifacts as needed
47. Fast & Reliable Only artifacts as needed Minimize git overhead The Journey to Performant Android CI Everything cached 📦 Later…
48. Fast & Reliable Only artifacts as needed Minimize git overhead The Journey to Performant Android CI Everything cached 📦 Later…
49. Fast & Reliable Only artifacts as needed Minimize git overhead The Journey to Performant Android CI Later… Build & Cache Techniques 📦
50. Fast & Reliable Only artifacts as needed The Journey to Performant Android CI Later… Build & Cache Techniques 📦 Git Custom Checkout & Fetch
51. Fast & Reliable Later… CI Pipeline Design The Journey to Performant Android CI Git Custom Checkout & Fetch Build & Cache Techniques 📦
52. 📦 The Journey to Performant Android CI Build & Cache Techniques
53. Build & Cache Techniques 📦
54. JVM Memory Sizing Heap Non-Heap
55. JVM Memory Sizing Heap Non-Heap Metaspace CodeCache Young & Old Gen CompressedClassPointers GC Buffers
56. JVM Memory Sizing Heap Non-Heap Metaspace CodeCache Young & Old Gen CompressedClassPointers GC Buffers
57. JVM Memory Sizing Gradle Daemon
58. JVM Memory Sizing Gradle Daemon Kotlin Daemon
59. JVM Memory Sizing Gradle Daemon Kotlin Daemon Gradle Workers
60. JVM Memory Sizing Gradle Daemon Kotlin Daemon Gradle Workers
61. JVM Memory Sizing Gradle Daemon Kotlin Daemon Gradle Workers Remaining Memory Divided for Workers
62. JVM Memory Sizing Gradle Daemon Kotlin Daemon Gradle Workers Remaining Memory Divided for Workers
63. JVM Memory Sizing Gradle Daemon Kotlin Daemon Gradle Workers Remaining Memory Divided for Workers
64. JVM Memory Sizing Gradle Daemon Kotlin Daemon Gradle Workers Remaining Memory Divided for Workers
65. JVM Memory Sizing Gradle Daemon Kotlin Daemon Gradle Workers Remaining Memory Divided for Workers
66. JVM Memory Sizing Gradle Daemon Kotlin Daemon Gradle Workers Remaining Memory Divided for Workers
67. JVM Memory Sizing Gradle Daemon Kotlin Daemon Gradle Workers Remaining Memory Divided for Workers
68. JVM Memory Sizing Gradle Daemon Kotlin Daemon Gradle Workers Remaining Memory Divided for Workers
69. JVM Memory Sizing Gradle Daemon Kotlin Daemon Gradle Workers Remaining Memory Divided for Workers
70. JVM Memory Sizing Non-Heap memory has internal competition
71. JVM Memory Sizing Gradle Daemon Kotlin Daemon Gradle Workers Remaining Memory Divided for Workers
72. JVM Memory Sizing Gradle Daemon Kotlin Daemon Gradle Workers Remaining Memory Divided for Workers Compilation Test
73. JVM Memory Sizing Gradle Workers Compilation Test
74. JVM Memory Sizing Gradle Workers Compilation Test org.gradle.workers.max maxParallelForks 256m 512m Default Heaps:
75. The number of milliseconds to hold onto soft references per MB of heap size, default 1000 = 2h 16m 32s 8GB Heap @ 1 MS Policy 8GB Heap @ 1000 MS Policy = 0h 0m 8s 8GB Heap @ 10 MS Policy = 0h 1m 22s SoftRefLRUPolicyMSPerMB JVM Memory Sizing
76. JVM Memory Sizing
77. JVM Memory Sizing Coming Soon: Tool for Sizing
78. tcmalloc malloc replacements jemalloc Freeing up as much memory as possible. Optimize for speed while keeping memory usage steady. Pending evaluation…
79. Android Lint Best Practices 🐉
80. Android Lint Best Practices 🐉 https://googlesamples.github.io/android-custom-lint-rules/ ✅ ignoreTestSources ❌ checkAllWarnings ✅ checkDependencies Do the following:
81. Build Cache Configuration Cache Dependency Cache Script Cache Gradle Caches
82. Build Cache EXECUTING Configuration Cache Dependency Cache Script Cache Gradle Caches
83. Build Cache EXECUTING Configuration Cache Dependency Cache Script Cache Gradle Caches
84. Build Cache EXECUTING Configuration Cache Dependency Cache Script Cache Gradle Caches
85. Build Cache CONFIGURING EXECUTING Configuration Cache Dependency Cache Script Cache Gradle Caches
86. Build Cache Configuration Cache Dependency Cache Script Cache Gradle Caches
87. Gradle Build Cache
88. • Reduce the number of inputs per task and make them more stable • Gradle build scans to observe performance Gradle Build Cache
89. Gradle Build Cache
90. Gradle Configuration Cache on CI
91. 2-3 min possible savings per job Gradle Configuration Cache on CI
92. Relies on caching outputs of configuration phase, which rely on all inputs used in the build. So we need to: 1. Find all inputs 2. Hash them 3. Search cache provider Gradle Configuration Cache on CI
93. 75 60 45 30 15 0 seconds Find Grep Find w/Prune ripgrep Source Hashing Search speed 🔎
94. ~/.gradle/caches/<< gradle_version >>/dependencies-accessors ~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/<< gradle_version >>/transforms ~/.gradle/caches/modules-2 ~/.gradle/caches/jars-9 .gradle/<< gradle_version >> .gradle/configuration-cache build-logic/**/build ci-cache.yml
95. $ aws s3 cp “$(pwd)” ”$S3_BUCKET_AND_PATH" \ --exclude “*" \ --include “*.zip" \ --recursive AWS S3 + ZIP
96. Gradle Configuration Cache on CI Reusing configuration cache.
97. Gradle Configuration Cache on CI 🚀
98. ➜ .gradle git:(main) ✗ du -sh 8.11.1 7.4M 8.11.1 ➜ .gradle git:(main) ✗ du -sh configuration-cache 1.7M configuration-cache Gradle Configuration Cache on CI
99. ➜ 8.11.1 du -sh transforms 1.1G transforms ➜ 8.11.1 du -sh generated-gradle-jars 177M generated-gradle-jars ➜ 8.11.1 du -sh kotlin-dsl 37M kotlin-dsl ➜ 8.11.1 du -sh dependencies-accessors 12M dependencies-accessors Gradle Configuration Cache on CI
100. 10 8 6 4 2 0 seconds tar gzip pigz zstd lz4 Compression Options - against 1 GB 🗜
101. 10 8 6 4 2 0 seconds tar gzip pigz zstd lz4 Compression Options - against 1 GB 🗜 with multiple cores
102. 10 8 6 4 2 0 seconds tar gzip pigz zstd lz4 Compression Options - against 1 GB 🗜 with multiple cores
103. 10 8 6 4 2 0 seconds tar gzip pigz zstd lz4 Compression Options - against 1 GB 🗜 with multiple cores
104. 10 8 6 4 2 0 seconds tar gzip pigz zstd lz4 Compression Options - against 1 GB 🗜 with multiple cores
105. zstd at its fastest speed has better compression than gzipped tar lz4 is lossless, which means its compression is almost nothing Multi-threaded didn’t make a difference in production Compression Options 🗜
106. AWS S3 + ZSTD # find sources via ripgrep # dry run # create zstd archives # full run $ upload zstd archives via AWS S3
107. Gradle Configuration Cache on CI
108. Gradle Configuration Cache on CI Oops
109. Gradle Caches on CI 🤔 Config Cache Dependencies Script Cache Build Cache
110. Gradle Caches on CI Config Cache Dependencies Script Cache Build Cache 💡 Task History
111. Gradle Caches on CI Config Cache Dependencies Script Cache Build Cache 💡 Task History
112. Gradle User Home in Docker COPY . /prebuild
113. Gradle User Home in Docker Custom git checkout Supporting Pipeline Layered images Track size & drift
114. Custom git checkout & fetch
115. GitLab limited options and inefficient Custom git checkout & fetch Minimize layer size
116. git clone commit tree blob
117. git clone HEAD commit tree blob
118. git clone: blobless HEAD commit tree blob
119. git clone: treeless HEAD commit tree blob
120. git clone: shallow HEAD commit tree blob
121. 75 60 45 30 15 0 seconds Custom git checkout & fetch default clone blobless clone treeless clone shallow clone
122. 75 60 45 30 15 0 seconds default clone blobless clone treeless clone shallow clone Custom git checkout & fetch treeless fetch
123. Custom git checkout & fetch
124. COPY . /prebuild Layered images
125. FROM zillow-docker.artifactory… COPY . /prebuild Layered images
126. FROM zillow-docker.artifactory… COPY . /prebuild RUN \ rm -rf /root/.gradle && \ mkdir -p /root/.gradle && \ mv /prebuild/gradle-user-home-backup / root/.gradle Layered images
127. FROM zillow-docker.artifactory… COPY . /prebuild RUN \ rm -rf /root/.gradle && \ mkdir -p /root/.gradle && \ mv /prebuild/gradle-user-home-backup / root/.gradle Layered images
128. # .dockerignore **/build/ Layered images
129. Layered images
130. Layered images
131. Layered images
132. Layered images Building Real Estate Debug APK Previously 30 minutes Down to 2 minutes on complete cache hit
133. 🧪 Layered images Android Unit Tests Previously 50 minutes Down to 4 minutes on complete cache hit
134. CI Pipeline Design
135. Automate building, publishing, using Docker images Flatten to reduce negative space and parallelize work Use downstream pipelines to organize and declutter CI Pipeline Design
136. CI Pipeline Design: Iterations
137. CI Pipeline Design: Iterations
138. CI Pipeline Design: Iterations
139. CI Pipeline Design: Critical Checks Group checks that always complete in 60 seconds or less. Fail entire pipeline if critical check fails, but otherwise let pipelines continue.
140. CI Pipeline Design: Critical Checks
141. CI Pipeline Design: Critical Checks
142. CI Pipeline Design: Team Jobs Dynamically run team specific work. Make it easy to keep author-team relationship up to date.
143. CI Pipeline Design: Team Jobs
144. CI Pipeline Design: Today
145. CI Pipeline Design: Today
146. CI Pipeline Design: Today
147. CI Pipeline Design: Today
148. CI Pipeline Design: Today
149. CI Pipeline Design: Downstream Included Build Gradle Code Analysis Unit Tests
150. Observability - Docker Overhead
151. Observability - Docker Overhead
152. 3 2 1 0 minutes Original Gradle Tools Observability - Docker Overhead
153. 3 2 1 0 minutes Original Gradle Tools Increased pull time nullified with pre-pulling images Observability - Docker Overhead
154. 3 2 1 0 minutes Original Gradle Tools Increased pull time nullified with pre-pulling images Observability - Docker Overhead
155. Observability - Code Drift git rev-list --left-right \ --count “${OLD_SHA}...${NEW_SHA}"
156. Observability - Code Drift
158. So how fast is it today?
159. p75 30 min
160. Observability - Code Drift
161. The Journey to Performant Android CI
162. What’s next? Emulator.wtf Local Build Perf Upgrading to Kotlin 2 💻
163. Jason Pearson Questions?