From Laptop Builds to Advanced CI

How do you transition from the solo-coder mindset to building a robust, automated CI pipeline that supercharges your team?

Once you get a basic pipeline running there are numerous aspects to evaluate:

* Are the results useful?
* You start to add unit test results and collect build artifacts. Is it fast and reliable enough that the team benefits from it?
* Do you need a merge queue or do you need better checks?
* What is consistently missed that you can automate?
* What tedious repetitive tasks can you transform into a welcome resource?
* Are you hitting resource limits that require an engineering investment to mitigate?
* What JVM options are right for your codebase?
* Is Docker useful? How far should you go in your optimizations?
* How do you secure the pipeline?

We'll delve into each topic and share how to apply our learnings to empower you. Along the way we will discuss how to approach stakeholders outside engineering to demonstrate the value it brings to a business. Join us for a saga of struggles and victories and how we transformed our CI pipeline at a modern scale-up business.

Transcript

1. From Laptop Builds to Advanced CI Oct 27th, 2023 Jason Pearson
 Senior Staff Engineer Formerly Hinge
2. First, about me Jason Pearson
7. CI experience since 2011. Built Android at Hinge, went from solo dev to 24 Android engineers. Also spent years in backend and fullstack roles across various companies.
8. Agenda Laptop Builds 💻 Automate All The Things Make it Faster
9. Make it Faster 12 9 6 3 0 Unit Tests Release R8 UI Tests Unit Tests Release R8 UI tests minutes
10. ~400 Modules 💻 290k lines of Kotlin 24 active contributors Context Dagger Hilt (KAPT) 🗡 Gradle Kotlin Scripts KSP (except Dagger)
11. Problems 🤯 Distracting from Feature Work ❌ Regressions 🏴☠ Security Concerns
13. Our Story
14. How every Android project starts 💻
15. How every Android project starts 💻
16. How every Android project starts 💻
17. 💻
18. 💻
19. 💻
20. 💻
21. 💻
22. 💻
23. 💻
24. 💻
25. 💻
26. 💻
27. 💻 Problems 🤯 Distracting from Feature Work
28. 💻 Problems 🤯 Distracting from Feature Work
29. 💻 Problems 🤯 Distracting from Feature Work
30. 💻 Problems
31. 💻
32. 💻 💻
33. 💻 💻 💻 💻
34. Problems 🤯 Distracting from Feature Work
35. Problems 🤯 Distracting from Feature Work ❌ Regressions
36. Problems 🤯 Distracting from Feature Work ❌ Regressions 🏴☠ Security Concerns
37. Problems 🤯 Distracting from Feature Work ❌ Regressions 🏴☠ Security Concerns CI Solutions 🏗 Automate Building & Publishing 👩🔬 Automate Tests 🔐 Automate Security
38. When do we build? What do we run? Self hosted or Managed? What should CI look like?
39. What should CI look like? This talk will focus on builds on commit push
40. Automating Security
41. Handle secrets appropriately Automating Security
42. Handle secrets appropriately Automating Security 🔐
43. Handle secrets appropriately Automating Security 🔐
44. Handle secrets appropriately No personal user tokens Automating Security 🔐
45. Testing on CI 👩🔬
46. Testing on CI 👩🔬
47. config.yml 🚧
48. version: 2.1 jobs: unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout config.yml 🚧
49. version: 2.1 jobs: unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest config.yml 🚧
50. 🚧
51. * What went wrong: Execution failed for task ':features:user-account:fresh- start:testInternalDebugUnitTest'. > There were failing tests. See the report at: file:///mnt/ramdisk/ features/user-account/fresh-start/build/reports/tests/ testInternalDebugUnitTest/index.html * Exception is: org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':features:user-account:fresh-start:testInternalDebugUnitTest'. at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$e xecuteIfValid$1(ExecuteActionsTaskExecuter.java:148) at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeI fValid(ExecuteActionsTaskExecuter.java:146) at 🚧
52. * What went wrong: Execution failed for task ':features:user-account:fresh- start:testInternalDebugUnitTest'. > There were failing tests. See the report at: file:///mnt/ramdisk/ features/user-account/fresh-start/build/reports/tests/ testInternalDebugUnitTest/index.html * Exception is: org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':features:user-account:fresh-start:testInternalDebugUnitTest'. at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$e xecuteIfValid$1(ExecuteActionsTaskExecuter.java:148) at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeI fValid(ExecuteActionsTaskExecuter.java:146) at 🚧
53. * What went wrong: Execution failed for task ':features:user-account:fresh- start:testInternalDebugUnitTest'. > There were failing tests. See the report at: file:///mnt/ramdisk/ features/user-account/fresh-start/build/reports/tests/ testInternalDebugUnitTest/index.html * Exception is: org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':features:user-account:fresh-start:testInternalDebugUnitTest'. at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$e xecuteIfValid$1(ExecuteActionsTaskExecuter.java:148) at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeI fValid(ExecuteActionsTaskExecuter.java:146) at 🚧
54. version: 2.1 jobs: unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest config.yml 🚧
55. unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest config.yml 🚧
56. unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - store_test_results: path: ~/test-results config.yml 🚧
57. unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | when: always - store_test_results: path: ~/test-results config.yml 🚧
58. unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | mkdir -p ~/test-results/junit/ when: always - store_test_results: path: ~/test-results config.yml 🚧
59. unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | mkdir -p ~/test-results/junit/ find . when: always - store_test_results: path: ~/test-results config.yml 🚧
60. unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | mkdir -p ~/test-results/junit/ find . \ -type f when: always - store_test_results: path: ~/test-results config.yml 🚧
61. unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | mkdir -p ~/test-results/junit/ find . \ -type f \ -regex “.*/build/test-results/.*xml when: always - store_test_results: path: ~/test-results config.yml 🚧
62. unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | mkdir -p ~/test-results/junit/ find . \ -type f \ -regex “.*/build/test-results/.*xml” \
 -exec cp {} when: always - store_test_results: path: ~/test-results config.yml 🚧
63. unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | mkdir -p ~/test-results/junit/ find . \ -type f \ -regex “.*/build/test-results/.*xml” \
 -exec cp {} ~/test-results/junit/ \; when: always - store_test_results: path: ~/test-results config.yml 🚧
64. unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | mkdir -p ~/test-results/junit/ find . \ -type f \ -regex “.*/build/test-results/.*xml” \
 -exec cp {} ~/test-results/junit/ \; when: always - store_test_results: path: ~/test-results config.yml 🚧
65. unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | mkdir -p ~/test-results/junit/ find . \ -type f \ -regex “.*/build/test-results/.*xml” \
 -exec cp {} ~/test-results/junit/ \; when: always - store_test_results: path: ~/test-results config.yml ✅
66. version: 2.1 jobs: ui_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew connectedDebugAndroidTest config.yml ✅ 🚧
67. version: 2.1 jobs: ui_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew connectedDebugAndroidTest config.yml ❌ ✅ 🚧
68. Do not run emulators on CI yourself ❌ Slow Flaky
69. But how should we automate UI tests? Don’t write them to begin with
70. But how should we automate UI tests? Don’t write them to begin with OR
71. But how should we automate UI tests? Don’t write them to begin with Use a fast vendor OR
72. 💻 Manual Publishing
73. Automate Publishing 💻
74. ᐅ BikeShed > ./gradlew tasks > Task :tasks ------------------------------------------------------------ Tasks runnable from root project 'BikeShed' ------------------------------------------------------------ Android tasks ------------- androidDependencies - Displays the Android dependencies of the project. signingReport - Displays the signing info for the base and test modules sourceSets - Prints out all the source sets defined in this project. Build tasks ----------- assemble - Assemble main outputs for all the variants. assembleAndroidTest - Assembles all the Test applications. build - Assembles and tests this project. buildDependents - Assembles and tests this project and all projects that depend on it. buildKotlinToolingMetadata - Build metadata json file containing information about the used Kotlin tooling buildNeeded - Assembles and tests this project and all projects it depends on.
75. Android tasks ------------- androidDependencies - Displays the Android dependencies of the project. signingReport - Displays the signing info for the base and test modules sourceSets - Prints out all the source sets defined in this project. Build tasks ----------- assemble - Assemble main outputs for all the variants. assembleAndroidTest - Assembles all the Test applications. build - Assembles and tests this project. buildDependents - Assembles and tests this project and all projects that depend on it. buildKotlinToolingMetadata - Build metadata json file containing information about the used Kotlin tooling buildNeeded - Assembles and tests this project and all projects it depends on. bundle - Assemble bundles for all the variants. clean - Deletes the build directory. compileDebugAndroidTestSources compileDebugSources compileDebugUnitTestSources compileReleaseSources compileReleaseUnitTestSources Build Setup tasks ----------------- init - Initializes a new Gradle build. wrapper - Generates Gradle wrapper files.
76. Android tasks ------------- androidDependencies - Displays the Android dependencies of the project. signingReport - Displays the signing info for the base and test modules sourceSets - Prints out all the source sets defined in this project. Build tasks ----------- assemble - Assemble main outputs for all the variants. assembleAndroidTest - Assembles all the Test applications. bundle - Assemble bundles for all the variants.
77. version: 2.1 jobs: build: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew assembleDebug - run: ./gradlew assembleDebugAndroidTest - run: scripts/upload_and_test.sh config.yml 👩🔬
78. version: 2.1 jobs: publish: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew bundleRelease config.yml
79. version: 2.1 jobs: publish: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew bundleRelease - run: ./gradlew publishBundle config.yml
80. version: 2.1 jobs: publish: docker: - image: circleci/android:api-34-node steps: - checkout - run: | openssl aes-256-cbc -d \ -in bikeshed-keystore-secret \ -out ./app/bikeshed.keystore \ -k $SECRET_KEY - run: ./gradlew bundleRelease - run: ./gradlew publishBundle config.yml
81. Static Analysis ktlint to ensure consistency in writing detekt for code smells xmlstarlet to validate XML structure
82. 🏗 Automate Building & Publishing 👩🔬 Automate Tests 🔐 Automate Security CI Solutions
84. “But why is it taking so long?”
85. 🏗 Automate Building & Publishing “But why is it taking so long?” ⏰
86. 🏗 Automate Building & Publishing 👩🔬 Automate Tests “But why is it taking so long?” ⏰
87. 🏗 Automate Building & Publishing 👩🔬 Automate Tests 🔐 Automate Security “But why is it taking so long?” ⏰
88. 🏗 Automate Building & Publishing 👩🔬 Automate Tests 🔐 Automate Security “But why is it taking so long?” ⏰
89. 🏗 Automate Building & Publishing 👩🔬 Automate Tests 🔐 Automate Security ⏰ 📦 Caching 🏭 Parallelization Merge Queue 🥷 Shadow Jobs Original Problems Scale Problems
90. This is what we want 12 9 6 3 0 Unit Tests Release R8 UI Tests Unit Tests Release R8 UI tests minutes
91. 12 9 6 3 0 Unit Tests Release R8 UI Tests Unit Tests Release R8 UI tests minutes How do we get to this?
92. 12 9 6 3 0 Unit Tests Release R8 UI Tests Unit Tests Release R8 UI tests minutes How do we get to this?
93. Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker 👩🔬 Unit Tests on CI
94. Cache Dependencies 📦
95. Gradle Caches Build Cache Configuration Cache Dependency Cache Script Cache
96. Gradle Caches Build Cache EXECUTING Configuration Cache Dependency Cache Script Cache
97. Gradle Caches Build Cache EXECUTING Configuration Cache Dependency Cache Script Cache
98. Gradle Caches Build Cache EXECUTING Configuration Cache Dependency Cache Script Cache
99. Gradle Caches Build Cache CONFIGURING EXECUTING Configuration Cache Dependency Cache Script Cache
100. Gradle Caches Build Cache Configuration Cache Dependency Cache Script Cache
101. 💻 Local Support Gradle Caches Build Cache 💻 💻 💻 Configuration Cache Dependency Cache Script Cache Local Support Local Support Local Support
102. 💻 Local Support Gradle Caches Build Cache 💻 💻 💻 Configuration Cache Dependency Cache Script Cache Local Support Local Support Local Support
103. Gradle Cache directories ~/.gradle ├── caches │ ├── 8.1 │ ├── 8.2 │ ├── ⋮ │ ├── buildcache—1 │ ├── jars-9 │ ├── journal-1 │ ├── modules-2 │ └── transforms-3 │ ├── ⋮
104. Gradle Cache directories ~/.gradle ├── caches │ ├── 8.1 │ ├── 8.2 │ ├── ⋮ │ ├── buildcache—1 │ ├── jars-9 │ ├── journal-1 │ ├── modules-2 │ └── transforms-3 │ ├── ⋮ Dependency Cache
105. Gradle Cache directories ~/.gradle ├── caches │ ├── 8.1 │ ├── 8.2 │ │ ├── generated-gradle-jars │ │ └── kotlin-dsl │ ├── ⋮ │ ├── buildcache—1 │ ├── jars-9 │ ├── journal-1 │ ├── modules-2 │ └── transforms-3 │ Dependency Cache Script Cache
106. ~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/modules-2 ci-cache.yml
107. 💻 Local Support Gradle Caches Build Cache 💻 💻 💻 Configuration Cache Dependency Cache Script Cache Local Support Local Support Local Support
108. Gradle Caches Build Cache 💻 Configuration Cache Dependency Cache Script Cache Local Support 💻 Local Support Remote Support Remote Support
109. Control Cache build dependencies 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker
110. Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change Cache build dependencies minutes cache save gradle execution gradle configuration cache restore git Docker
111. Gradle Properties Prioritizing quick one-liner quick wins None of these are defaults
112. # If you don't turn these on you aren't # getting any performance wins from # Gradle Build Cache or modularization. org.gradle.caching=true org.gradle.parallel=true gradle.properties
113. # Kotlin 1.7+ only # Good reason to upgrade if you are on # an older version. kotlin.incremental.useClasspathSnapshot =true gradle.properties
114. # Incremental compilation with KAPT. # Only if you still use KAPT. kapt.incremental.apt =true gradle.properties
115. # Stop generating BuildConfig Java stubs. # If this is the last Java in your codebase # you could see a decent performance win. android.enableBuildConfigAsBytecode =true gradle.properties
116. # Stop generating BuildConfig files # if you don't use them. android.defaults.buildfeatures.buildconfig =false gradle.properties
117. # Experimental flag for parallelizing # R8 work - only useful if you have # multiple R8 tasks. android.r8.maxWorkers =2 gradle.properties
118. # Summary org.gradle.caching=true org.gradle.parallel=true kotlin.incremental.useClasspathSnapshot=true kapt.incremental.apt=true android.enableBuildConfigAsBytecode=true android.defaults.buildfeatures.buildconfig=false android.r8.maxWorkers=2 gradle.properties
119. Gradle Properties
120. Gradle Properties Other engineers ask me, "Where do you find these?" github.com/androidx/androidx and other OSS projects
121. Gradle Plugins & Tweaks
122. Gradle Plugins & Tweaks Gradle Cache Fix Plugin Disable LiveLiterals if you use Compose
123. # Re-renables Android Cache fix plugin for currently disabled defaults we believe are safe for a speed boost. systemProp.org.gradle.android.cache- fix.MergeSourceSetFolders.caching.enabled=true systemProp.org.gradle.android.cache- fix.ZipMergingTask.caching.enabled=true gradle.properties
124. android { composeOptions { useLiveLiterals = project.providers.gradleProperty("useLiveLiterals") .get() .toBooleanStrict() } } build.gradle.kts
125. android { composeOptions { useLiveLiterals = false } } build.gradle.kts
126. Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change Cache build dependencies minutes cache save gradle execution gradle configuration cache restore git Docker
127. Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change Cache build dependencies minutes cache save gradle execution gradle configuration cache restore git Docker
128. 🏭 Parallelization
129. 🏭 Parallelization
130. Overhead Cost 📦 📦 📦 📦 🏭 Parallelization
131. Overhead Cost 📦 📦 📦 📦 Depending on how you organize your CI jobs these costs can become significant. 🏭 Parallelization
133. “Main is failing even though we passed CI”
135. Changed Shared Behavior Changed CI Checks Added “regression” when run in future builds
136. Merge Queue
137. Cache Build Steps 📦
138. Gradle Caches Build Cache 💻 Configuration Cache Dependency Cache Script Cache Local Support 💻 Local Support Remote Support Remote Support
139. Gradle Caches Build Cache 💻 Configuration Cache Dependency Cache Script Cache Local Support 💻 Local Support Remote Support Remote Support
140. Gradle Build Cache - Cacheable Tasks
141. Gradle Build Cache - Modularize • Reduce the number of inputs per task and make them more stable • Watch Siggi’s talk about graph theory • Gradle build scans to observe performance
142. Gradle Remote Cache • Do it: https://docs.gradle.com/build-cache-node/ • 90 Days, 300GB, use validation scripts • Most local and CI runs will benefit from build cache hits • Free
143. Gradle Enterprise
145. • Provides support for features beyond remote cache • Really helps figure out failures, cache misses, and performance issues across a team • Looking forward to using test prediction
146. Gradle Caches Build Cache 💻 Configuration Cache Dependency Cache Script Cache Local Support 💻 Local Support Remote Support Remote Support
147. Gradle Caches Build Cache 💻 Configuration Cache Dependency Cache Script Cache Remote Support Local Support Remote Support Remote Support
148. Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker Gradle Build Cache Optimization
149. Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker Gradle Build Cache Optimization
150. Is it worth it to use the latest JDK
151. Is it worth it to use the latest JDK Yes, 19+
152. Control JDK 11 -> JDK 19 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker
153. Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker JDK 11 -> JDK 19
154. Upgrade Tooling
155. Upgrade Tooling “Will JDK 19
 work on our project?”
156. Upgrade Tooling “Will version X of dependency Y work on our project?”
157. 🥷 Shadow Jobs “Will version X of dependency Y work on our project?”
158. 🥷 Shadow Jobs Run nightly checks against your project with latest snapshots of new versions. “Will version X of dependency Y work on our project?” Zac Sweers
159. 🥷 Shadow Jobs Run nightly checks against your project with latest snapshots of new versions. “Will version X of dependency Y work on our project?” Zac Sweers
160. JVM Argument Tuning
161. JVM Argument Tuning DISCLAIMER: I am not accountable for any of these settings. If they don’t work for you there are so many reasons why your setup would handle them differently.
162. Garbage Collection Algo Tradeoff 🗑
163. Garbage Collection Algo Tradeoff Each JVM GC algorithm makes a choice about what they optimize for and how much.
164. Garbage Collection Algo Tradeoff
165. Garbage Collection Algo Tradeoff Throughput Transactions per second
166. Garbage Collection Algo Tradeoff Throughput Latency Transactions per second Time spent in GC pauses
167. Garbage Collection Algo Tradeoff Throughput Footprint Latency Transactions per second Time spent in GC pauses Rough memory footprint size
168. Garbage Collection Algo Tradeoff Throughput Footprint Latency
169. Garbage Collection Algo Tradeoff Throughput Footprint Latency
170. Garbage Collection Algo Tradeoff Throughput Footprint Latency
171. Garbage Collection Algo Tradeoff Throughput Footprint Latency ParallelGC G1GC
172. Garbage Collection Algo Tradeoff Throughput Footprint Latency ParallelGC G1GC
173. Garbage Collection Algo Tradeoff Throughput Footprint Latency ParallelGC G1GC
174. ParallelGC Garbage Collection Algo Tradeoff G1GC
175. ParallelGC Garbage Collection Algo Tradeoff G1GC • Optimize for speed • Maximum footprint • Might have longer GC pauses • Balance throughput • Lower footprint • Shorter GC pauses
176. ParallelGC Garbage Collection Algo Tradeoff G1GC • Optimize for speed • Maximum footprint • Might have longer GC pauses • Balance throughput • Lower footprint • Shorter GC pauses Use when resource constrained Use when resources allow
177. kotlin.daemon.jvm.options When passing arguments, follow these rules: Use the minus sign - only before the arguments Xmx, XX:MaxMetaspaceSize, and XX:ReservedCodeCacheSize. Separate arguments with commas (,) without spaces. Arguments that come after a space will be used for the Gradle daemon, not for the Kotlin daemon. Kotlin JVM Arguments
178. kotlin.daemon.jvm.options When passing arguments, follow these rules: Use the minus sign - only before the arguments Xmx, XX:MaxMetaspaceSize, and XX:ReservedCodeCacheSize. Separate arguments with commas (,) without spaces. Arguments that come after a space will be used for the Gradle daemon, not for the Kotlin daemon. Kotlin JVM Arguments
179. kotlin.daemon.jvm.options When passing arguments, follow these rules: Use the minus sign - only before the arguments Xmx, XX:MaxMetaspaceSize, and XX:ReservedCodeCacheSize. Separate arguments with commas (,) without spaces. Arguments that come after a space will be used for the Gradle daemon, not for the Kotlin daemon. Kotlin JVM Arguments
180. kotlin.daemon.jvmargs You can add the kotlin.daemon.jvmargs property in the gradle.properties fi le: kotlin.daemon.jvmargs=-Xmx1500m -Xms=500m Kotlin JVM Arguments
181. kotlin.daemon.jvmargs You can add the kotlin.daemon.jvmargs property in the gradle.properties fi le: kotlin.daemon.jvmargs=-Xmx1500m -Xms=500m Using spaces at the beginning will cause all arguments to be ignored Kotlin JVM Arguments
182. Xmx (max) JVM Heap Size Xms (min) • Amount of memory it takes to compile your biggest task / module. • For most projects this is going to be your main app module’s minification step. • Somewhere between 80 and 100% of Xmx. • Heap only grows when GC happens, so increasing it avoids this penalty.
183. JVM Heap Size & Modularization are related
184. MaxMetaspaceSize Metaspace CompressedClassSpaceSize • class & non-class space • unlimited by default • virtual size of compressed classes • defaults to 1GB
185. Metaspace https://stuefe.de/posts/metaspace/analyze-metaspace-with-jcmd/
186. CompressedOops Metaspace CompressedClassPointers • Defaults on < 30GB heap • Uses 4 fewer bytes • Restricts memory heap to 32GB • Available if CompressedOops is on
187. What if we turn off CompressedClassSpace + CompressedOops? 20 15 10 5 0 On Off On Off minutes
188. Metaspace CompressedOops & unlimited MaxMetaspace is going to be better for majority of Android projects.
189. Metaspace CompressedOops & unlimited MaxMetaspace is going to be better for majority of Android projects. (so stick with the defaults)
190. Metaspace ~1GB Metaspace helps avoid early Metaspace-related GCs
191. Measuring Metaspace jcmp <pid> VM.metaspace
192. CodeCache Where the JVM stores its bytecode into compiled native code
193. CodeCache Where the JVM stores its bytecode into compiled native code
194. CodeCache ReservedCodeCacheSize is the maximum amount for cached bytecode
195. CodeCache So we gave ReservedCodeCacheSize 512mb
196. CodeCache To determine what number you should use, measure via –XX:+PrintCodeCache
197. Measuring CodeCache In JVM args: –XX:+PrintCodeCache
198. SoftRefLRUPolicyMSPerMB The number of milliseconds to hold onto soft references per MB of heap size, default 1000
199. SoftRefLRUPolicyMSPerMB The number of milliseconds to hold onto soft references per MB of heap size, default 1000 = 2h 16m 32s 8GB Heap @ 1000 MS Policy
200. SoftRefLRUPolicyMSPerMB The number of milliseconds to hold onto soft references per MB of heap size, default 1000 = 2h 16m 32s 8GB Heap @ 1000 MS Policy 8GB Heap @ 10 MS Policy = 0h 1m 22s
201. SoftRefLRUPolicyMSPerMB The number of milliseconds to hold onto soft references per MB of heap size, default 1000 = 2h 16m 32s 8GB Heap @ 1 MS Policy 8GB Heap @ 1000 MS Policy = 0h 0m 8s 8GB Heap @ 10 MS Policy = 0h 1m 22s
202. Gradle Max Workers --max-workers The maximum number of active workers to allow Should probably be number of performance cores, possibly 1-2 less Will not spawn more workers is memory heap requirements cannot be met
203. jemalloc / tcmalloc Memory fragmentation slows down builds, use a better malloc algorithm
204. malloc alternatives Memory fragmentation slows down builds, use a better malloc algorithm
205. malloc alternatives We saw no speed difference with jemalloc, but it did notice 10% less memory usage
206. malloc alternatives Therefore use jemalloc (or tcmalloc) to be able to run more daemons in parallel with less memory pressure.
207. What do we actually pick for JVM args?
208. 🥷 Shadow Jobs 1. Instrument your build with JVM args as parameters 2. Run parameterized tests 3. Profit What do we actually pick for JVM args?
209. • GC Algo = It Depends (try for Parallel) • SoftRefLRUPolicyMSPerMB = 1 • CodeCache = 256MB - 512MB • MaxMetaspace = Some GB or unlimited • Metaspace = 64MB - 1GB • Xmx = 4GB - 8GB • Xms = 3GB - 7GB • Max Workers=Performance Cores - 2 What do we actually pick?
210. • jcmp for Metaspace • –XX:+PrintCodeCache for CodeCache • Measure CPU & Memory resource usage, lots of tools available JVM Measurement Tools
211. Control JVM Optimizations 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker
212. Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker JVM Optimizations
213. Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker JVM Optimizations
214. Gradle Caches Build Cache Configuration Cache Dependency Cache Script Cache Remote Support Remote Support Remote Support 💻 Local Support
215. Gradle Caches Build Cache Configuration Cache Dependency Cache Script Cache Remote Support Remote Support Remote Support 💻 Local Support
216. # Enables the configuration cache: org.gradle.configuration-cache=true org.gradle.configuration-cache.problems=warn org.gradle.configuration-cache.inputs.unsafe.ignore.file-system- checks=(see github.com/androidx/androidx repo gradle.properties) gradle.properties
217. Gradle Cache directories ~/.gradle ├── caches │ └── ⋮ Dependency Cache Script Cache
218. Gradle Cache directories ~/.gradle ├── caches │ └── ⋮ ~/project ├── .gradle ├── app ├── build ├── gradle └── ⋮ Dependency Cache Script Cache
219. Gradle Cache directories ~/project ├── .gradle ├── app ├── build ├── gradle └── ⋮
220. Gradle Cache directories ~/project ├── .gradle │ │ │ │ ├── app ├── build ├── gradle └── ⋮
221. Gradle Cache directories ~/project ├── .gradle │ ├── 8.1 │ ├── 8.2 │ ├── ⋮ │ └── configuration-cache ├── app ├── build ├── gradle └── ⋮
222. ~/project ├── .gradle │ ├── 8.1 │ ├── 8.2 │ ├── ⋮ │ └── configuration-cache ├── app ├── build ├── gradle └── ⋮ Configuration Cache Gradle Cache directories
223. ~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/modules-2 .gradle/<< gradle_version >> .gradle/configuration-cache ci-cache.yml
224. As of Gradle 8.1 encryption was added. Asking Gradle support - configuration cache on CI is unsupported Gradle Configuration Cache on CI
225. As of Gradle 8.1 encryption was added. Gradle's response has been that configuration cache on CI is not supported Gradle Configuration Cache on CI
226. As of Gradle 8.1 encryption was added. Gradle's response has been that configuration cache on CI is not supported. But we persisted anyway... Gradle Configuration Cache on CI
227. Gradle Configuration Cache on CI Environment Variable Environment Variables Prefixed File system Entry File System property Value from custom source 35 1 2007 4 52 5 build/reports/configuration-cache/4tnz..i67j/27o6..b0jl/configuration-cache-report.html We looked at the reports for all inputs
228. But despite everything 100% of inputs being consistent we continued to get an unhelpful error message: Gradle Configuration Cache on CI
229. Gradle Configuration Cache on CI Calculating task graph as no configuration cache is available for tasks
230. Gradle Configuration Cache on CI “You need to cache the encryption key.” — Inaki Villar
231. Gradle Configuration Cache on CI Caveat to this method - you will be caching any secrets used by this build, so either: • Only use this for builds that do not rely on secrets • Knowingly store this with the security risk
233. Gradle Configuration Cache on CI
234. Gradle User Home directory ~/.gradle ├── caches │ ├── 8.1 │ │ … │ ├── 8.4 │ │ └── cc-keystore │ │ ├── cache.properties │ │ ├── cc-keystore.lock │ │ └── gradle.keystore │ ├── │ ├── buildcache—1 │ ├── jars-9 │ ├── journal-1 Configuration Cache Encryption
235. ~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/modules-2 .gradle/<< gradle_version >> .gradle/configuration-cache ci-cache.yml
236. ~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/<< gradle_version >>/cc-keystore ~/.gradle/caches/modules-2 .gradle/<< gradle_version >> .gradle/configuration-cache ci-cache.yml
237. Gradle Configuration Cache on CI Calculating task graph as no configuration cache is available for tasks
238. unit_tests: docker: - image: circleci/android:api-34-node steps: - run: name: Clean Gradle Cache command: | find . -type f -name '*.lock' -delete find . -type f -name 'gc.properties' -delete find ~/.gradle -type f -name '*.lock' -delete find ~/.gradle -type f -name 'gc.properties' -delete when: always - save_cache: paths: - ~/.gradle/caches/<< parameters.gradle_version >>/ generated-gradle-jars - ~/.gradle/caches/<< parameters.gradle_version >>/ kotlin-dsl - .gradle/configuration-cache - ~/.gradle/caches/<< parameters.gradle_version >>/ cc-keystore - .gradle/<< parameters.gradle_version >> config.yml
239. unit_tests: docker: - image: circleci/android:api-34-node steps: - run: name: Clean Gradle Cache command: | find . -type f -name '*.lock' -delete find . -type f -name 'gc.properties' -delete find ~/.gradle -type f -name '*.lock' -delete find ~/.gradle -type f -name 'gc.properties' -delete when: always - save_cache: paths: - ~/.gradle/caches/<< parameters.gradle_version >>/ generated-gradle-jars - ~/.gradle/caches/<< parameters.gradle_version >>/ kotlin-dsl - .gradle/configuration-cache - ~/.gradle/caches/<< parameters.gradle_version >>/ cc-keystore - .gradle/<< parameters.gradle_version >> config.yml 🤦
240. Gradle Configuration Cache on CI Calculating task graph as configuration cache cannot be reused because an input to task ‘:gradle-plugin:generateExternalPluginSpecBuilders' has changed.
241. Gradle Configuration Cache on CI Calculating task graph as configuration cache cannot be reused because an input to task ‘:gradle-plugin:generateExternalPluginSpecBuilders' has changed.
242. “Gradle has compilation avoidance for non- ABI changes in included builds, but due to a bug any inline function would cause all build scripts to be recompiled even when there is a non-ABI change.” - Nicklas Ansman Gradle Configuration Cache on CI
243. ~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/<< gradle_version >>/cc-keystore ~/.gradle/caches/modules-2 .gradle/<< gradle_version >> .gradle/configuration-cache gradle-plugin/build/pluginDescriptors ci-cache.yml
244. Gradle Configuration Cache on CI Could not load the value of field `patterns` of `org.gradle.configurationcache.serialization.codecs.DirectoryTree Spec` bean found in field `fileSystemInputs` of `org.gradle.configurationcache.fingerprint.ConfigurationCacheFing erprint$WorkInputs` bean found in Gradle runtime. > Failed to instrument class org/jetbrains/kotlin/gradle/tasks/ KotlinCompile$ScriptFilterSpec in ClassLoaderScopeIdentifier.Id{coreAndPlugins:settings[:gradle- plugin]:buildSrc[:gradle-plugin]:root-project[:gradle-plugin] (export)}
245. Gradle Configuration Cache on CI Could not load the value of field `patterns` of `org.gradle.configurationcache.serialization.codecs.DirectoryTree Spec` bean found in field `fileSystemInputs` of `org.gradle.configurationcache.fingerprint.ConfigurationCacheFing erprint$WorkInputs` bean found in Gradle runtime. > Failed to instrument class org/jetbrains/kotlin/gradle/tasks/ KotlinCompile$ScriptFilterSpec in ClassLoaderScopeIdentifier.Id{coreAndPlugins:settings[:gradle- plugin]:buildSrc[:gradle-plugin]:root-project[:gradle-plugin] (export)}
246. ~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/<< gradle_version >>/cc-keystore ~/.gradle/caches/modules-2 .gradle/<< gradle_version >> .gradle/configuration-cache gradle-plugin/build/pluginDescriptors ci-cache.yml
247. ~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/<< gradle_version >>/cc-keystore ~/.gradle/caches/modules-2 .gradle/<< gradle_version >> .gradle/configuration-cache gradle-plugin/build ci-cache.yml
248. ~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/<< gradle_version >>/cc-keystore ~/.gradle/caches/modules-2 ~/.gradle/caches/jars-9 ~/.gradle/caches/transforms-3 .gradle/<< gradle_version >> .gradle/configuration-cache gradle-plugin/build ci-cache.yml
249. Reusing configuration cache. Gradle Configuration Cache on CI
250. Gradle Configuration Cache on CI
251. Gradle Configuration Cache on CI
252. “Wow, I was prepared for a faster PR time but that was blazing” - Jacob Duron Gradle Configuration Cache on CI
253. Gradle Caches Build Cache Configuration Cache Dependency Cache Script Cache Remote Support Remote Support Remote Support 💻 Local Support
254. Gradle Caches Build Cache Configuration Cache Dependency Cache Script Cache Remote Support Remote Support Remote Support Remote Support
255. Control Gradle Configuration Cache Reuse 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker
256. Control Gradle Configuration Cache Reuse 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker
257. 40 30 20 10 0 Gradle Configuration Cache Reuse Control Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker
258. Overhead Cost Optimizations 📦 git clone pulls 400 MB big images (2.5 GB) Minutes to save cache ~60 seconds to restore cache
259. image size 2560 2048 1536 1024 512 Circle/ Android 0 JRE Bare megabytes Android
260. download & extract time 75 60 45 30 15 0 seconds Circle/ Android JRE Bare Android
261. download & extract time 75 60 45 30 15 0 seconds Circle/ Android JRE Bare Android
262. 500 400 300 200 100 0 default no blobs no trees shallow megabytes clone & checkout size
263. 75 60 45 30 15 0 seconds default no blobs no trees shallow clone & checkout time
264. 75 60 45 30 15 0 seconds default no blobs no trees shallow clone & checkout time
265. Why did we go with treeless if its not the fastest? Shallow is faster, but it comes with a heavy penalty for any additional git operations clone & checkout time
266. prune 40 30 20 10 0 Control Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker
267. Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes prune cache save gradle execution gradle configuration cache restore git Docker
268. 40 30 20 10 0 Control Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes clone optimization cache save gradle execution gradle configuration cache restore git Docker
269. 40 30 20 10 0 Control Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes clone optimization cache save gradle execution gradle configuration cache restore git Docker
270. 40 30 20 10 0 Control Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes Present Day Unit Test Runs cache save gradle execution gradle configuration cache restore git Docker
271. Present Day CI Runs 12 9 6 3 0 Unit Tests Release R8 UI Tests Unit Tests Release R8 UI tests minutes
272. CI Speed Optimizations Cache Dependencies Cache Build Tasks Cache Configuration JVM Tuning Compiler Flags 📦 treeless or shallow clone prune images 🏋 📦 Cache Optimizations 🚩
273. CI Speed Optimizations Cache Dependencies Cache Build Tasks Cache Configuration JVM Tuning Compiler Flags 📦 treeless or shallow clone prune images 🏋 📦 Cache Optimizations 🚩 🚧 ✅ ✅ ✅ ✅ Future Talk
274. Topics not covered 👩💻 Local Dev Environment 🏭 Parallelization 📸 Automatic Screenshot Testing
275. Metaspace Citations Thomas Stüfe Blog https://stuefe.de/posts/metaspace/what-is-metaspace/ https://stuefe.de/posts/metaspace/metaspace-architecture/ https://stuefe.de/posts/metaspace/what-is-compressed-class-space/ https://stuefe.de/posts/metaspace/sizing-metaspace/ https://stuefe.de/posts/metaspace/analyze-metaspace-with-jcmd/ Oracle Class Metadata, PermGen and changes to native memory allocation
276. Gradle Documentation Gradle Files & Directories https://docs.gradle.org/current/userguide/directory_layout.html Gradle CI Ephemeral Best Practices https://docs.gradle.org/current/userguide/ dependency_resolution.html#sub:ephemeral-ci-cache Gradle Recommends using Daemons for CI https://docs.gradle.org/current/userguide/ gradle_daemon.html#continuous_integration
277. Kotlin JVM Args Documentation New Method https://kotlinlang.org/docs/gradle-compilation-and-caches.html#kotlin- daemon-jvmargs-property Deprecated Method https://kotlinlang.org/docs/gradle-compilation-and-caches.html#kotlin- daemon-jvm-options-system-property
278. JDK Version Documentation JDK Release Notes Summaries for GC Algo Changes 19 | 20 | 21 JVM SoftReferences Tuning CodeCache
279. JVM SoftReferences JVM SoftReferences
280. JVM CodeCache Tuning CodeCache
281. Shadow Jobs by Zac Sweers https://slack.engineering/shadow-jobs/