Gradle build: The time is now

Transcript

1. GRADLE BUILD: The time is now Alexey Bykov
2. 2 Alexey Bykov Senior Software Engineer 6 years Google Developer Expert @nonewsss
3. Dad what's that dark place over there?
4. Goals Feel comfortable about
5. Goals Feel comfortable about Improve build time
6. 6 Build time. Why?
7. 7 Such sample not applies to Reddit or any other company I may worked at Build time. Why? ~ 10 engineers ~ 12 m ~ 20 builds (daily)
8. 8 Such sample not applies to Reddit or any other company I may worked at Build time. Why? ~ 1000 builds ~ 200 h Weekly ~ 10 engineers ~ 12 m ~ 20 builds (daily)
9. 9 ~ 200 h
10. 10 Gradle starter
11. 11 Gradle starter Tasks Plugins Projects Build Lifecycle Caching Properties
12. 12 Gradle starter Tasks Plugins Projects Build Lifecycle Caching Properties
13. 13 What is a build?
14. 14 Build Sequence of steps to transform 
 source code to a final product Step Step Step binary Src
15. 15 Steps == Tasks Task Task Task binary Src
16. 16 Tasks can be asynchronous Task Task Task binary Src
17. 17 Input & output Task
18. 18 Input & output Task @Input @InputFiles Input
19. 19 Input & output Task @Input @TaskAction @InputFiles Input
20. 20 Input & output Task @Input @OutputDirectory @TaskAction @InputFiles Input Output
21. 21 JavaCompile Java Version Example
22. 22 Example JavaCompile Java Version Source code
23. 23 Example JavaCompile Java Version Source code Compiler args Class files
24. 24 Tasks can be dependant Task Task Task Input Output Output Output
25. 25 Tasks can be cacheable Task @CacheableTask Input Output
26. 26 Tasks can be cacheable Task Input Output Cache @CacheableTask
27. 27 Tasks labels
28. 28 BUILD SUCCESSFUL in 6m 49s 7960 actionable tasks: 130 executed, 371 from cache, 7459 up-to-date Tasks labels
29. 29 BUILD SUCCESSFUL in 6m 49s 7960 actionable tasks: 130 executed, 371 from cache, 7459 up-to-date Tasks labels
30. 30 UP-TO-DATE Incremental cache 
 applied
31. 31 FROM CACHE Build cache applied
32. 32 SKIPPED Excluded explicitly 
 or has predicate
33. 33 NO SOURCE Can’t be executed, 
 as there is no code
34. 34 EXECUTED (NO LABEL) Task run its action
35. 35 Task sources — Build script (kts/groovy) — Build Src — External (like plugin)
36. 36 Gradle starter Tasks Plugins Projects Build Lifecycle Caching Properties
37. 37 Project — Main Gradle API — Set of the tasks — 1 to 1 dependency with build.gradle
38. 38 Gradle starter Tasks Plugins Projects Build Lifecycle Caching Properties
39. 39 Plugins — Configuration of settings and tasks
40. 40 Plugins — Configuration of settings and tasks — Example: * Android Gradle Plugin (AGP) * Kotlin Gradle Plugin (KGP)
41. 41 Gradle starter Tasks Plugins Projects Build Lifecycle Caching Properties
42. 42 Build Lifecycle 1. Configuration What to do? (Configure tasks)
43. 43 1. Configuration What to do? (Configure tasks) Build Lifecycle 2. Execution Tasks
44. 44 Gradle starter Tasks Plugins Projects Build Lifecycle Caching Properties
45. Caching Gradle 
 Daemon JVM process
46. Caching Gradle 
 Daemon JVM process * Long-running process
47. Caching Gradle 
 Daemon JVM process * Long-running process * Enabled by default
48. Caching Gradle 
 Daemon JVM process * Long-running process * Enabled by default * Not stable on the CI
49. Caching Gradle 
 Daemon JVM process * Long-running process * Enabled by default * Not stable on the CI org.gradle.daemon=false #gradle.properties
50. Caching Gradle 
 Daemon JVM process In memory
51. Caching Configuration 
 cache
52. Caching Configuration 
 cache * Experimental
53. Caching Configuration 
 cache * Experimental * project/.gradle/configuration-cache
54. Caching Configuration 
 cache * Experimental * project/.gradle/configuration-cache * Works based on input/output 
 principle
55. Caching Gradle 
 Daemon JVM process Configuration 
 cache In memory In project
56. Caching Incremental 
 cache
57. Caching Incremental 
 cache * Short-term cache
58. Caching Incremental 
 cache * Short-term cache * Stores in build folders
59. Caching Incremental 
 cache * Short-term cache * Can be easily invalidated * Stores in build folders
60. Caching In memory In project Gradle 
 Daemon JVM process Configuration 
 cache Incremental 
 cache
61. Caching Build 
 cache
62. Caching Build 
 cache * Long-term cache
63. Caching Build 
 cache * Long-term cache * ~.gradle/caches/build-cache1/
64. Caching
65. 65 Example JavaCompile Java Version Source code Compiler args Class files
66. 66 Example $hash $hash $hash Java Version Source code Compiler args
67. 67 Example $hash $hash $hash CacheKey
68. 68 Example $hash $hash $hash CacheKey ffb08d41a85389bbbe6e8bbe480f0560
69. Build cache
70. 70 Example $hash $hash $hash CacheKey Build 
 cache
71. 71 Example $hash $hash $hash CacheKey Build 
 cache
72. 72 Example $hash $hash $hash CacheKey Build 
 cache JavaCompile Class files
73. Build cache Each CacheKey in build-cache 
 is a compressed byte-code
74. Caching Local In memory In project Out project Gradle 
 Daemon JVM process Configuration 
 cache Incremental 
 cache Build 
 cache
75. Caching Dependency 
 cache
76. Caching Dependency 
 cache * .jars or *.aars
77. Caching Dependency 
 cache * .jars or *.aars ~./gradle/cache/modules-2
78. Caching In memory In project Out project Gradle 
 Daemon JVM process Configuration 
 cache Incremental 
 cache Build 
 cache Dependency 
 cache
79. Caching Local In memory In project Out project Gradle 
 Daemon JVM process Configuration 
 cache Incremental 
 cache Build 
 cache Dependency 
 cache
80. Caching Local In memory In project Out project Gradle 
 Daemon JVM process Configuration 
 cache Incremental 
 cache Build 
 cache Dependency 
 cache Remote
81. Caching Local In memory In project Out project Gradle 
 Daemon JVM process Configuration 
 cache Incremental 
 cache Build 
 cache Dependency 
 cache Build 
 cache Remote
82. Remote caching Build 
 cache Stores on remote server
83. Remote caching Build 
 cache Stores on remote server Copies to the local cache
84. Caching Local In memory In project Out project Gradle 
 Daemon JVM process Configuration 
 cache Incremental 
 cache Build 
 cache Dependency 
 cache Build 
 cache Remote
85. Caching Local In memory In project Out project Gradle 
 Daemon JVM process Configuration 
 cache Incremental 
 cache Build 
 cache Dependency 
 cache Build 
 cache Proxy Remote
86. Jitpack was down for ~2 days https://github.com/jitpack/jitpack.io/issues/5337
87. Remote Caching Proxy Your own maven
88. Remote Caching Proxy Your own maven Which syncs data
89. Caching Local In memory In project Out project Gradle 
 Daemon JVM process Configuration 
 cache Incremental 
 cache Build 
 cache Dependency 
 cache Build 
 cache Proxy Remote
90. 90 Cache:Conslusion
91. 91 Incremental applied Incremental cache Input Action
92. 92 Incremental applied Incremental cache Input Action UP-TO-DATED
93. 93 Incremental applied Incremental cache Input Action
94. 94 Build cache applied Incremental cache Input Action Build Cache FROM-CACHE
95. 95 No cache Incremental cache Input Action Build Cache
96. 96 No cache Incremental cache Input Build Cache NO LABEL Action
97. 97 Gradle starter Tasks Plugins Projects Build Lifecycle Caching Properties
98. 98 Properties File to easily configure Gradle
99. 99 Properties File to easily configure Gradle Can be: ~./gradle.properties Project/gradle.properties
100. 100 Properties File to easily configure Gradle Can be: ~./gradle.properties Project/gradle.properties Gradle merge them
101. 101 Properties File to easily configure Gradle Can be: ~./gradle.properties Project/gradle.properties Gradle merge them Global > Local
102. 102 Measure!
104. 104 Measure! Locally?
105. 105 Measure! Locally? * Manually? * Automatically?
106. 106 Measure! Locally? * Manually? * Automatically? CI?
107. 107 Measure! Locally? * Manually? * Automatically? CI? * Incremental?
108. 108 Measure! Locally? * Manually? * Automatically? CI? * Incremental? * Clean build?
109. 109 Measure! Locally? * Manually? * Automatically? CI? * Incremental? * Clean build? Where to send?
110. 110 Measure! Locally? * Manually? * Automatically? CI? * Incremental? * Clean build? Where to send? HOW???
111. 111 1. Locally
112. 112 1. Locally ~ daily, on a laptop
113. 113 1. Locally ~ daily, on a laptop ~ incremental build
114. 114 1. Locally ~ daily, on a laptop ~ incremental build ~ incremental changes
115. 115 1. Locally ~ daily, on a laptop ~ incremental build ~ incremental changes ~ clean build
116. 116 1. Locally ~ daily, on a laptop ~ incremental build ~ incremental changes ~ clean build ~ configuration
117. 117 1. Locally ~ daily, on a laptop ~ incremental build ~ incremental changes ~ clean build ~ no external monitor ~ configuration
118. 118 1. Locally ~ daily, on a laptop ~ incremental build ~ incremental changes ~ clean build ~ no external monitor ~ no android studio ~ configuration
119. 119 1. Locally ~ daily, on a laptop ~ incremental build ~ incremental changes ~ clean build ~ no external monitor ~ no android studio ~ cli ~ configuration
120. 120 Collect automatically
121. 121 Collect automatically ~ Only if you have > 20 engineers
122. 122 Collect automatically ~ Only if you have > 20 engineers ~ Tool: Talailot https://github.com/cdsap/Talaiot
123. 123 2. CI
124. 124 2. CI Do it together with local measurement
125. 125 3. How?
126. 126 3. How? ~ Build Scan
127. 127 3. How? ~ Build Scan Most powerful Good visualisation Shares build outside
128. 128 3. How?
129. 129 3. How? ~ Build Scan ~ Gradle profiler
130. 130 Gradle profiler https://github.com/gradle/gradle-profiler ~ brew install gradle-profiler
131. 131 Scenario no_op { tasks = [“assembleDebug"] } #incremental
132. 132 Scenario #build cache clean_no_op { tasks = ["clean", "assembleDebug"] } no_op { tasks = [“assembleDebug"] } #incremental
133. 133 Scenario #build cache clean_no_op { tasks = ["clean", "assembleDebug"] } no_op { tasks = [“assembleDebug"] } #incremental incremental_changes { tasks = ["assembleDebug"] apply-abi-change-to = “sample/src/main/…TeamsViewModel.kt” }
134. 134 Scenario no_op { tasks = [“assembleDebug"] } clean_no_op { tasks = ["clean", "assembleDebug"] } incremental_changes { tasks = ["assembleDebug"] apply-abi-change-to = "sample/src/main/java/com/alexbykov/oknetwork/data/TeamsViewModel.kt" } #incremental #build cache #incremental changes
135. 135 Scenario gradle-profiler --benchmark \ 
 --scenario-file profiler-scenario
136. 136 Scenario gradle-profiler --benchmark \ 
 --scenario-file profiler-scenario \ --measure-config-time
137. 137 Output
138. 138 CSV
139. 139 Configuration time
140. 140 After ~ Parse it
141. 141 After ~ Parse it ~ Send it to somewhere..
142. 142 Somewhere ~ Influx db
143. 143 Somewhere ~ Influx db ~ Elastic search
144. 144 Somewhere ~ Influx db ~ Elastic search ~ Graphite …… or…
145. 145 Google sheet
146. 146 Google sheet + Forms ~ Link sheet with forms
147. 147 Google sheet + Forms ~ Link sheet with forms ~ Know form ids and send shorturl.at/mzNOY
148. 148 Let’s improve!
149. 149 Disable Jetifier
150. 150 Disable Jetifier ~ appsupport -> androidx
151. 151 Disable Jetifier ~ appsupport -> androidx ~ ./gradlew checkJetifier
152. 152 gradle.properties android.enableJetifier=false
153. 153 Non-Transitive R classes
154. 154 Problem app feature1 Feature3 feature2
155. 155 Problem app feature1 Feature3 feature2 R class R class R class R class
156. 156 Content merging app feature1 Feature3 feature2 R class (app, f1,f2,f3) R class (f2, f3) R class (f2, f3) R class (f1)
157. 157 Non-transitive app feature1 Feature3 feature2 R class (f2) R class (f3) R class (app) R class (f1)
158. 158 Non-transitive ~ R.strings -> YourModule.R.strings ~ All resources
159. 159 gradle.properties android.nonTransitiveRClass=true shorturl.at/gK567
160. 160 Virtual file system
161. 161 Virtual file system shorturl.at/bzIJO org.gradle.vfs.watch=true
162. 162 Don’t use buildSrc folder
163. 163 ~ KTS & Dependency Most common use-case
164. 164 Problems ~ Incremental cache invalidated
165. 165 ~ Build cache invalidated Problems ~ Incremental cache invalidated shorturl.at/eruJM NO LABEL
166. 166 Version Catalog shorturl.at/oxY02
167. 167 Version Catalog ~ No cache issues
168. 168 Version Catalog ~ No cache issues ~ .toml file (can be standalone)
169. 169 Version Catalog ~ No cache issues ~ Bundling dependencies ~ .toml file (can be standalone)
170. 170 Version Catalog [versions] groovy = "3.0.5" checkstyle = "8.37"
171. 171 Version Catalog [libraries] groovy-core = { module = "org.codehaus.groovy:groovy", version.ref = "groovy" } groovy-json = { module = "org.codehaus.groovy:groovy-json", version.ref = "groovy" } [versions] groovy = "3.0.5" checkstyle = "8.37"
172. 172 Bundling [bundles] groovy = ["groovy-core", "groovy-json", "groovy- nio"]
173. 173 Bundling [bundles] groovy = ["groovy-core", "groovy-json", "groovy-nio"] implementation libs.groovy
174. 174 Focus plugin
175. 175 Focus plugin shorturl.at/hioEM ~ Allows to manage settings.gradle
176. 176 Focus plugin ~ Allows to manage settings.gradle ~ Mostly useful in monorepo shorturl.at/hioEM
177. 177 Java version
178. 178 Java version ~ Have Java 8? —> Java 11
179. 179 Java version ~ Have Java 8? —> Java 11 ~ Have Java 11 —> A/B tests & experiments
180. 180 Tune gradle properties
181. 181 Tune gradle properties ~ Always do A/B tests with GC -XX:+UseParallelGC -XX:+UseG1GC (default)
182. 182 Tune gradle properties ~ Always do A/B tests with GC ~ A/B tests with heap size org.gradle.jvmargs=-Xmx..G kotlin.daemon.jvmargs=-Xmx..g
183. 183 Tune gradle properties ~ Always do A/B tests with GC ~ A/B tests with heap size ~ Choose proper one for you 1. Kotlin daemon > Gradle Daemon 2. Kotlin daemon < Gradle Daemon 3. Kotlin daemon == Gradle Daemon
184. 184 Some ideas to test android.disableAutomaticComponentCreation=true android.defaults.buildfeatures.buildconfig=false android.defaults.buildfeatures.aidl=false android.defaults.buildfeatures.renderscript=false android.defaults.buildfeatures.resvalues=false android.defaults.buildfeatures.shaders=false
185. 185 Never apply new properties without testing!
186. 186 kapt —> KSP
187. 187 kapt —> KSP ~25% No stubs generation
188. 188 KSP Support ~ Room ~ Moshi ~ Glide ~ Hilt ~ Dagger2 ~ Anvil
189. 189 Hilt problem
190. 190 Hilt problem ~ Works based on subcomponents ~ Subcomponents generates 
 in parent component
191. 191 Modularisation
192. 192 Modularisation ~ api/impl
193. 193 Modularisation ~ api/impl ~ Restrict merging to app module
194. 194 https://github.com/danger/danger Danger!
195. 195 Parallel
196. 196 Parallel org.gradle.parallel=true
197. 197 Parallel org.gradle.parallel=true Workers count == CPU cores
198. 198 Apply configuration cache
199. 199 Configuration cache org.gradle.unsafe.configuration-cache=true org.gradle.unsafe.configuration-cache-problems=warn shorturl.at/buUVZ
200. 200 Configuration time 3 minutes ~10 seconds Before After
201. 201 Project isolation (future)
202. 202 Project isolation (future) shorturl.at/gyJRV org.gradle.unsafe.isolated-projects=true
203. 203 Build Cache
204. 204 Build Cache org.gradle.caching=true
205. 205 Remote build cache shorturl.at/LQT17
206. 206 Remote build cache Remote Build 
 cache Local machine Mr CI
207. 207 Remote build cache Remote Build 
 cache Local machine Mr CI Read/W rite
208. 208 Remote build cache Remote Build 
 cache Local machine Mr CI Read/W rite Read
209. Preventing cache misses ~ shorturl.at/CMWZ7 ~ shorturl.at/EMNSU ~ shorturl.at/vY389
210. Build cache—fix plugin shorturl.at/ckls3
211. 211 Gradle doctor plugin shorturl.at/abBOS
212. 212 Gradle best practices shorturl.at/cjEQX
213. 213 Other ~ Extract native to libraries ~ Don’t use flavors and build variants ~ Always test an impact before merging ~ Use latest tools ~ Keep things under control
214. 214 Gradle enterprise https://gradle.com/roi-calculator/
215. 215 Official guides shorturl.at/dfhw2 shorturl.at/EGKQ4
216. 216 Build time with all of this ~12 minutes ~2 minutes Before After
217. 217 Learn Gradle https://gradle.com/training/ ~ Trainings: https://blog.gradle.org/how-gradle-works-1 ~ Gradle blog: ~ Jendrik Johannes: https://www.youtube.com/@jjohannes
218. 218 Slides @nonewsss