Beyond Modularization: Scaling your Android Build with Gradle

Beyond Modularization: Scaling your Android Build with Gradle

Tips and tricks on how to scale your Android build with Gradle.
How to measure performance improvements and gain insights using Gradle Enterprise.

https://gradle.com/enterprise/trial/

Video of this talk is available here: https://vimeo.com/430648415

5f69045a2ca496221cfc624405917cdf?s=128

Nelson Osacky

June 18, 2020
Tweet

Transcript

  1. Beyond Modularization: Scaling Android Builds Nelson Osacky

  2. Me • Previously Android Engineer • Large projects • SoundCloud

    • Square • Small startups • Solutions Engineer at Gradle
  3. Me • Gradle Plugin Maintainer • Fladle - Easily Scale

    Instrumentation Tests on Firebase https://github.com/runningcode/fladle • Delect - Replace Dagger with Dagger Reflect https://github.com/soundcloud/delect/ • Gradle Doctor - Actionable Insights for your build https://github.com/runningcode/gradle-doctor
  4. Solutions Engineer https://gradle.com/enterprise/trial/

  5. None
  6. Gradle Build Tool Gradle Enteprise

  7. Android Gradle Plugin Android Studio

  8. Kotlin Intellij

  9. Android Development Feature Development Tech Debt (There are other things

    too)
  10. Tech Debt Refactoring Build Speed

  11. Build Speed is Tech Debt And it always pays off

  12. Cost of Builds 60s waste * 50 builds / day

    * 50 devs = 42 hours lost / day
  13. Cost of Builds 60s waste * 50 builds / day

    * 50 devs = 42 hours lost / day not including lost focus https://gradle.com/roi-calculator
  14. Fast Builds Matter

  15. Cost of Builds 60s waste * 50 builds / day

    * 50 devs = 42 hours lost / day hire 5 new people without paying them! no recruiting https://gradle.com/roi-calculator
  16. Build Speed is Tech Debt And it always pays off

    And is easy to justify working on it
  17. Motivated? Where do we start?

  18. None
  19. Update All The Things ⬢ Gradle 6.5 ⬢ Android Gradle

    Plugin 4.0.0 ⬢ Gradle Enterprise Plugin 3.3.4 ⬢ Kotlin 1.3.72 ⬢ Third party plugins ⬢ Third party libraries New Performance APIs Caching Improvements Task Configuration Avoidance Background Scan Uploads Incremental Annotation Processors Compiler Perf Improvements
  20. Update All The Things plugins { id "com.github.ben-manes.versions" version "0.28.0"

    } ./gradlew dependencyUpdates -Drevision=release https://github.com/ben-manes/gradle-versions-plugin
  21. None
  22. Build Lifecycle •Initialization •Configuration •Execution

  23. Build Lifecycle Initialization Sets up the environment for the build

    and determines which projects will take part in it.
  24. Build Lifecycle Configuration All build scripts of all projects are

    executed. Constructs and configures the task graph.
  25. Build Lifecycle Execution Runs the tasks graph in order.

  26. Where to start?

  27. Check for Red Flags

  28. What is a red flag?

  29. If nothing changes, no tasks should execute.

  30. ./gradlew assembleDebug ./gradlew assembleDebug BUILD SUCCESSFUL in 2s 58 actionable

    tasks: 58 up-to-date
  31. ./gradlew assembleDebug ./gradlew assembleDebug BUILD SUCCESSFUL in 5s 359 actionable

    tasks: 3 executed, 356 from cache
  32. Check for Red Flags Regularly

  33. BUILD SUCCESSFUL in 5s 359 actionable tasks: 3 executed, 356

    up-to-date Publishing build scan... https://gradle.com/s/yj5jtd4kh6ucc Build Analyzer results available
  34. BUILD SUCCESSFUL in 5s 359 actionable tasks: 3 executed, 356

    up-to-date Publishing build scan... https://gradle.com/s/yj5jtd4kh6ucc Build Analyzer results available Gradle Build Scans Android Studio Build Analyzer
  35. Android Studio Build Analyzer

  36. Run a build scan ./gradlew assembleDebug --scan https://scans.gradle.com/

  37. None
  38. Filter

  39. None
  40. None
  41. 9 tasks ran Bugsnag

  42. None
  43. None
  44. #perfmatters

  45. None
  46. None
  47. 1 sec per project is high

  48. None
  49. 25 sec is red flag!

  50. apply plugin: 'witness'

  51. //apply plugin: 'witness'

  52. None
  53. 5.8s to 0.5s by disabling two plugins!

  54. Configuration time savings affect every build

  55. Configuration time should be roughly 5s per 100 projects

  56. Disable plugins for local builds

  57. engBuild pattern

  58. engBuild != buildVariant

  59. engBuild == Gradle property https://docs.gradle.org/current/userguide/build_environment.html#sec:gradle_configuration_properties

  60. enabled by default ext.engBuild = (project.findProperty('engBuild') ?: 'true') .toBoolean()

  61. if (!engBuild) { apply plugin: 'com.google.firebase.crashlytics' apply plugin: 'com.google.firebase.firebase-perf' }

  62. ./gradlew assembleDebug ./gradlew assembleRelease -PengBuild=false disable on CI or testing

    release builds
  63. engBuild=false disable on CI or testing release builds gradle.properties

  64. engBuild pattern

  65. 16627 tasks is a red flag

  66. Use Task Configuration Avoidance https://docs.gradle.org/current/userguide/task_configuration_avoidance.html

  67. api vs implementation

  68. Module A Module B Module C dependencies { implementation project('module-b')

    implementation project('module-c') } dependencies { implementation project('module-c') }
  69. Module A Module B Module C dependencies { implementation project('module-b')

    } dependencies { api project('module-c') } module-c must always used when consuming module-b
  70. api vs implementation convenience vs performance

  71. Real World Example

  72. https://github.com/k9mail/k-9 K9 Mail App

  73. K9 Mail App https://github.com/k9mail/k-9/blob/master/app/core/build.gradle#L12 apiaproject(':plugins:openpgp-api-lib:openpgp-api')

  74. K9 Mail App https://github.com/k9mail/k-9/blob/master/app/core/build.gradle#L12 apiaproject(':plugins:openpgp-api-lib:openpgp-api')

  75. apiaproject(':plugins:openpgp-api-lib:openpgp-api') K9 Mail App

  76. if (project.hasProperty('useImpl')) { implementation project(':plugins:openpgp-api-lib:openpgp-api') } else { apiaproject(':plugins:openpgp-api-lib:openpgp-api') }

    K9 Mail App
  77. ./gradlew :app:k9mail:assembleDebug ./gradlew :app:k9mail:assembleDebug -PuseImpl K9 Mail App

  78. Profile it!

  79. https://github.com/gradle/gradle-profiler Gradle Profiler

  80. Let's write scenarios Gradle Profiler

  81. change_api_dep { } performance.scenarios Gradle Profiler

  82. change_api_dep { title = "Change using api dependency" } performance.scenarios

    Gradle Profiler
  83. change_api_dep { title = "Change using api dependency" tasks =

    ["app:k9mail:assembleDebug"] } performance.scenarios Gradle Profiler
  84. change_api_dep { title = "Change using api dependency" tasks =

    ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/ openintents/openpgp/OpenPgpApiManager.java" } performance.scenarios Gradle Profiler
  85. change_api_dep { title = "Change using api dependency" tasks =

    ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/ openintents/openpgp/OpenPgpApiManager.java" } performance.scenarios Gradle Profiler
  86. change_api_dep { title = "Change using api dependency" tasks =

    ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } change_impl_dep { title = "Change using impl dependency" tasks = ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } performance.scenarios Gradle Profiler
  87. change_impl_dep { title = "Change using impl dependency" tasks =

    ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/ openintents/openpgp/OpenPgpApiManager.java" } performance.scenarios Gradle Profiler
  88. change_impl_dep { title = "Change using impl dependency" tasks =

    ["app:k9mail:assembleDebug"] gradle-args = ["-PuseImpl"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/ openintents/openpgp/OpenPgpApiManager.java" } performance.scenarios Gradle Profiler
  89. change_impl_dep { title = "Change using impl dependency" tasks =

    ["app:k9mail:assembleDebug"] gradle-args = ["-PuseImpl"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/ openintents/openpgp/OpenPgpApiManager.java" } performance.scenarios Gradle Profiler
  90. change_api_dep { title = "Change using api dependency" tasks =

    ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } change_impl_dep { title = "Change using impl dependency" tasks = ["app:k9mail:assembleDebug"] gradle-args = ["-PuseImpl"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } performance.scenarios Gradle Profiler
  91. change_api_dep { title = "Change using api dependency" tasks =

    ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } change_impl_dep { title = "Change using impl dependency" tasks = ["app:k9mail:assembleDebug"] gradle-args = ["-PuseImpl"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } performance.scenarios Gradle Profiler
  92. gradle-profiler --benchmark --scenario-file performance.scenarios Gradle Profiler

  93. 1.stop daemon 2.clean 3.6 warm up builds 4.10 benchmark builds

    Gradle Profiler
  94. None
  95. html report and csv report

  96. Build Time (milliseconds) 0 3250 6500 9750 13000 #1 #2

    #3 #4 #5 #6 #7 #8 #9 #10 Change using api dependency Change using impl dependency
  97. scenario Change using api dependency Change using impl dependency mean

    11603.0 10151.7 median 11529.5 10049.5 confidence 0.0 99.98429477157690
  98. ~15% improvement

  99. why?

  100. gradle-profiler --profile buildscan --scenario-file performance.scenarios

  101. gradle-profiler --profile buildscan --scenario-file performance.scenarios * Results written to /Users/no/workspace/k-9/profile-out-14

    Scenario Change using api dependency using Gradle 6.5 - Build scan for measured build #1: https://gradle.com/s/ozev5czodllsk Scenario Change using impl dependency using Gradle 6.5 - Build scan for measured build #1: https://gradle.com/s/rxujhuyxqlybu
  102. https://scans.gradle.com/s/ozev5czodllsk/timeline?outcomeFilter=SUCCESS api Dependency

  103. https://scans.gradle.com/s/rxujhuyxqlybu/timeline?outcomeFilter=SUCCESS implementation Dependency

  104. Which tasks ran?

  105. https://scans.gradle.com/s/ozev5czodllsk/timeline?outcomeFilter=SUCCESS api Dependency

  106. https://scans.gradle.com/s/rxujhuyxqlybu/timeline?outcomeFilter=SUCCESS implementation Dependency

  107. https://scans.gradle.com/s/ozev5czodllsk/timeline?outcomeFilter=SUCCESS api Dependency

  108. https://scans.gradle.com/s/ozev5czodllsk/timeline?outcomeFilter=SUCCESS api Dependency

  109. What exactly is the difference?

  110. Enterprise Feature Alert

  111. Capture Task Input Files https://docs.gradle.com/enterprise/gradle-plugin/#capturing_task_input_files

  112. gradleEnterprise { buildScan { captureTaskInputFiles = true } } https://docs.gradle.com/enterprise/gradle-plugin/#capturing_task_input_files

  113. Build Scan Comparison

  114. Build Scan Comparison

  115. Build Scan Comparison Build B uses api

  116. Adding unnecessary inputs to tasks.

  117. How do we fix it?

  118. https://github.com/autonomousapps/dependency-analysis-android-gradle-plugin Dependency Analysis Plugin plugins { id "com.autonomousapps.dependency-analysis" version "0.49.0"

    }
  119. None
  120. Annotation Processors

  121. Make sure they are incremental!

  122. w: [kapt] Incremental annotation processing requested, but support is disabled

    because the following processors are not incremental: androidx.room.RoomProcessor (NON_INCREMENTAL).
  123. android { ... defaultConfig { ... javaCompileOptions { annotationProcessorOptions {

    arguments += [ "room.schemaLocation":"$projectDir/schemas".toString(), "room.incremental":"true"] } } } }
  124. Room

  125. android { ... defaultConfig { ... javaCompileOptions { annotationProcessorOptions {

    arguments += [ "room.schemaLocation":"$projectDir/schemas".toString(), "room.incremental":"true"] } } } }
  126. "room.schemaLocation":"$projectDir/schemas".toString(), Remote Cache miss!

  127. "room.schemaLocation":"$projectDir/schemas".toString(), Remote Cache miss! https://developers.soundcloud.com/blog/gradle-remote-build-cache-misses

  128. Move Room to it's own Room https://github.com/gradle/android-cache-fix-gradle-plugin Module

  129. None
  130. Disabling unused AGP Features

  131. Cost of Generating BuildConfig All 186 are UP-TO-DATE.

  132. Unused so Proguard will remove it later

  133. BuildConfig adds java sources to pure kotlin modules

  134. Which BuildConfig?

  135. Which version code?

  136. Which version code?

  137. VERSION_CODE is 1 by default in library modules

  138. Aidl what?

  139. Also adds task configuration time.

  140. None
  141. android.defaults.buildfeatures.buildconfig=false android.defaults.buildfeatures.aidl=false android.defaults.buildfeatures.renderscript=false android.defaults.buildfeatures.resvalues=false android.defaults.buildfeatures.shaders=false gradle.properties New in AGP 4.0.0

    https://developer.android.com/studio/releases/gradle-plugin#buildFeatures
  142. Disable for all projects

  143. Enable when needed

  144. android.buildFeatures { buildConfig = true } app/build.gradle

  145. Code Multipliers

  146. AndroidX Jetifier

  147. Migrates support-library- dependent libraries to rely on Android X packages.

  148. android.enableJetifier=true

  149. None
  150. android.enableJetifier=false

  151. Dependency Substitution https://issuetracker.google.com/issues/156449751

  152. Stuck on the Jetifier?

  153. plugins { id "com.github.plnice.canidropjetifier" version "0.5" } Can I drop

    Jetifier? ./gradlew -Pandroid.enableJetifier=false canIDropJetifier
  154. Jetify At Home ./jetifier-standalone -i <source-library> -o <output-library> https://developer.android.com/studio/command-line/jetifier

  155. Jetify At Home Upload to nexus

  156. Flavors

  157. Flavor Explosion

  158. None
  159. Flavor Explosion When you add a build dimension multiplying build

    speed in a surprising surprise.
  160. Flavor Explosion BuildType * BuildFlavor = BuildVariants

  161. Flavor Explosion BuildType * BuildFlavor = BuildVariants BuildVariant are code

    multipliers
  162. Flavor Explosion 2 BuildTypes * 3 BuildFlavors = 6 Variants

    6x code 6x tasks 6x resources 6x cache utilization
  163. def prop = project.findProperty("enabledBuildVariant") ?: "freeDebug" android.variantFilter { variant ->

    if (variant.name != prop) { variant.ignore = true } } ./gradlew assemblePaidDebug -PenableBuildVariant=paidDebug
  164. Flavor Explosion • Mostly a configuration time win. • Permanently

    disable flavors you don't need.
  165. How to cut build times in half

  166. How to cut build times in half.

  167. apply plugin: 'com.android.library'

  168. assembleDebug assembleRelease

  169. 2x code 2x resources 2x tasks 2x tests !!!!!

  170. How often does the debug and release code differ for

    library modules?
  171. ./gradlew test runs all tests twice

  172. ./gradlew testDebug java module tests don't run

  173. Recompile everything when switching to release

  174. CI takes twice as long

  175. if (engBuild) { } Disable unused flavors in library modules

  176. if (engBuild) { android.variantFilter { variant -> } } Disable

    unused flavors in library modules
  177. if (engBuild) { android.variantFilter { variant -> if (variant.name ==

    'release') { variant.ignore = true } } } Disable unused flavors in library modules
  178. if (engBuild) { android.variantFilter { variant -> if (variant.name ==

    'release') { variant.ignore = true } } } Disable unused flavors in library modules
  179. if (engBuild) { android.variantFilter { variant -> if (variant.name ==

    'release') { variant.ignore = true } } } Disable unused flavors in library modules
  180. if (engBuild) { android.variantFilter { variant -> if (variant.name ==

    'debug') { variant.ignore = true } } } Disable unused flavors in library modules
  181. android.variantFilter { variant -> if (variant.name == 'debug') { variant.ignore

    = true } } Disable unused flavors in library modules android.debug.matchingFallbacks = ['release'] in application module
  182. android.variantFilter { variant -> if (variant.name == 'debug') { variant.ignore

    = true } } Disable unused flavors in library modules
  183. Build and test times cut in half

  184. Switching to release build doesn't recompile everything

  185. Disable stuff! #protip

  186. Disable stuff! #protip measure!

  187. ./gradlew help Measure configuration time

  188. Disable debug variant Disable debug variant and AGP features Disable

    AGP features Standard build 0.0 200.0 400.0 600.0 800.0 Avg Configuration Time (ms) Average of 10 builds help using Gradle Profiler
  189. 783ms to 688ms ~= 13% faster

  190. 13% configuration time improvement that scales

  191. Regular build assembleDebug

  192. assembleDebug Debug disabled, AGP features disabled

  193. Disable More Things https://github.com/android/gradle-recipes

  194. engBuild Ideas

  195. engBuild Ideas •Higher minSdk •Disables Multidex •Disable coreLibraryDesugaring •Strip other

    languages •Strip large images •Strip out native libraries •Try new versions of AGP https://medium.com/@runningcode/testing-new-versions-of-the-android-gradle-plugin-ea80df978316
  196. Always Measure!

  197. Upcoming Performance Enhancements

  198. File System Watching

  199. Fast input snapshotting

  200. File System Watching https://blog.gradle.org/introducing-file-system-watching org.gradle.unsafe.watch-fs=true

  201. Configuration Caching

  202. Configuration Caching

  203. Configuration Caching The build scripts of all projects which are

    part of the build are executed.
  204. ext.engBuild = (project.findProperty('engBuild') ?: 'true') .toBoolean()

  205. ext.engBuild = providers.gradleProperty("engBuild") .forUseAtConfigurationTime() .orElse("true") .map { it.toBoolean() }

  206. if (!engBuild) { apply plugin: 'com.google.firebase.crashlytics' apply plugin: 'com.google.firebase.firebase-perf' }

  207. if (!engBuild.get()) { apply plugin: 'com.google.firebase.crashlytics' apply plugin: 'com.google.firebase.firebase-perf' }

  208. Improving Android Build Performance Devoxx Belgium 2018 Cédric Champeau https://www.youtube.com/watch?v=StahHZ-L83k

    https://speakerdeck.com/melix/improving-android-build-performance
  209. Build Bigger, Better: Gradle for Large Projects Google I/O 2019

    Aurimas Liutikas and Xavier Ducrohet https://www.youtube.com/watch?v=sQC9-Rj2yLI
  210. The Secrets of the Build Scan Plugin and the Internals

    of Gradle Virtual Android Makers Paris 2020 Me https://www.youtube.com/watch?v=lgaqS0pmUzk
  211. Measuring remote build cache performance https://medium.com/@runningcode https://github.com/runningcode/gradle-doctor

  212. Thank you! Need advice? osacky.com https://gradle.com/enterprise/trial/ https://speakerdeck.com/runningcode/beyond-modularization-scaling-your-android-build-with-gradle