Make Your Build Great Again (DroidConSF 2017)

Slow builds have been plaguing Android development since the very beginning, especially for large multi-dex projects. As libraries tend to grow in size and the more libraries an application consumes it will slow down the build, especially when an application goes over the mutli-dex limit. Libraries aren't the only thing that can slow down the build, adding many Gradle plugins and repositories can increase the time it takes to configure the Gradle build. This talk will be centered around how I was able to decrease Yammer for Android's Gradle build times by optimizing our use of the Android Gradle plugin and the Gradle setup of our multi-project build and will give several tools and tips on how to help you profile and decrease your build times as well.

Video: https://www.youtube.com/watch?v=rvwAlbtbtmM

Transcript

1. Jared Burrows Make Your Build Great Again
2. Motivation • Yammer is a social networking service used for communication within an organization • Current debug builds are currently multidex • More features and more code leads to slower builds • More libraries leads to slower builds
3. Results Faster CI builds, Merge pull requests faster, Iterate faster Before After Reduction Time(min) ~18+ ~11 38.89%
4. What can we optimize? • Software
 - Gradle
 - Android Gradle Plugin
 - Android Studio/Intellij • Hardware
 - Increase CPU
 - Increase Memory

5. What to look for • Cause of the build slowness • Identify bottlenecks during start up and between tasks • Find and remove unnecessary dependencies/plugins • Module and build.gradle structure
6. Where to start • Start from the bottom of your tooling • Better developer laptops and/or build agents • Optimize and configure Gradle • Optimize and configure Android Gradle Plugin • Optimize and configure Android Studio
7. Optimizing your Gradle setup
8. Keep your software up to date • Try to use the latest version of Gradle - 4.3 • Ensures compatibility between minor versions • Make transitioning to the next major version easier • Use Gradle with the latest major version of JVM - 1.8
9. gradle.properties
10. Enable the Gradle daemon • Each time you run Gradle, it kicks off a new instance of Gradle • Use the daemon to avoid the cost of JVM startup time for each build • Starting in 3.0+, the Daemon is on by default!
11. gradle.properties org.gradle.daemon=true Example “gradle.properties” file usage
12. Increase daemon’s heap size • By default Gradle will reserves 1GB of heap space • Android projects usually need more memory • As of AGP 2.1+, you want at least 2GB for “Dex in Process” • Giving more memory will help dramatically speed up builds for larger projects
13. gradle.properties, cont. org.gradle.daemon=true Example “gradle.properties” file usage
14. gradle.properties, cont. org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m Example “gradle.properties” file usage
15. Enable parallel builds • Execute Gradle tasks in parallel to speed up builds • Helpful for multi-module builds that are decoupled • JavaCompile tasks - Compile in separate process • Test tasks - Run tests in separate process
16. gradle.properties, cont. org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m Example “gradle.properties” file usage
17. gradle.properties, cont. org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true Example “gradle.properties” file usage
18. JavaCompile tasks separate process tasks.withType(JavaCompile) { options.fork = true } Configure all “JavaCompile” tasks to fork
19. Running tests in separate process tasks.withType(Test) { def maxCount = gradle.startParameter.maxWorkerCount maxParallelForks = (maxCount < 2) ? 1 : maxCount / 2 forEvery = 100 } Configure all “Test” tasks to fork every 100 tests
20. Running tests in separate process, cont. tasks.withType(Test) { maxParallelForks = Runtime.runtime.availableProcessors().intdiv(2) ?: 1 forkEvery = 100 } Configure all “Test” tasks to fork every 100 tests
21. Running tests in separate process, cont. $ gradlew testDebug > Task :testDebugUnitTest <============-> 97% EXECUTING [17s] > :testDebugUnitTest > 17 tests completed > :testDebugUnitTest > Executing test com.example.SomeTest 
 BUILD SUCCESSFUL in 17s 28 actionable tasks: 26 executed, 2 up-to-date Unit tests running in 1 worker process
22. Running tests in separate process, cont. $ gradlew testDebug > Task :testDebugUnitTest <============-> 97% EXECUTING [17s] > :testDebugUnitTest > 17 tests completed > :testDebugUnitTest > Executing test com.example.SomeTest1 > :testDebugUnitTest > Executing test com.example.SomeTest2 > :testDebugUnitTest > Executing test com.example.SomeTest3 > :testDebugUnitTest > Executing test com.example.SomeTest4 BUILD SUCCESSFUL in 10s 28 actionable tasks: 2 executed, 26 up-to-date Unit tests running in 4 parallel workers processes
23. Enable configure on demand • Only configure projects needed based on the specified task • For projects with many subprojects, it avoids unnecessary configuration, saving time • Works best for projects that have many decoupled modules • Using “subprojects”/“allprojects” removes these benefits since they are executed eagerly
24. gradle.properties, cont. org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true Example “gradle.properties” file usage
25. gradle.properties, cont. org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true org.gradle.configureondemand=true Example “gradle.properties” file usage
26. Enable build cache • Incremental builds will helps avoid work that is already done • When switching branches, you are forced to rebuild over again but local build cache remembers this previous results • Extra beneficial when build different product flavors for Android builds (even different task names, reuse between flavors) • When working on a team, you might want to consider Gradle’s remote build cache, so you can share build outputs with developers
27. gradle.properties, cont. org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true org.gradle.configureondemand=true Example “gradle.properties” file usage
28. gradle.properties, cont. org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true org.gradle.configureondemand=true org.gradle.caching=true Example “gradle.properties” file usage
29. build.gradle
30. Apply plugins judiciously • Similar to adding dependencies, only add them to the modules that need them • Try not to use “subprojects” or “allprojects” block to apply plugins unless all of the subprojects actually use them! • Limiting the scope of applying your plugins help speed up builds
31. Avoid plugins that block/slow configuration • Check for plugins that resolve/configure slowly • Applying plugins + evaluating build scripts + running “afterEvaluate” = Configuration time • From Stefan Oehme’s talk, “Android Performance Pitfalls”, at Gradle Summit 2017, he mentioned that having a configuration time of less than 1 second or less is good youtube.com/watch?v=TGUwevlaNNU
32. Avoid plugins that block/slow configuration $ gradlew > Task :help Welcome to Gradle 4.3. To run a build, run gradle <task> ... To see a list of available tasks, run gradle tasks To see a list of command-line options, run gradle --help To see more detail about a task, run gradle help --task <task> BUILD SUCCESSFUL in 0s 1 actionable task: 1 executed Normal execution time
33. Avoid plugins that block/slow configuration buildscript { repositories { google() maven { url "https://plugins.gradle.org/m2/" } } dependencies { classpath "com.android.tools.build:gradle:3.0.0" classpath "some:slow-plugin:1.0.0" } } apply plugin: "com.android.application" apply plugin: "slow-plugin" Example of a slow plugin slowing down the configuration
34. Avoid plugins that block/slow configuration buildscript { repositories { google() maven { url "https://plugins.gradle.org/m2/" } } dependencies { classpath "com.android.tools.build:gradle:3.0.0" classpath "some:slow-plugin:1.0.0" } } apply plugin: "com.android.application" apply plugin: "slow-plugin" Example of a slow plugin slowing down the configuration
35. Avoid plugins that block/slow configuration buildscript { def isCi = rootProject.hasProperty("ci") repositories { google() maven { url "https://plugins.gradle.org/m2/" } } dependencies { classpath "com.android.tools.build:gradle:3.0.0" classpath "some:slow-plugin:1.0.0" } } apply plugin: "com.android.application" apply plugin: "slow-plugin" Use a property to check if we are on the CI
36. Avoid plugins that block/slow configuration buildscript { def isCi = rootProject.hasProperty("ci") repositories { google() maven { url "https://plugins.gradle.org/m2/" } } dependencies { classpath "com.android.tools.build:gradle:3.0.0" if (isCi) classpath "some:slow-plugin:1.0.0" } } apply plugin: "com.android.application" if (isCi) apply plugin: "slow-plugin" Use the property to disable the plugin from being applied locally
37. Minimize repository count • Most popular repositories include: • jcenter() • mavenCentral() • maven { url "https://plugins.gradle.org/m2/" } • google() (Gradle 4+) or maven { url "https://maven.google.com" } • maven { url "https://jitpack.io" }
38. Minimize repository count allprojects { repositories { jcenter() mavenCentral() maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } } Most likely using too many repositories here!
39. Minimize repository count • jcenter() and mavenCentral() - may host the same dependencies • google() - only provides Android and Google dependencies • maven { url "https://plugins.gradle.org/m2/" } - hosts Gradle plugins and mirrors dependencies from jcenter() • maven { url "https://jitpack.io" } - should be used sparingly as it is used to grab a specific branch or commit of a particular open source repository
40. Minimize repository count allprojects { repositories { jcenter() mavenCentral() maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } } What to remove?
41. Minimize repository count allprojects { repositories { jcenter() mavenCentral() maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } } What to remove?
42. Minimize repository count repositories { jcenter() mavenCentral() maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } What to remove?
43. Minimize repository count What to remove? repositories { jcenter() mavenCentral() maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } }
44. Minimize repository count What to remove? repositories { jcenter() maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } }
45. Minimize repository count What to remove? repositories { jcenter() maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } }
46. Minimize repository count What to remove? repositories { maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } }
47. Minimize repository count What to remove? repositories { maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } }
48. Minimize repository count Minimal as possible for this example! repositories { maven { url "https://plugins.gradle.org/m2/" } google() }
49. Avoid dynamic dependencies (+) • Can cause unexpected version updates • Difficulty resolving version differences • Slower builds caused by always checking for updates
50. Avoid dynamic dependencies dependencies { compile "com.android.support:appcompat-v7:+" } Don’t: Use dynamic dependencies
51. Avoid dynamic dependencies Don’t: Use dynamic dependencies dependencies { compile "com.android.support:appcompat-v7:+" }
52. Avoid dynamic dependencies • Instead of guessing or grabbing the latest dependency via (+), let’s find the latest release • Using Ben Mane’s Gradle Version plugin, you can find the latest plugin and dependencies versions • Instead of having to search for each dependency individually
53. Avoid dynamic dependencies $ gradlew dependencyUpdates > Task :dependencyUpdates ------------------------------------------------------------ : Project Dependency Updates (report to plain text file) ------------------------------------------------------------ The following dependencies are using the latest milestone version: - com.android.support:appcompat-v7:27.0.0 - com.github.ben-manes:gradle-versions-plugin:0.17.0 Generated report file build/dependencyUpdates/report.txt BUILD SUCCESSFUL in 0s 1 actionable task: 1 executed github.com/ben-manes/gradle-versions-plugin
54. Avoid dynamic dependencies dependencies { compile "com.android.support:appcompat-v7:+" } Don’t: Use dynamic dependencies
55. Avoid dynamic dependencies Do: Use static dependencies dependencies { compile "com.android.support:appcompat-v7:27.0.0" }
56. Avoid dynamic dependencies For dependencies to be certain versions configurations.all { resolutionStrategy.force "com.android.support:support-annotations:27.0.0" }
57. Avoid unnecessary/unused dependencies • For mobile builds, try not to add large dependencies • Scan for and remove extra/unused dependencies that maybe sitting in the “compile”/“implementation” configurations • Determine which libraries that you are not using heavily or should not be on mobile (eg. Guava, Jackson instead of Gson) • My talk called, “The Road to Single DEX” goes into more detail
58. Enable incremental builds • Allow Gradle to resources already compiled resources to make the builds even faster • It only compiles only those java classes that were changed or that are dependencies to the changed classes
59. Incremental builds tasks.withType(JavaCompile) { options.fork = true } From our previous JavaCompile example
60. Incremental builds tasks.withType(JavaCompile) { options.incremental = true options.fork = true } JavaCompile with incremental builds and forking
61. Incremental builds • However, libraries such as AutoValue, Glide, Butterknife, Dagger that use annotation processing disable incremental builds! • In AGP 3.0+, I believe support for annotation processing and incremental builds are a priority
62. Incremental builds JavaCompile with incremental builds and forking tasks.withType(JavaCompile) { options.incremental = true options.fork = true }
63. Incremental builds tasks.withType(JavaCompile) { options.incremental = true options.fork = true doFirst { println "Task ${path} annotation processors: ${effectiveAnnotationProcessorPath.asPath}" } } JavaCompile tasks will print annotation processors
64. Optimizing Your AGP Setup
65. Keep your software up to date • Try to use the latest version of Android Gradle Plugin - 3.0.0 • Keep the SDK tools and platform tools updated to the latest revisions • By keeping the SDK up to date, the Android Gradle Plugin can utilize the latest tools to optimize your build(speed, performance and bug fixes)
66. Removing unnecessary flavors android { buildTypes { debug {} release { minifyEnabled = true } } flavorDimensions "dim" productFlavors { nightly { dimension "dim" } prod { dimension "dim" } } } Example show buildTypes with productFlavors
67. Removing unnecessary flavors $ gradlew tasks > Task :tasks Build tasks ----------- assemble - Assembles all variants of all applications and secondary packages. assembleAndroidTest - Assembles all the Test applications. assembleDebug - Assembles all Debug builds. assembleNightly - Assembles all Nightly builds. assembleProd - Assembles all Prod builds. assembleRelease - Assembles all Release builds.
 BUILD SUCCESSFUL in 0s 1 actionable task: 1 executed Produces 4 tasks - Debug/Nightly/Prod/Release
68. Removing unnecessary flavors def isNightly = project.hasProperty("nightly") android { buildTypes { debug { if (isNightly) { minifyEnabled = true applicationIdSuffix = ".nightly" } else { applicationIdSuffix = ".debug" } } release { minifyEnabled = true } } } Exact same example with no productFlavors. Use “gradlew -Pnightly” for “nightly”.
69. Removing unnecessary flavors $ gradlew tasks > Task :tasks Build tasks ----------- assemble - Assembles all variants of all applications and secondary packages. assembleAndroidTest - Assembles all the Test applications. assembleDebug - Assembles all Debug builds. assembleRelease - Assembles all Release builds.
 BUILD SUCCESSFUL in 0s 1 actionable task: 1 executed Produces 2 tasks - Debug/Release
70. Removing unnecessary flavors android { buildTypes { debug {} release { minifyEnabled = true } } flavorDimensions "dim" productFlavors { nightly { dimension "dim" } prod { dimension "dim" } } variantFilter { variant -> def names = variant.flavors*.name if (names.contains("nightly")) { variant.ignore = true } } } You can also use “android.variantFilter”. This example removes “nightly”.
71. Avoid compiling unnecessary resources • Use "resConfigs" to filter out localizations that you do not want/support in your app
72. Avoid compiling unnecessary resources Using “resConfigs” to keep English only android { defaultConfig { resConfigs "en" } }
73. Avoid compiling unnecessary resources Using “resConfigs” to keep multiple resources or languages android { defaultConfig {
 resConfigs "en","de","fr"
 }
 }
74. Static constants for builds • Using dynamic variables that change your current build.gradle files or resources will add to the build time • Resources changed on each build prevent Instant Run from performing a code swap • Use static values for debug builds to prevent a full rebuilds of the APK
75. Static constants for builds android { compileSdkVersion 27 buildToolsVersion "27.0.0" defaultConfig { applicationId "burrows.apps.example" versionCode new Date().format("ddMMyyHHmm").toInteger() versionName "1.0" minSdkVersion 19 targetSdkVersion 27 } } On each build, the “versionCode” is updated with a new value
76. Static constants for builds android { compileSdkVersion 27 buildToolsVersion "27.0.0" defaultConfig { applicationId "burrows.apps.example" versionCode new Date().format("ddMMyyHHmm").toInteger() versionName "1.0" minSdkVersion 19 targetSdkVersion 27 } } On each build, the “versionCode” is updated with a new value
77. Static constants for builds def isRelease = project.hasProperty("release") android { compileSdkVersion 27 buildToolsVersion "27.0.0" defaultConfig { applicationId "burrows.apps.example" versionCode isRelease ? new Date().format("ddMMyyHHmm").toInteger() : 1 versionName "1.0" minSdkVersion 19 targetSdkVersion 27 } } Now, only release builds will be given an updated value
78. Create library modules • Gradle will only compile the modules you modify • Cache those outputs for future builds • Improves effectiveness of “configuration on demand” and “parallel” execution • Particularly helpful when packaging by feature
79. Disable PNG crunching • If you can’t convert your PNGs to WebP, use PNG crunching • By disabling, AGP will not compress the PNGs on each build • Starting in AGP 3.0.0+, PNG crunching is disabled by default for “debug” build types
80. Disable PNG crunching android { aaptOptions { cruncherEnabled = project.hasProperty("ci") } } For AGP <= 2.3.3, use “aaptOptions.cruncherEnabled = false”
81. Avoid legacy multidex • “minSdkVersion” < 21 - legacy mutlidex • “minSdkVersion” >= 21 - native multidex • We can speed up development builds by using “productFlavors” or Gradle properties to toggle the “minSdkVersion” for faster build times
82. Avoid legacy multidex android { compileSdkVersion 27 buildToolsVersion "27.0.0" defaultConfig { applicationId "burrows.apps.example" versionCode 1 versionName "1.0" minSdkVersion 19 targetSdkVersion 27 multiDexEnabled true } } Example that uses legacy multidex
83. Avoid legacy multidex android { compileSdkVersion 27 buildToolsVersion "27.0.0" defaultConfig { applicationId "burrows.apps.example" versionCode 1 versionName "1.0" minSdkVersion 19 targetSdkVersion 27 multiDexEnabled true } } Example that uses legacy multidex
84. Avoid legacy multidex android { compileSdkVersion 27 buildToolsVersion "27.0.0" defaultConfig { applicationId "burrows.apps.example" versionCode 1 versionName "1.0" minSdkVersion rootProject.hasProperty("lollipop") ? 21 : 19 targetSdkVersion 27 multiDexEnabled true } } Example that uses legacy multidex
85. Disable pre-dexing libraries on CI • Java class files need to be converted to DEX files • Helps to speed up the build for incremental builds locally • On a CI server, where you run clean builds, pre-dexing can have the reverse affect, so we can disable it
86. Disable pre-dexing libraries android { dexOptions { preDexLibraries = !project.hasProperty("ci") } } Set “preDexLibraries” to false when on the CI
87. Enable Build Cache • Stores certain outputs that the Android plugin for Gradle generates when building your project (ex. unpackaged AARs, pre-dexed dependencies) • Improves performance of clean builds by reusing cached files • Starting in AGP 2.3.0+, build cache is enabled by default
88. gradle.properties, cont. org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true org.gradle.configureondemand=true org.gradle.caching=true Example “gradle.properties” file usage
89. gradle.properties, cont. # Gradle specific org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true org.gradle.configureondemand=true org.gradle.caching=true Example “gradle.properties” file usage
90. gradle.properties, cont. # Gradle specific org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true org.gradle.configureondemand=true org.gradle.caching=true # Android specific android.enableBuildCache=true Example “gradle.properties” file usage
91. Enable new DEX Compiler • DEX compilation is the process of transforming .class bytecode into .dex bytecode • DX vs D8: compiles faster and outputs smaller .dex files • D8 has the same or better app runtime performance • Starting in AGP 3.0.0+, you can toggle this on to test it • Starting in AGP 3.1.0+, this is enabled by default
92. gradle.properties, cont. # Gradle specific org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true org.gradle.configureondemand=true org.gradle.caching=true # Android specific android.enableBuildCache=true Example “gradle.properties” file usage
93. gradle.properties, cont. # Gradle specific org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true org.gradle.configureondemand=true org.gradle.caching=true # Android specific android.enableBuildCache=true android.enableD8=true Example “gradle.properties” file usage
94. Optimizing Your Android Studio Usage
95. Convert images to WebP • Reduces image sizes by converting to WebP • Does not need to compress at build time(PNG crunching) • By converting beforehand, this will help speed up your build
96. Convert images to WebP Right-click on “drawable” folders, click on “Convert to WebP”
97. Enable Instant Run • Push certain code and resource changes to your running app without building a new APK • Some cases, without even restarting the current activity
98. Enable Instant Run In “Preferences”, under “Build, Execution, Deployment”, turn on “Enable Instant Run”
99. Enable offline mode • If do not need to update dependencies or plugins frequently, you would greatly benefit from compiling offline • This skips dependency resolution and downloading and uses cache only • For command line: “gradlew --offline”
100. Enable offline mode In “Preferences”, under “Build, Execution, Deployment”, turn on “Offline work”
101. What now?
102. Profiling builds • “gradlew --profile” (built in) • “gradlew --scan” (docs.gradle.com/build-scan-plugin) • For more tools: github.com/gradle/gradle-profiler
103. Profiling builds Example of “gradlew clean assembleDebug --profile”
104. Profiling builds Example of “gradlew clean assembleDebug --scan”
105. Results Faster CI builds, Merge pull requests faster, Iterate faster Before After Reduction Time(min) ~18+ ~11 38.89%
106. Thank you! Questions? twitter.com/jaredsburrows github.com/jaredsburrows jaredsburrows@gmail.com jaredsburrows.com