Share
Make Your Build Great Again (DroidConSF 2017)
Slow builds have been plaguing Android development since the very beginning, especially for large multi-dex projects. As libraries tend to grow in size and the more libraries an application consumes it will slow down the build, especially when an application goes over the mutli-dex limit. Libraries aren't the only thing that can slow down the build, adding many Gradle plugins and repositories can increase the time it takes to configure the Gradle build. This talk will be centered around how I was able to decrease Yammer for Android's Gradle build times by optimizing our use of the Android Gradle plugin and the Gradle setup of our multi-project build and will give several tools and tips on how to help you profile and decrease your build times as well.
Video: https://www.youtube.com/watch?v=rvwAlbtbtmM
Jared Burrows
Transcript
106 slides
-
Motivation • Yammer is a social networking service used for
communication within an organization • Current debug builds are currently multidex • More features and more code leads to slower builds • More libraries leads to slower builds -
Results Faster CI builds, Merge pull requests faster, Iterate faster
Before After Reduction Time(min) ~18+ ~11 38.89% -
What can we optimize? • Software - Gradle - Android
Gradle Plugin - Android Studio/Intellij • Hardware - Increase CPU - Increase Memory -
What to look for • Cause of the build slowness
• Identify bottlenecks during start up and between tasks • Find and remove unnecessary dependencies/plugins • Module and build.gradle structure -
Where to start • Start from the bottom of your
tooling • Better developer laptops and/or build agents • Optimize and configure Gradle • Optimize and configure Android Gradle Plugin • Optimize and configure Android Studio -
Keep your software up to date • Try to use
the latest version of Gradle - 4.3 • Ensures compatibility between minor versions • Make transitioning to the next major version easier • Use Gradle with the latest major version of JVM - 1.8 -
Enable the Gradle daemon • Each time you run Gradle,
it kicks off a new instance of Gradle • Use the daemon to avoid the cost of JVM startup time for each build • Starting in 3.0+, the Daemon is on by default! -
Increase daemon’s heap size • By default Gradle will reserves
1GB of heap space • Android projects usually need more memory • As of AGP 2.1+, you want at least 2GB for “Dex in Process” • Giving more memory will help dramatically speed up builds for larger projects -
Enable parallel builds • Execute Gradle tasks in parallel to
speed up builds • Helpful for multi-module builds that are decoupled • JavaCompile tasks - Compile in separate process • Test tasks - Run tests in separate process -
JavaCompile tasks separate process tasks.withType(JavaCompile) { options.fork = true }
Configure all “JavaCompile” tasks to fork -
Running tests in separate process tasks.withType(Test) { def maxCount =
gradle.startParameter.maxWorkerCount maxParallelForks = (maxCount < 2) ? 1 : maxCount / 2 forEvery = 100 } Configure all “Test” tasks to fork every 100 tests -
Running tests in separate process, cont. tasks.withType(Test) { maxParallelForks =
Runtime.runtime.availableProcessors().intdiv(2) ?: 1 forkEvery = 100 } Configure all “Test” tasks to fork every 100 tests -
Running tests in separate process, cont. $ gradlew testDebug >
Task :testDebugUnitTest <============-> 97% EXECUTING [17s] > :testDebugUnitTest > 17 tests completed > :testDebugUnitTest > Executing test com.example.SomeTest BUILD SUCCESSFUL in 17s 28 actionable tasks: 26 executed, 2 up-to-date Unit tests running in 1 worker process -
Running tests in separate process, cont. $ gradlew testDebug >
Task :testDebugUnitTest <============-> 97% EXECUTING [17s] > :testDebugUnitTest > 17 tests completed > :testDebugUnitTest > Executing test com.example.SomeTest1 > :testDebugUnitTest > Executing test com.example.SomeTest2 > :testDebugUnitTest > Executing test com.example.SomeTest3 > :testDebugUnitTest > Executing test com.example.SomeTest4 BUILD SUCCESSFUL in 10s 28 actionable tasks: 2 executed, 26 up-to-date Unit tests running in 4 parallel workers processes -
Enable configure on demand • Only configure projects needed based
on the specified task • For projects with many subprojects, it avoids unnecessary configuration, saving time • Works best for projects that have many decoupled modules • Using “subprojects”/“allprojects” removes these benefits since they are executed eagerly -
Enable build cache • Incremental builds will helps avoid work
that is already done • When switching branches, you are forced to rebuild over again but local build cache remembers this previous results • Extra beneficial when build different product flavors for Android builds (even different task names, reuse between flavors) • When working on a team, you might want to consider Gradle’s remote build cache, so you can share build outputs with developers -
Apply plugins judiciously • Similar to adding dependencies, only add
them to the modules that need them • Try not to use “subprojects” or “allprojects” block to apply plugins unless all of the subprojects actually use them! • Limiting the scope of applying your plugins help speed up builds -
Avoid plugins that block/slow configuration • Check for plugins that
resolve/configure slowly • Applying plugins + evaluating build scripts + running “afterEvaluate” = Configuration time • From Stefan Oehme’s talk, “Android Performance Pitfalls”, at Gradle Summit 2017, he mentioned that having a configuration time of less than 1 second or less is good youtube.com/watch?v=TGUwevlaNNU -
Avoid plugins that block/slow configuration $ gradlew > Task :help
Welcome to Gradle 4.3. To run a build, run gradle <task> ... To see a list of available tasks, run gradle tasks To see a list of command-line options, run gradle --help To see more detail about a task, run gradle help --task <task> BUILD SUCCESSFUL in 0s 1 actionable task: 1 executed Normal execution time -
Avoid plugins that block/slow configuration buildscript { repositories { google()
maven { url "https://plugins.gradle.org/m2/" } } dependencies { classpath "com.android.tools.build:gradle:3.0.0" classpath "some:slow-plugin:1.0.0" } } apply plugin: "com.android.application" apply plugin: "slow-plugin" Example of a slow plugin slowing down the configuration -
Avoid plugins that block/slow configuration buildscript { repositories { google()
maven { url "https://plugins.gradle.org/m2/" } } dependencies { classpath "com.android.tools.build:gradle:3.0.0" classpath "some:slow-plugin:1.0.0" } } apply plugin: "com.android.application" apply plugin: "slow-plugin" Example of a slow plugin slowing down the configuration -
Avoid plugins that block/slow configuration buildscript { def isCi =
rootProject.hasProperty("ci") repositories { google() maven { url "https://plugins.gradle.org/m2/" } } dependencies { classpath "com.android.tools.build:gradle:3.0.0" classpath "some:slow-plugin:1.0.0" } } apply plugin: "com.android.application" apply plugin: "slow-plugin" Use a property to check if we are on the CI -
Avoid plugins that block/slow configuration buildscript { def isCi =
rootProject.hasProperty("ci") repositories { google() maven { url "https://plugins.gradle.org/m2/" } } dependencies { classpath "com.android.tools.build:gradle:3.0.0" if (isCi) classpath "some:slow-plugin:1.0.0" } } apply plugin: "com.android.application" if (isCi) apply plugin: "slow-plugin" Use the property to disable the plugin from being applied locally -
Minimize repository count • Most popular repositories include: • jcenter()
• mavenCentral() • maven { url "https://plugins.gradle.org/m2/" } • google() (Gradle 4+) or maven { url "https://maven.google.com" } • maven { url "https://jitpack.io" } -
Minimize repository count allprojects { repositories { jcenter() mavenCentral() maven
{ url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } } Most likely using too many repositories here! -
Minimize repository count • jcenter() and mavenCentral() - may host
the same dependencies • google() - only provides Android and Google dependencies • maven { url "https://plugins.gradle.org/m2/" } - hosts Gradle plugins and mirrors dependencies from jcenter() • maven { url "https://jitpack.io" } - should be used sparingly as it is used to grab a specific branch or commit of a particular open source repository -
Minimize repository count allprojects { repositories { jcenter() mavenCentral() maven
{ url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } } What to remove? -
Minimize repository count allprojects { repositories { jcenter() mavenCentral() maven
{ url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } } What to remove? -
Minimize repository count repositories { jcenter() mavenCentral() maven { url
"https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } What to remove? -
Minimize repository count What to remove? repositories { jcenter() mavenCentral()
maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } -
Minimize repository count What to remove? repositories { jcenter() maven
{ url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } -
Minimize repository count What to remove? repositories { jcenter() maven
{ url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } -
Minimize repository count What to remove? repositories { maven {
url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } -
Minimize repository count What to remove? repositories { maven {
url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } -
Minimize repository count Minimal as possible for this example! repositories
{ maven { url "https://plugins.gradle.org/m2/" } google() } -
Avoid dynamic dependencies (+) • Can cause unexpected version updates
• Difficulty resolving version differences • Slower builds caused by always checking for updates -
Avoid dynamic dependencies Don’t: Use dynamic dependencies dependencies { compile
"com.android.support:appcompat-v7:+" } -
Avoid dynamic dependencies • Instead of guessing or grabbing the
latest dependency via (+), let’s find the latest release • Using Ben Mane’s Gradle Version plugin, you can find the latest plugin and dependencies versions • Instead of having to search for each dependency individually -
Avoid dynamic dependencies $ gradlew dependencyUpdates > Task :dependencyUpdates ------------------------------------------------------------
: Project Dependency Updates (report to plain text file) ------------------------------------------------------------ The following dependencies are using the latest milestone version: - com.android.support:appcompat-v7:27.0.0 - com.github.ben-manes:gradle-versions-plugin:0.17.0 Generated report file build/dependencyUpdates/report.txt BUILD SUCCESSFUL in 0s 1 actionable task: 1 executed github.com/ben-manes/gradle-versions-plugin -
Avoid dynamic dependencies Do: Use static dependencies dependencies { compile
"com.android.support:appcompat-v7:27.0.0" } -
Avoid dynamic dependencies For dependencies to be certain versions configurations.all
{ resolutionStrategy.force "com.android.support:support-annotations:27.0.0" } -
Avoid unnecessary/unused dependencies • For mobile builds, try not to
add large dependencies • Scan for and remove extra/unused dependencies that maybe sitting in the “compile”/“implementation” configurations • Determine which libraries that you are not using heavily or should not be on mobile (eg. Guava, Jackson instead of Gson) • My talk called, “The Road to Single DEX” goes into more detail -
Enable incremental builds • Allow Gradle to resources already compiled
resources to make the builds even faster • It only compiles only those java classes that were changed or that are dependencies to the changed classes -
Incremental builds tasks.withType(JavaCompile) { options.fork = true } From our
previous JavaCompile example -
Incremental builds tasks.withType(JavaCompile) { options.incremental = true options.fork = true
} JavaCompile with incremental builds and forking -
Incremental builds • However, libraries such as AutoValue, Glide, Butterknife,
Dagger that use annotation processing disable incremental builds! • In AGP 3.0+, I believe support for annotation processing and incremental builds are a priority -
Incremental builds JavaCompile with incremental builds and forking tasks.withType(JavaCompile) {
options.incremental = true options.fork = true } -
Incremental builds tasks.withType(JavaCompile) { options.incremental = true options.fork = true
doFirst { println "Task ${path} annotation processors: ${effectiveAnnotationProcessorPath.asPath}" } } JavaCompile tasks will print annotation processors -
Keep your software up to date • Try to use
the latest version of Android Gradle Plugin - 3.0.0 • Keep the SDK tools and platform tools updated to the latest revisions • By keeping the SDK up to date, the Android Gradle Plugin can utilize the latest tools to optimize your build(speed, performance and bug fixes) -
Removing unnecessary flavors android { buildTypes { debug {} release
{ minifyEnabled = true } } flavorDimensions "dim" productFlavors { nightly { dimension "dim" } prod { dimension "dim" } } } Example show buildTypes with productFlavors -
Removing unnecessary flavors $ gradlew tasks > Task :tasks Build
tasks ----------- assemble - Assembles all variants of all applications and secondary packages. assembleAndroidTest - Assembles all the Test applications. assembleDebug - Assembles all Debug builds. assembleNightly - Assembles all Nightly builds. assembleProd - Assembles all Prod builds. assembleRelease - Assembles all Release builds. BUILD SUCCESSFUL in 0s 1 actionable task: 1 executed Produces 4 tasks - Debug/Nightly/Prod/Release -
Removing unnecessary flavors def isNightly = project.hasProperty("nightly") android { buildTypes
{ debug { if (isNightly) { minifyEnabled = true applicationIdSuffix = ".nightly" } else { applicationIdSuffix = ".debug" } } release { minifyEnabled = true } } } Exact same example with no productFlavors. Use “gradlew -Pnightly” for “nightly”. -
Removing unnecessary flavors $ gradlew tasks > Task :tasks Build
tasks ----------- assemble - Assembles all variants of all applications and secondary packages. assembleAndroidTest - Assembles all the Test applications. assembleDebug - Assembles all Debug builds. assembleRelease - Assembles all Release builds. BUILD SUCCESSFUL in 0s 1 actionable task: 1 executed Produces 2 tasks - Debug/Release -
Removing unnecessary flavors android { buildTypes { debug {} release
{ minifyEnabled = true } } flavorDimensions "dim" productFlavors { nightly { dimension "dim" } prod { dimension "dim" } } variantFilter { variant -> def names = variant.flavors*.name if (names.contains("nightly")) { variant.ignore = true } } } You can also use “android.variantFilter”. This example removes “nightly”. -
Avoid compiling unnecessary resources • Use "resConfigs" to filter out
localizations that you do not want/support in your app -
Avoid compiling unnecessary resources Using “resConfigs” to keep English only
android { defaultConfig { resConfigs "en" } } -
Avoid compiling unnecessary resources Using “resConfigs” to keep multiple resources
or languages android { defaultConfig { resConfigs "en","de","fr" } } -
Static constants for builds • Using dynamic variables that change
your current build.gradle files or resources will add to the build time • Resources changed on each build prevent Instant Run from performing a code swap • Use static values for debug builds to prevent a full rebuilds of the APK -
Static constants for builds android { compileSdkVersion 27 buildToolsVersion "27.0.0"
defaultConfig { applicationId "burrows.apps.example" versionCode new Date().format("ddMMyyHHmm").toInteger() versionName "1.0" minSdkVersion 19 targetSdkVersion 27 } } On each build, the “versionCode” is updated with a new value -
Static constants for builds android { compileSdkVersion 27 buildToolsVersion "27.0.0"
defaultConfig { applicationId "burrows.apps.example" versionCode new Date().format("ddMMyyHHmm").toInteger() versionName "1.0" minSdkVersion 19 targetSdkVersion 27 } } On each build, the “versionCode” is updated with a new value -
Static constants for builds def isRelease = project.hasProperty("release") android {
compileSdkVersion 27 buildToolsVersion "27.0.0" defaultConfig { applicationId "burrows.apps.example" versionCode isRelease ? new Date().format("ddMMyyHHmm").toInteger() : 1 versionName "1.0" minSdkVersion 19 targetSdkVersion 27 } } Now, only release builds will be given an updated value -
Create library modules • Gradle will only compile the modules
you modify • Cache those outputs for future builds • Improves effectiveness of “configuration on demand” and “parallel” execution • Particularly helpful when packaging by feature -
Disable PNG crunching • If you can’t convert your PNGs
to WebP, use PNG crunching • By disabling, AGP will not compress the PNGs on each build • Starting in AGP 3.0.0+, PNG crunching is disabled by default for “debug” build types -
Disable PNG crunching android { aaptOptions { cruncherEnabled = project.hasProperty("ci")
} } For AGP <= 2.3.3, use “aaptOptions.cruncherEnabled = false” -
Avoid legacy multidex • “minSdkVersion” < 21 - legacy mutlidex
• “minSdkVersion” >= 21 - native multidex • We can speed up development builds by using “productFlavors” or Gradle properties to toggle the “minSdkVersion” for faster build times -
Avoid legacy multidex android { compileSdkVersion 27 buildToolsVersion "27.0.0" defaultConfig
{ applicationId "burrows.apps.example" versionCode 1 versionName "1.0" minSdkVersion 19 targetSdkVersion 27 multiDexEnabled true } } Example that uses legacy multidex -
Avoid legacy multidex android { compileSdkVersion 27 buildToolsVersion "27.0.0" defaultConfig
{ applicationId "burrows.apps.example" versionCode 1 versionName "1.0" minSdkVersion 19 targetSdkVersion 27 multiDexEnabled true } } Example that uses legacy multidex -
Avoid legacy multidex android { compileSdkVersion 27 buildToolsVersion "27.0.0" defaultConfig
{ applicationId "burrows.apps.example" versionCode 1 versionName "1.0" minSdkVersion rootProject.hasProperty("lollipop") ? 21 : 19 targetSdkVersion 27 multiDexEnabled true } } Example that uses legacy multidex -
Disable pre-dexing libraries on CI • Java class files need
to be converted to DEX files • Helps to speed up the build for incremental builds locally • On a CI server, where you run clean builds, pre-dexing can have the reverse affect, so we can disable it -
Disable pre-dexing libraries android { dexOptions { preDexLibraries = !project.hasProperty("ci")
} } Set “preDexLibraries” to false when on the CI -
Enable Build Cache • Stores certain outputs that the Android
plugin for Gradle generates when building your project (ex. unpackaged AARs, pre-dexed dependencies) • Improves performance of clean builds by reusing cached files • Starting in AGP 2.3.0+, build cache is enabled by default -
gradle.properties, cont. # Gradle specific org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true org.gradle.configureondemand=true org.gradle.caching=true
# Android specific android.enableBuildCache=true Example “gradle.properties” file usage -
Enable new DEX Compiler • DEX compilation is the process
of transforming .class bytecode into .dex bytecode • DX vs D8: compiles faster and outputs smaller .dex files • D8 has the same or better app runtime performance • Starting in AGP 3.0.0+, you can toggle this on to test it • Starting in AGP 3.1.0+, this is enabled by default -
gradle.properties, cont. # Gradle specific org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true org.gradle.configureondemand=true org.gradle.caching=true
# Android specific android.enableBuildCache=true Example “gradle.properties” file usage -
gradle.properties, cont. # Gradle specific org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true org.gradle.configureondemand=true org.gradle.caching=true
# Android specific android.enableBuildCache=true android.enableD8=true Example “gradle.properties” file usage -
Convert images to WebP • Reduces image sizes by converting
to WebP • Does not need to compress at build time(PNG crunching) • By converting beforehand, this will help speed up your build -
Convert images to WebP Right-click on “drawable” folders, click on
“Convert to WebP” -
Enable Instant Run • Push certain code and resource changes
to your running app without building a new APK • Some cases, without even restarting the current activity -
Enable Instant Run In “Preferences”, under “Build, Execution, Deployment”, turn
on “Enable Instant Run” -
Enable offline mode • If do not need to update
dependencies or plugins frequently, you would greatly benefit from compiling offline • This skips dependency resolution and downloading and uses cache only • For command line: “gradlew --offline” -
Profiling builds • “gradlew --profile” (built in) • “gradlew --scan”
(docs.gradle.com/build-scan-plugin) • For more tools: github.com/gradle/gradle-profiler -
Results Faster CI builds, Merge pull requests faster, Iterate faster
Before After Reduction Time(min) ~18+ ~11 38.89%