Make Your Build Great Again (DroidConSF 2017)

Slide 1

Slide 1 text

Jared Burrows Make Your Build Great Again

Slide 2

Slide 2 text

Motivation • Yammer is a social networking service used for communication within an organization • Current debug builds are currently multidex • More features and more code leads to slower builds • More libraries leads to slower builds

Slide 3

Slide 3 text

Results Faster CI builds, Merge pull requests faster, Iterate faster Before After Reduction Time(min) ~18+ ~11 38.89%

Slide 4

Slide 4 text

What can we optimize? • Software  - Gradle  - Android Gradle Plugin  - Android Studio/Intellij • Hardware  - Increase CPU  - Increase Memory 

Slide 5

Slide 5 text

What to look for • Cause of the build slowness • Identify bottlenecks during start up and between tasks • Find and remove unnecessary dependencies/plugins • Module and build.gradle structure

Slide 6

Slide 6 text

Where to start • Start from the bottom of your tooling • Better developer laptops and/or build agents • Optimize and configure Gradle • Optimize and configure Android Gradle Plugin • Optimize and configure Android Studio

Slide 7

Slide 7 text

Optimizing your Gradle setup

Slide 8

Slide 8 text

Keep your software up to date • Try to use the latest version of Gradle - 4.3 • Ensures compatibility between minor versions • Make transitioning to the next major version easier • Use Gradle with the latest major version of JVM - 1.8

Slide 9

Slide 9 text

gradle.properties

Slide 10

Slide 10 text

Enable the Gradle daemon • Each time you run Gradle, it kicks off a new instance of Gradle • Use the daemon to avoid the cost of JVM startup time for each build • Starting in 3.0+, the Daemon is on by default!

Slide 11

Slide 11 text

gradle.properties org.gradle.daemon=true Example “gradle.properties” ﬁle usage

Slide 12

Slide 12 text

Increase daemon’s heap size • By default Gradle will reserves 1GB of heap space • Android projects usually need more memory • As of AGP 2.1+, you want at least 2GB for “Dex in Process” • Giving more memory will help dramatically speed up builds for larger projects

Slide 13

Slide 13 text

gradle.properties, cont. org.gradle.daemon=true Example “gradle.properties” ﬁle usage

Slide 14

Slide 14 text

gradle.properties, cont. org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m Example “gradle.properties” ﬁle usage

Slide 15

Slide 15 text

Enable parallel builds • Execute Gradle tasks in parallel to speed up builds • Helpful for multi-module builds that are decoupled • JavaCompile tasks - Compile in separate process • Test tasks - Run tests in separate process

Slide 16

Slide 16 text

gradle.properties, cont. org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m Example “gradle.properties” ﬁle usage

Slide 17

Slide 17 text

gradle.properties, cont. org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true Example “gradle.properties” ﬁle usage

Slide 18

Slide 18 text

JavaCompile tasks separate process tasks.withType(JavaCompile) { options.fork = true } Conﬁgure all “JavaCompile” tasks to fork

Slide 19

Slide 19 text

Running tests in separate process tasks.withType(Test) { def maxCount = gradle.startParameter.maxWorkerCount maxParallelForks = (maxCount < 2) ? 1 : maxCount / 2 forEvery = 100 } Conﬁgure all “Test” tasks to fork every 100 tests

Slide 20

Slide 20 text

Running tests in separate process, cont. tasks.withType(Test) { maxParallelForks = Runtime.runtime.availableProcessors().intdiv(2) ?: 1 forkEvery = 100 } Conﬁgure all “Test” tasks to fork every 100 tests

Slide 21

Slide 21 text

Running tests in separate process, cont. $ gradlew testDebug > Task :testDebugUnitTest <============-> 97% EXECUTING [17s] > :testDebugUnitTest > 17 tests completed > :testDebugUnitTest > Executing test com.example.SomeTest   BUILD SUCCESSFUL in 17s 28 actionable tasks: 26 executed, 2 up-to-date Unit tests running in 1 worker process

Slide 22

Slide 22 text

Running tests in separate process, cont. $ gradlew testDebug > Task :testDebugUnitTest <============-> 97% EXECUTING [17s] > :testDebugUnitTest > 17 tests completed > :testDebugUnitTest > Executing test com.example.SomeTest1 > :testDebugUnitTest > Executing test com.example.SomeTest2 > :testDebugUnitTest > Executing test com.example.SomeTest3 > :testDebugUnitTest > Executing test com.example.SomeTest4 BUILD SUCCESSFUL in 10s 28 actionable tasks: 2 executed, 26 up-to-date Unit tests running in 4 parallel workers processes

Slide 23

Slide 23 text

Enable configure on demand • Only configure projects needed based on the specified task • For projects with many subprojects, it avoids unnecessary configuration, saving time • Works best for projects that have many decoupled modules • Using “subprojects”/“allprojects” removes these benefits since they are executed eagerly

Slide 24

Slide 24 text

gradle.properties, cont. org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true Example “gradle.properties” ﬁle usage

Slide 25

Slide 25 text

gradle.properties, cont. org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true org.gradle.configureondemand=true Example “gradle.properties” ﬁle usage

Slide 26

Slide 26 text

Enable build cache • Incremental builds will helps avoid work that is already done • When switching branches, you are forced to rebuild over again but local build cache remembers this previous results • Extra beneficial when build different product flavors for Android builds (even different task names, reuse between flavors) • When working on a team, you might want to consider Gradle’s remote build cache, so you can share build outputs with developers

Slide 27

Slide 27 text

gradle.properties, cont. org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true org.gradle.configureondemand=true Example “gradle.properties” ﬁle usage

Slide 28

Slide 28 text

gradle.properties, cont. org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true org.gradle.configureondemand=true org.gradle.caching=true Example “gradle.properties” ﬁle usage

Slide 29

Slide 29 text

build.gradle

Slide 30

Slide 30 text

Apply plugins judiciously • Similar to adding dependencies, only add them to the modules that need them • Try not to use “subprojects” or “allprojects” block to apply plugins unless all of the subprojects actually use them! • Limiting the scope of applying your plugins help speed up builds

Slide 31

Slide 31 text

Avoid plugins that block/slow configuration • Check for plugins that resolve/configure slowly • Applying plugins + evaluating build scripts + running “afterEvaluate” = Configuration time • From Stefan Oehme’s talk, “Android Performance Pitfalls”, at Gradle Summit 2017, he mentioned that having a configuration time of less than 1 second or less is good youtube.com/watch?v=TGUwevlaNNU

Slide 32

Slide 32 text

Avoid plugins that block/slow conﬁguration $ gradlew > Task :help Welcome to Gradle 4.3. To run a build, run gradle ... To see a list of available tasks, run gradle tasks To see a list of command-line options, run gradle --help To see more detail about a task, run gradle help --task BUILD SUCCESSFUL in 0s 1 actionable task: 1 executed Normal execution time

Slide 33

Slide 33 text

Avoid plugins that block/slow conﬁguration buildscript { repositories { google() maven { url "https://plugins.gradle.org/m2/" } } dependencies { classpath "com.android.tools.build:gradle:3.0.0" classpath "some:slow-plugin:1.0.0" } } apply plugin: "com.android.application" apply plugin: "slow-plugin" Example of a slow plugin slowing down the conﬁguration

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Avoid plugins that block/slow conﬁguration buildscript { def isCi = rootProject.hasProperty("ci") repositories { google() maven { url "https://plugins.gradle.org/m2/" } } dependencies { classpath "com.android.tools.build:gradle:3.0.0" classpath "some:slow-plugin:1.0.0" } } apply plugin: "com.android.application" apply plugin: "slow-plugin" Use a property to check if we are on the CI

Slide 36

Slide 36 text

Avoid plugins that block/slow conﬁguration buildscript { def isCi = rootProject.hasProperty("ci") repositories { google() maven { url "https://plugins.gradle.org/m2/" } } dependencies { classpath "com.android.tools.build:gradle:3.0.0" if (isCi) classpath "some:slow-plugin:1.0.0" } } apply plugin: "com.android.application" if (isCi) apply plugin: "slow-plugin" Use the property to disable the plugin from being applied locally

Slide 37

Slide 37 text

Minimize repository count • Most popular repositories include: • jcenter() • mavenCentral() • maven { url "https://plugins.gradle.org/m2/" } • google() (Gradle 4+) or maven { url "https://maven.google.com" } • maven { url "https://jitpack.io" }

Slide 38

Slide 38 text

Minimize repository count allprojects { repositories { jcenter() mavenCentral() maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } } Most likely using too many repositories here!

Slide 39

Slide 39 text

Minimize repository count • jcenter() and mavenCentral() - may host the same dependencies • google() - only provides Android and Google dependencies • maven { url "https://plugins.gradle.org/m2/" } - hosts Gradle plugins and mirrors dependencies from jcenter() • maven { url "https://jitpack.io" } - should be used sparingly as it is used to grab a speciﬁc branch or commit of a particular open source repository

Slide 40

Slide 40 text

Minimize repository count allprojects { repositories { jcenter() mavenCentral() maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } } What to remove?

Slide 41

Slide 41 text

Minimize repository count allprojects { repositories { jcenter() mavenCentral() maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } } What to remove?

Slide 42

Slide 42 text

Minimize repository count repositories { jcenter() mavenCentral() maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } } What to remove?

Slide 43

Slide 43 text

Minimize repository count What to remove? repositories { jcenter() mavenCentral() maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } }

Slide 44

Slide 44 text

Minimize repository count What to remove? repositories { jcenter() maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } }

Slide 45

Slide 45 text

Minimize repository count What to remove? repositories { jcenter() maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } }

Slide 46

Slide 46 text

Minimize repository count What to remove? repositories { maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } }

Slide 47

Slide 47 text

Minimize repository count What to remove? repositories { maven { url "https://plugins.gradle.org/m2/" } google() maven { url "https://jitpack.io" } }

Slide 48

Slide 48 text

Minimize repository count Minimal as possible for this example! repositories { maven { url "https://plugins.gradle.org/m2/" } google() }

Slide 49

Slide 49 text

Avoid dynamic dependencies (+) • Can cause unexpected version updates • Difﬁculty resolving version differences • Slower builds caused by always checking for updates

Slide 50

Slide 50 text

Avoid dynamic dependencies dependencies { compile "com.android.support:appcompat-v7:+" } Don’t: Use dynamic dependencies

Slide 51

Slide 51 text

Avoid dynamic dependencies Don’t: Use dynamic dependencies dependencies { compile "com.android.support:appcompat-v7:+" }

Slide 52

Slide 52 text

Avoid dynamic dependencies • Instead of guessing or grabbing the latest dependency via (+), let’s ﬁnd the latest release • Using Ben Mane’s Gradle Version plugin, you can ﬁnd the latest plugin and dependencies versions • Instead of having to search for each dependency individually

Slide 53

Slide 53 text

Avoid dynamic dependencies $ gradlew dependencyUpdates > Task :dependencyUpdates ------------------------------------------------------------ : Project Dependency Updates (report to plain text file) ------------------------------------------------------------ The following dependencies are using the latest milestone version: - com.android.support:appcompat-v7:27.0.0 - com.github.ben-manes:gradle-versions-plugin:0.17.0 Generated report file build/dependencyUpdates/report.txt BUILD SUCCESSFUL in 0s 1 actionable task: 1 executed github.com/ben-manes/gradle-versions-plugin

Slide 54

Slide 54 text

Avoid dynamic dependencies dependencies { compile "com.android.support:appcompat-v7:+" } Don’t: Use dynamic dependencies

Slide 55

Slide 55 text

Avoid dynamic dependencies Do: Use static dependencies dependencies { compile "com.android.support:appcompat-v7:27.0.0" }

Slide 56

Slide 56 text

Avoid dynamic dependencies For dependencies to be certain versions configurations.all { resolutionStrategy.force "com.android.support:support-annotations:27.0.0" }

Slide 57

Slide 57 text

Avoid unnecessary/unused dependencies • For mobile builds, try not to add large dependencies • Scan for and remove extra/unused dependencies that maybe sitting in the “compile”/“implementation” conﬁgurations • Determine which libraries that you are not using heavily or should not be on mobile (eg. Guava, Jackson instead of Gson) • My talk called, “The Road to Single DEX” goes into more detail

Slide 58

Slide 58 text

Enable incremental builds • Allow Gradle to resources already compiled resources to make the builds even faster • It only compiles only those java classes that were changed or that are dependencies to the changed classes

Slide 59

Slide 59 text

Incremental builds tasks.withType(JavaCompile) { options.fork = true } From our previous JavaCompile example

Slide 60

Slide 60 text

Incremental builds tasks.withType(JavaCompile) { options.incremental = true options.fork = true } JavaCompile with incremental builds and forking

Slide 61

Slide 61 text

Incremental builds • However, libraries such as AutoValue, Glide, Butterknife, Dagger that use annotation processing disable incremental builds! • In AGP 3.0+, I believe support for annotation processing and incremental builds are a priority

Slide 62

Slide 62 text

Incremental builds JavaCompile with incremental builds and forking tasks.withType(JavaCompile) { options.incremental = true options.fork = true }

Slide 63

Slide 63 text

Incremental builds tasks.withType(JavaCompile) { options.incremental = true options.fork = true doFirst { println "Task ${path} annotation processors: ${effectiveAnnotationProcessorPath.asPath}" } } JavaCompile tasks will print annotation processors

Slide 64

Slide 64 text

Optimizing Your AGP Setup

Slide 65

Slide 65 text

Keep your software up to date • Try to use the latest version of Android Gradle Plugin - 3.0.0 • Keep the SDK tools and platform tools updated to the latest revisions • By keeping the SDK up to date, the Android Gradle Plugin can utilize the latest tools to optimize your build(speed, performance and bug ﬁxes)

Slide 66

Slide 66 text

Slide 67

Slide 67 text

Removing unnecessary ﬂavors $ gradlew tasks > Task :tasks Build tasks ----------- assemble - Assembles all variants of all applications and secondary packages. assembleAndroidTest - Assembles all the Test applications. assembleDebug - Assembles all Debug builds. assembleNightly - Assembles all Nightly builds. assembleProd - Assembles all Prod builds. assembleRelease - Assembles all Release builds.  BUILD SUCCESSFUL in 0s 1 actionable task: 1 executed Produces 4 tasks - Debug/Nightly/Prod/Release

Slide 68

Slide 68 text

Removing unnecessary ﬂavors def isNightly = project.hasProperty("nightly") android { buildTypes { debug { if (isNightly) { minifyEnabled = true applicationIdSuffix = ".nightly" } else { applicationIdSuffix = ".debug" } } release { minifyEnabled = true } } } Exact same example with no productFlavors. Use “gradlew -Pnightly” for “nightly”.

Slide 69

Slide 69 text

Removing unnecessary ﬂavors $ gradlew tasks > Task :tasks Build tasks ----------- assemble - Assembles all variants of all applications and secondary packages. assembleAndroidTest - Assembles all the Test applications. assembleDebug - Assembles all Debug builds. assembleRelease - Assembles all Release builds.  BUILD SUCCESSFUL in 0s 1 actionable task: 1 executed Produces 2 tasks - Debug/Release

Slide 70

Slide 70 text

Removing unnecessary ﬂavors android { buildTypes { debug {} release { minifyEnabled = true } } flavorDimensions "dim" productFlavors { nightly { dimension "dim" } prod { dimension "dim" } } variantFilter { variant -> def names = variant.flavors*.name if (names.contains("nightly")) { variant.ignore = true } } } You can also use “android.variantFilter”. This example removes “nightly”.

Slide 71

Slide 71 text

Avoid compiling unnecessary resources • Use "resConﬁgs" to ﬁlter out localizations that you do not want/support in your app

Slide 72

Slide 72 text

Avoid compiling unnecessary resources Using “resConﬁgs” to keep English only android { defaultConfig { resConfigs "en" } }

Slide 73

Slide 73 text

Avoid compiling unnecessary resources Using “resConﬁgs” to keep multiple resources or languages android { defaultConfig {  resConfigs "en","de","fr"  }  }

Slide 74

Slide 74 text

Static constants for builds • Using dynamic variables that change your current build.gradle ﬁles or resources will add to the build time • Resources changed on each build prevent Instant Run from performing a code swap • Use static values for debug builds to prevent a full rebuilds of the APK

Slide 75

Slide 75 text

Static constants for builds android { compileSdkVersion 27 buildToolsVersion "27.0.0" defaultConfig { applicationId "burrows.apps.example" versionCode new Date().format("ddMMyyHHmm").toInteger() versionName "1.0" minSdkVersion 19 targetSdkVersion 27 } } On each build, the “versionCode” is updated with a new value

Slide 76

Slide 76 text

Slide 77

Slide 77 text

Static constants for builds def isRelease = project.hasProperty("release") android { compileSdkVersion 27 buildToolsVersion "27.0.0" defaultConfig { applicationId "burrows.apps.example" versionCode isRelease ? new Date().format("ddMMyyHHmm").toInteger() : 1 versionName "1.0" minSdkVersion 19 targetSdkVersion 27 } } Now, only release builds will be given an updated value

Slide 78

Slide 78 text

Create library modules • Gradle will only compile the modules you modify • Cache those outputs for future builds • Improves effectiveness of “conﬁguration on demand” and “parallel” execution • Particularly helpful when packaging by feature

Slide 79

Slide 79 text

Disable PNG crunching • If you can’t convert your PNGs to WebP, use PNG crunching • By disabling, AGP will not compress the PNGs on each build • Starting in AGP 3.0.0+, PNG crunching is disabled by default for “debug” build types

Slide 80

Slide 80 text

Disable PNG crunching android { aaptOptions { cruncherEnabled = project.hasProperty("ci") } } For AGP <= 2.3.3, use “aaptOptions.cruncherEnabled = false”

Slide 81

Slide 81 text

Avoid legacy multidex • “minSdkVersion” < 21 - legacy mutlidex • “minSdkVersion” >= 21 - native multidex • We can speed up development builds by using “productFlavors” or Gradle properties to toggle the “minSdkVersion” for faster build times

Slide 82

Slide 82 text

Avoid legacy multidex android { compileSdkVersion 27 buildToolsVersion "27.0.0" defaultConfig { applicationId "burrows.apps.example" versionCode 1 versionName "1.0" minSdkVersion 19 targetSdkVersion 27 multiDexEnabled true } } Example that uses legacy multidex

Slide 83

Slide 83 text

Slide 84

Slide 84 text

Avoid legacy multidex android { compileSdkVersion 27 buildToolsVersion "27.0.0" defaultConfig { applicationId "burrows.apps.example" versionCode 1 versionName "1.0" minSdkVersion rootProject.hasProperty("lollipop") ? 21 : 19 targetSdkVersion 27 multiDexEnabled true } } Example that uses legacy multidex

Slide 85

Slide 85 text

Disable pre-dexing libraries on CI • Java class ﬁles need to be converted to DEX ﬁles • Helps to speed up the build for incremental builds locally • On a CI server, where you run clean builds, pre-dexing can have the reverse affect, so we can disable it

Slide 86

Slide 86 text

Disable pre-dexing libraries android { dexOptions { preDexLibraries = !project.hasProperty("ci") } } Set “preDexLibraries” to false when on the CI

Slide 87

Slide 87 text

Enable Build Cache • Stores certain outputs that the Android plugin for Gradle generates when building your project (ex. unpackaged AARs, pre-dexed dependencies) • Improves performance of clean builds by reusing cached ﬁles • Starting in AGP 2.3.0+, build cache is enabled by default

Slide 88

Slide 88 text

Slide 89

Slide 89 text

gradle.properties, cont. # Gradle specific org.gradle.daemon=true org.gradle.jvmargs=-Xmx2048m org.gradle.parallel=true org.gradle.configureondemand=true org.gradle.caching=true Example “gradle.properties” ﬁle usage

Slide 90

Slide 90 text

Slide 91

Slide 91 text

Enable new DEX Compiler • DEX compilation is the process of transforming .class bytecode into .dex bytecode • DX vs D8: compiles faster and outputs smaller .dex ﬁles • D8 has the same or better app runtime performance • Starting in AGP 3.0.0+, you can toggle this on to test it • Starting in AGP 3.1.0+, this is enabled by default

Slide 92

Slide 92 text

Slide 93

Slide 93 text

Slide 94

Slide 94 text

Optimizing Your Android Studio Usage

Slide 95

Slide 95 text

Convert images to WebP • Reduces image sizes by converting to WebP • Does not need to compress at build time(PNG crunching) • By converting beforehand, this will help speed up your build

Slide 96

Slide 96 text

Convert images to WebP Right-click on “drawable” folders, click on “Convert to WebP”

Slide 97

Slide 97 text

Enable Instant Run • Push certain code and resource changes to your running app without building a new APK • Some cases, without even restarting the current activity

Slide 98

Slide 98 text

Enable Instant Run In “Preferences”, under “Build, Execution, Deployment”, turn on “Enable Instant Run”

Slide 99

Slide 99 text

Enable offline mode • If do not need to update dependencies or plugins frequently, you would greatly benefit from compiling offline • This skips dependency resolution and downloading and uses cache only • For command line: “gradlew --offline”

Slide 100

Slide 100 text

Enable ofﬂine mode In “Preferences”, under “Build, Execution, Deployment”, turn on “Ofﬂine work”

Slide 101

Slide 101 text

What now?

Slide 102

Slide 102 text

Profiling builds • “gradlew --profile” (built in) • “gradlew --scan” (docs.gradle.com/build-scan-plugin) • For more tools: github.com/gradle/gradle-profiler

Slide 103

Slide 103 text

Proﬁling builds Example of “gradlew clean assembleDebug --proﬁle”

Slide 104

Slide 104 text

Proﬁling builds Example of “gradlew clean assembleDebug --scan”

Slide 105

Slide 105 text

Results Faster CI builds, Merge pull requests faster, Iterate faster Before After Reduction Time(min) ~18+ ~11 38.89%

Slide 106

Slide 106 text

Thank you! Questions? twitter.com/jaredsburrows github.com/jaredsburrows [email protected] jaredsburrows.com