Squeezing the Last Drop of Performance Out of Your Gradle Builds (Droidcon Paris 2015)

Slide 1

Slide 1 text

source: https://xkcd.com/303/ 1 / 91

Slide 2

Slide 2 text

Squeezing the Last Drop of Performance Out of Your Gradle Builds 2 / 91

Slide 3

Slide 3 text

Obligatory Disclaimer - YMMV Your Mileage May Vary 3 / 91

Slide 4

Slide 4 text

Always Measure First 4 / 91

Slide 5

Slide 5 text

Gradle Build Lifecycle Three distinct phases: 1. Initialization 5 / 91

Slide 6

Slide 6 text

Gradle Build Lifecycle Three distinct phases: 1. Initialization 2. Conﬁguration 6 / 91

Slide 7

Slide 7 text

Gradle Build Lifecycle Three distinct phases: 1. Initialization 2. Conﬁguration 3. Execution 7 / 91

Slide 8

Slide 8 text

Gradle Build Lifecycle Execution - the useful part where Gradle builds your app 8 / 91

Slide 9

Slide 9 text

Gradle Build Lifecycle Execution - the useful part where Gradle builds your app Initialization, Conﬁguration - the time Gradle spends on ﬁguring out how to build your app, the overhead 9 / 91

Slide 10

Slide 10 text

Initialization and Conﬁguration The overhead tax you have to pay whenever you invoke Gradle 10 / 91

Slide 11

Slide 11 text

Measuring the Overhead Test subject: Google IO app 28 libraries 53149 method references http://github.com/google/iosched 11 / 91

Slide 12

Slide 12 text

Measuring the Overhead The --dry-run command line ﬂag skips running all tasks $ ./gradlew :android:assembleDebug --dry-run ... :android:dexDebug SKIPPED :android:validateDebugSigning SKIPPED :android:packageDebug SKIPPED :android:zipalignDebug SKIPPED :android:assembleDebug SKIPPED BUILD SUCCESSFUL Total time: 4.256 secs 12 / 91

Slide 13

Slide 13 text

Measuring the Overhead You can use the --profile ﬂag to get more information $ ./gradlew :android:assembleDebug --dry-run --profile $ open build/reports/profile/profile-2015-10-27-08-06-31.html 13 / 91

Slide 14

Slide 14 text

Conﬁgure on Demand Without --configure-on-demand --configure-on-demand: $ ./gradlew :android:assembleDebug --dry-run ... BUILD SUCCESSFUL Total time: 4.286 secs With --configure-on-demand --configure-on-demand: $ ./gradlew :android:assembleDebug --dry-run --configure-on-demand ... BUILD SUCCESSFUL Total time: 3.574 secs 14 / 91

Slide 15

Slide 15 text

Conﬁgure on Demand Enable conﬁgure on demand when invoking from the command line: echo echo 'org.gradle.configureondemand=true' >> \ ~/.gradle/gradle.properties 15 / 91

Slide 16

Slide 16 text

Conﬁgure on Demand Enable conﬁgure on demand when invoking from the command line: echo echo 'org.gradle.configureondemand=true' >> \ ~/.gradle/gradle.properties Enable for Studio in Preferences -> Build, Execution, Deployment -> Compiler 16 / 91

Slide 17

Slide 17 text

The Win So Far We've gone from 4.3 seconds to 3.6 seconds A 17% win! 17 / 91

Slide 18

Slide 18 text

Gradle Daemon Without daemon: $ ./gradlew :android:assembleDebug --dry-run --no-daemon ... BUILD SUCCESSFUL Total time: 3.63 secs With daemon: $ ./gradlew :android:assembleDebug --dry-run --daemon ... BUILD SUCCESSFUL Total time: 1.709 secs 18 / 91

Slide 19

Slide 19 text

Gradle Daemon Always use the Gradle daemon for command line: echo echo 'org.gradle.daemon=true' >> ~/.gradle/gradle.properties 19 / 91

Slide 20

Slide 20 text

Gradle Daemon Always use the Gradle daemon for command line: echo echo 'org.gradle.daemon=true' >> ~/.gradle/gradle.properties For Android Studio, make sure that the Use in- process build option is set. You can ﬁnd it in Preferences -> Build, Execution, Deployment -> Compiler 20 / 91

Slide 21

Slide 21 text

The Win So Far We've gone from 4.3 seconds to 1.7 seconds A 60% win! 21 / 91

Slide 22

Slide 22 text

Gradle Version Gradle 2.2.1: $ ./gradlew :android:assembleDebug --dry-run ... BUILD SUCCESSFUL Total time: 1.536 secs Gradle 2.8: $ ./gradlew :android:assembleDebug --dry-run ... BUILD SUCCESSFUL Total time: 0.931 secs 22 / 91

Slide 23

Slide 23 text

Gradle Version Update gradle in gradle/wrapper/gradle- wrapper.properties #Tue Dec 02 14:11:25 EST 2014 distributionBase=GRADLE_USER_HOME distributionPath=wrapper/dists zipStoreBase=GRADLE_USER_HOME zipStorePath=wrapper/dists distributionUrl=https\://services.gradle.org/distributions/gradle-2.8-all.zip 23 / 91

Slide 24

Slide 24 text

The Win So Far We've gone from 4.3 seconds to 0.9 seconds A 78% win! 24 / 91

Slide 25

Slide 25 text

JVM Version Java 1.6 Java 1.8 $ export export JAVA_HOME=`/usr/libexec/java_home -v '1.6'` $ ./gradlew :android:assembleDebug --dry-run --configure-on-demand ... BUILD SUCCESSFUL Total time: 1.29 secs $ export export JAVA_HOME=`/usr/libexec/java_home -v '1.8'` $ ./gradlew :android:assembleDebug --dry-run --configure-on-demand ... BUILD SUCCESSFUL Total time: 0.871 secs 25 / 91

Slide 26

Slide 26 text

JVM Version Change the JDK used for Gradle in Android Studio. In File -> Project Structure: 26 / 91

Slide 27

Slide 27 text

I/O Gradle does tons of IO! $ ./gradlew :android:assembleDebug $ strace -ff -o assemble.trace ./gradlew :android:assembleDebug $ grep -E "^(open|stat|fstat)" assemble.trace.* | wc -l 34479 27 / 91

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Avoid Heavy Computation def def cmd cmd = 'git rev-list HEAD HEAD --first-parent --count' def def gitVersion gitVersion = cmd.execute().text.trim().toInteger() android { defaultConfig { versionCode gitVersion } } $ time git rev-list master --first-parent --count 23030 hub rev-list master --first-parent --count 0.21s user 0.05s system 53% cpu 0.470 total That's a 470ms overhead for every Gradle build 29 / 91

Slide 30

Slide 30 text

Avoid Heavy Computation def def gitVersion gitVersion() { if if (!System System.getenv('CI_BUILD')) { // don't care return return 1 } def def cmd cmd = 'git rev-list HEAD HEAD --first-parent --count' cmd.execute().text.trim().toInteger() } android { defaultConfig { versionCode gitVersion() } } 30 / 91

Slide 31

Slide 31 text

Dependencies dependencies { compile 'com.google.code.gson:gson:2.+' } 31 / 91

Slide 32

Slide 32 text

Dependencies dependencies { compile 'com.google.code.gson:gson:2.+' } $ date Tue Oct 27 09:22:29 EET 2015 $ ./gradlew :android:help help BUILD SUCCESSFUL Total time: 0.693 secs 32 / 91

Slide 33

Slide 33 text

Dependencies dependencies { compile 'com.google.code.gson:gson:2.+' } $ date Tue Oct 27 09:22:29 EET 2015 $ ./gradlew :android:help help BUILD SUCCESSFUL Total time: 0.693 secs $ date Wed Oct 28 09:23:19 EET 2015 $ ./gradlew :android:help help BUILD SUCCESSFUL Total time: 1.84 secs 33 / 91

Slide 34

Slide 34 text

Dependencies dependencies { compile 'com.google.code.gson:gson:2.+' } $ date Thu Oct 29 09:29:06 EET 2015 $ ./gradlew :android:help help --offline BUILD SUCCESSFUL Total time: 0.756 secs 34 / 91

Slide 35

Slide 35 text

Dependencies dependencies { compile 'com.google.code.gson:gson:2.3' } 35 / 91

Slide 36

Slide 36 text

Dependencies dependencies { compile 'com.google.code.gson:gson:2.3' } Reproducible builds, yaaay \o/ 36 / 91

Slide 37

Slide 37 text

Continuous Builds $ time ./gradlew :android:test testDebugUnitTest BUILD SUCCESSFUL Total time: 0.859 secs 1.06s user 0.11s system 89% cpu 1.305 total 37 / 91

Slide 38

Slide 38 text

Continuous Builds $ time ./gradlew :android:test testDebugUnitTest --continuous BUILD SUCCESSFUL Total time: 0.858 secs Waiting for for changes to input files of tasks... (ctrl-d to exit exit 38 / 91

Slide 39

Slide 39 text

TL;DR Conﬁguration on Demand 39 / 91

Slide 40

Slide 40 text

TL;DR Conﬁguration on Demand Gradle Daemon (and/or the in-process build in AS) 40 / 91

Slide 41

Slide 41 text

TL;DR Conﬁguration on Demand Gradle Daemon (and/or the in-process build in AS) Newer versions of Gradle are faster 41 / 91

Slide 42

Slide 42 text

TL;DR Conﬁguration on Demand Gradle Daemon (and/or the in-process build in AS) Newer versions of Gradle are faster JVM 1.8 is more performant than 1.6 42 / 91

Slide 43

Slide 43 text

TL;DR Conﬁguration on Demand Gradle Daemon (and/or the in-process build in AS) Newer versions of Gradle are faster JVM 1.8 is more performant than 1.6 Avoid doing expensive things during the conﬁguration phase 43 / 91

Slide 44

Slide 44 text

Slide 45

Slide 45 text

Execution Phase All of this is cool but my builds are still slow? 45 / 91

Slide 46

Slide 46 text

Task Graph $ ./gradlew :jar 46 / 91

Slide 47

Slide 47 text

Task Graph $ ./gradlew :jar :jar 47 / 91

Slide 48

Slide 48 text

Task Graph $ ./gradlew :jar :jar :classes 48 / 91

Slide 49

Slide 49 text

Task Graph $ ./gradlew :jar :jar :classes :compileJava 49 / 91

Slide 50

Slide 50 text

Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 50 / 91

Slide 51

Slide 51 text

Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 51 / 91

Slide 52

Slide 52 text

Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 52 / 91

Slide 53

Slide 53 text

Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 53 / 91

Slide 54

Slide 54 text

Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 54 / 91

Slide 55

Slide 55 text

Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 55 / 91

Slide 56

Slide 56 text

Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 56 / 91

Slide 57

Slide 57 text

Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 57 / 91

Slide 58

Slide 58 text

Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 58 / 91

Slide 59

Slide 59 text

Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 59 / 91

Slide 60

Slide 60 text

Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 60 / 91

Slide 61

Slide 61 text

Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 61 / 91

Slide 62

Slide 62 text

Task Graph 62 / 91

Slide 63

Slide 63 text

Task Graph :package :dex :compileJava :processResources res.ap_ R.java *.class classes.dex 63 / 91

Slide 64

Slide 64 text

Build Breakdown - Iosched New ﬁle src/main/res/values/new_file.xml: Hello! $ ./gradlew :android:assembleDebug --profile BUILD SUCCESSFUL Total time: 7.664 secs 64 / 91

Slide 65

Slide 65 text

Build Breakdown - Iosched 65 / 91

Slide 66

Slide 66 text

Build Breakdown - Iosched Package cost: 1.297s Resource change cost: 0.939s merge resources - 0.208s process resources (aapt) - 0.731s Java change cost: 4.462s dex - 3.766s javac - 0.876s 66 / 91

Slide 67

Slide 67 text

Changing Values android { defaultConfig { buildConfigField('String', 'BUILD_DATE', "\"${new Date()} ${new Date()}\"") } } 67 / 91

Slide 68

Slide 68 text

Changing Values android { defaultConfig { resValue 'string', 'build_date', "${new Date()} ${new Date()}" } } 68 / 91

Slide 69

Slide 69 text

Changing Values android { buildTypes { debug { // don't care resValue 'string', 'build_date', 'dev-build' } release { resValue 'string', 'build_date', "${new Date()}" } } } 69 / 91

Slide 70

Slide 70 text

The Dexer Predexing For each library dependency, dex the classes into a .dex ﬁle and cache it. E.g.: gson:2.3 -> gson-2.3-9f28...2e3c.jar 70 / 91

Slide 71

Slide 71 text

The Dexer Predexing For each library dependency, dex the classes into a .dex ﬁle and cache it. E.g.: gson:2.3 -> gson-2.3-9f28...2e3c.jar Dexing Dex the application .class ﬁles, merge the resulting dex with the pre-dexed libraries. 71 / 91

Slide 72

Slide 72 text

The Dexer :dex :javaCompile :preDex *.class lib1.dex, lib2.dex, etc 72 / 91

Slide 73

Slide 73 text

The minSdk 21 Trick android { productFlavors { dev { multiDexEnabled true minSdkVersion 21 } prod {} } } 73 / 91

Slide 74

Slide 74 text

The minSdk 21 Trick $ ./gradlew :android:assembleDevDebug BUILD SUCCESSFUL Total time: 4.633 secs $ ./gradlew :android:assembleProdDebug BUILD SUCCESSFUL Total time: 6.599 secs 74 / 91

Slide 75

Slide 75 text

The minSdk 21 Trick android { productFlavors { dev { multiDexEnabled true minSdkVersion 21 } prod { multiDexEnabled true minSdkVersion 15 } } } 75 / 91

Slide 76

Slide 76 text

The minSdk 21 Trick $ ./gradlew :android:assembleDevDebug BUILD SUCCESSFUL Total time: 4.416 secs $ ./gradlew :android:assembleProdDebug BUILD SUCCESSFUL Total time: 20.703 secs 76 / 91

Slide 77

Slide 77 text

Really Large Projects Build times in the order of minutes 77 / 91

Slide 78

Slide 78 text

Really Large Projects Test subject: android-large-project (artiﬁcial!) Depends on 45 libraries Source contains 3 copies of Guava Produces 3 dex ﬁles and 150640 method references One line Java change takes 50 seconds $ ./gradlew :app:assembleDebug BUILD SUCCESSFUL Total time: 49.695 secs 78 / 91

Slide 79

Slide 79 text

Really Large Projects One line Java change with minSdkVersion 21 takes 19 seconds android { defaultConfig { minSdkVersion 21 multiDexEnabled true } } $ ./gradlew :app:assembleDebug BUILD SUCCESSFUL Total time: 18.759 secs 79 / 91

Slide 80

Slide 80 text

Really Large Projects Test subject: android-large-project (artiﬁcial!) Single line Java change --profile breakdown: 8.115s dex 6.662s javac 2.886s package 80 / 91

Slide 81

Slide 81 text

Utilizing Predexing We have 3 guavas in app/src/main/java, lets move two of them out settings.gradle: app/build.gradle: dependencies { compile project(':guava1') compile project(':guava2') } include include ':app', ':guava1', ':guava2' // guava3 still in :app! 81 / 91

Slide 82

Slide 82 text

Utilizing Predexing Single line Java change: $ ./gradlew :app:assembleDebug --profile BUILD SUCCESSFUL Total time: 9.592 secs Proﬁle output: 2.745s dex 2.362s javac 2.953s package 82 / 91

Slide 83

Slide 83 text

Multi-Project Builds Let's make a change in both guava1 and guava2 libraries. $ echo echo 'class A {}' >> guava1/src/main/java/A.java $ echo echo 'class B {}' >> guava2/src/main/java/B.java $ ./gradlew :app:assembleDebug BUILD SUCCESSFUL Total time: 16.722 secs Wait, what? Shouldn't this be around 19 seconds? 83 / 91

Slide 84

Slide 84 text

Multi-Project Builds javac in both guava1 and guava2 javac in app predex in app runs for guava1.jar and guava2.jar dex in app is UP-TO-DATE since the app classes haven't changed 84 / 91

Slide 85

Slide 85 text

Parallel Task Execution Speeding up the build by using the --parallel ﬂag. $ echo echo 'class A {}' >> guava1/src/main/java/A.java $ echo echo 'class B {}' >> guava2/src/main/java/B.java $ ./gradlew :app:assembleDebug --parallel BUILD SUCCESSFUL Total time: 14.303 secs Requires decoupled projects! 85 / 91

Slide 86

Slide 86 text

Incremental Javac compileJava { options.incremental = true } $ echo echo 'class A {}' >> guava1/src/main/java/A.java $ echo echo 'class B {}' >> guava2/src/main/java/B.java $ ./gradlew :app:assembleDebug --parallel BUILD SUCCESSFUL Total time: 11.37 secs 86 / 91

Slide 87

Slide 87 text

Incremental Javac Works only for pure Java projects (for now) http://b.android.com/82411 87 / 91

Slide 88

Slide 88 text

The Future Looks Brighter The gradle-experimental plugin will help with conﬁguration times The new Jack & Jill toolchain promises an incremental java->dex path 88 / 91

Slide 89

Slide 89 text

Alternative Build Systems BUCK, Bazel, Pants The builds are really fast IDE and community support not there yet Requires rather substantial restructuring of the project 89 / 91

Slide 90

Slide 90 text

JRebel for Android Relies on Gradle for resource processing and javac Makes the dex, package and install steps incremental Changes are applied into the already running VM http://bit.ly/jra-droidcon-gradle 90 / 91

Slide 91

Slide 91 text

Squeezing the Last Drop of Performance Out of Your Gradle Builds madis.pink@zeroturnaround.com http://twitter.com/madisp http://speakerdeck.com/madisp 91 / 91