Squeezing the Last Drop of Performance Out of Your Gradle Builds (Droidcon Paris 2015)

This talk was given at Droidcon Paris 2015.

Recording on YouTube: https://www.youtube.com/watch?v=AbNhen_zn-c

As developers we spend way too much time waiting for things. We all know waiting is no fun and we’d rather spend time developing and testing our app. Sometimes we’re waiting on our integration test suite, sometimes it’s our app, but quite often Gradle is the time waster. The latter is often regarded as a black box that’s way slower than Ant/Eclipse ever was, and that’s saying something!

In this session we’ll see how having a deeper understanding of your Gradle build can help you identify major bottlenecks. We’ll look at how minimizing the configuration and dependency resolution overhead helps to reduce your cycle times when using host-based tools, like the new unit testing support, Robolectric or JRebel for Android. We’ll also explore ways of structuring your project in a way that takes advantage of Gradle incremental builds.

Transcript

1. source: https://xkcd.com/303/ 1 / 91
2. Squeezing the Last Drop of Performance Out of Your Gradle Builds 2 / 91
3. Obligatory Disclaimer - YMMV Your Mileage May Vary 3 / 91
4. Always Measure First 4 / 91
5. Gradle Build Lifecycle Three distinct phases: 1. Initialization 5 / 91
6. Gradle Build Lifecycle Three distinct phases: 1. Initialization 2. Configuration 6 / 91
7. Gradle Build Lifecycle Three distinct phases: 1. Initialization 2. Configuration 3. Execution 7 / 91
8. Gradle Build Lifecycle Execution - the useful part where Gradle builds your app 8 / 91
9. Gradle Build Lifecycle Execution - the useful part where Gradle builds your app Initialization, Configuration - the time Gradle spends on figuring out how to build your app, the overhead 9 / 91
10. Initialization and Configuration The overhead tax you have to pay whenever you invoke Gradle 10 / 91
11. Measuring the Overhead Test subject: Google IO app 28 libraries 53149 method references http://github.com/google/iosched 11 / 91
12. Measuring the Overhead The --dry-run command line flag skips running all tasks $ ./gradlew :android:assembleDebug --dry-run ... :android:dexDebug SKIPPED :android:validateDebugSigning SKIPPED :android:packageDebug SKIPPED :android:zipalignDebug SKIPPED :android:assembleDebug SKIPPED BUILD SUCCESSFUL Total time: 4.256 secs 12 / 91
13. Measuring the Overhead You can use the --profile flag to get more information $ ./gradlew :android:assembleDebug --dry-run --profile $ open build/reports/profile/profile-2015-10-27-08-06-31.html 13 / 91
14. Configure on Demand Without --configure-on-demand --configure-on-demand: $ ./gradlew :android:assembleDebug --dry-run ... BUILD SUCCESSFUL Total time: 4.286 secs With --configure-on-demand --configure-on-demand: $ ./gradlew :android:assembleDebug --dry-run --configure-on-demand ... BUILD SUCCESSFUL Total time: 3.574 secs 14 / 91
15. Configure on Demand Enable configure on demand when invoking from the command line: echo echo 'org.gradle.configureondemand=true' >> \ ~/.gradle/gradle.properties 15 / 91
16. Configure on Demand Enable configure on demand when invoking from the command line: echo echo 'org.gradle.configureondemand=true' >> \ ~/.gradle/gradle.properties Enable for Studio in Preferences -> Build, Execution, Deployment -> Compiler 16 / 91
17. The Win So Far We've gone from 4.3 seconds to 3.6 seconds A 17% win! 17 / 91
18. Gradle Daemon Without daemon: $ ./gradlew :android:assembleDebug --dry-run --no-daemon ... BUILD SUCCESSFUL Total time: 3.63 secs With daemon: $ ./gradlew :android:assembleDebug --dry-run --daemon ... BUILD SUCCESSFUL Total time: 1.709 secs 18 / 91
19. Gradle Daemon Always use the Gradle daemon for command line: echo echo 'org.gradle.daemon=true' >> ~/.gradle/gradle.properties 19 / 91
20. Gradle Daemon Always use the Gradle daemon for command line: echo echo 'org.gradle.daemon=true' >> ~/.gradle/gradle.properties For Android Studio, make sure that the Use in- process build option is set. You can find it in Preferences -> Build, Execution, Deployment -> Compiler 20 / 91
21. The Win So Far We've gone from 4.3 seconds to 1.7 seconds A 60% win! 21 / 91
22. Gradle Version Gradle 2.2.1: $ ./gradlew :android:assembleDebug --dry-run ... BUILD SUCCESSFUL Total time: 1.536 secs Gradle 2.8: $ ./gradlew :android:assembleDebug --dry-run ... BUILD SUCCESSFUL Total time: 0.931 secs 22 / 91
23. Gradle Version Update gradle in gradle/wrapper/gradle- wrapper.properties #Tue Dec 02 14:11:25 EST 2014 distributionBase=GRADLE_USER_HOME distributionPath=wrapper/dists zipStoreBase=GRADLE_USER_HOME zipStorePath=wrapper/dists distributionUrl=https\://services.gradle.org/distributions/gradle-2.8-all.zip 23 / 91
24. The Win So Far We've gone from 4.3 seconds to 0.9 seconds A 78% win! 24 / 91
25. JVM Version Java 1.6 Java 1.8 $ export export JAVA_HOME=`/usr/libexec/java_home -v '1.6'` $ ./gradlew :android:assembleDebug --dry-run --configure-on-demand ... BUILD SUCCESSFUL Total time: 1.29 secs $ export export JAVA_HOME=`/usr/libexec/java_home -v '1.8'` $ ./gradlew :android:assembleDebug --dry-run --configure-on-demand ... BUILD SUCCESSFUL Total time: 0.871 secs 25 / 91
26. JVM Version Change the JDK used for Gradle in Android Studio. In File -> Project Structure: 26 / 91
27. I/O Gradle does tons of IO! $ ./gradlew :android:assembleDebug $ strace -ff -o assemble.trace ./gradlew :android:assembleDebug $ grep -E "^(open|stat|fstat)" assemble.trace.* | wc -l 34479 27 / 91
28. Avoid Heavy Computation def def cmd cmd = 'git rev-list HEAD HEAD --first-parent --count' def def gitVersion gitVersion = cmd.execute().text.trim().toInteger() android { defaultConfig { versionCode gitVersion } } 28 / 91
29. Avoid Heavy Computation def def cmd cmd = 'git rev-list HEAD HEAD --first-parent --count' def def gitVersion gitVersion = cmd.execute().text.trim().toInteger() android { defaultConfig { versionCode gitVersion } } $ time git rev-list master --first-parent --count 23030 hub rev-list master --first-parent --count 0.21s user 0.05s system 53% cpu 0.470 total That's a 470ms overhead for every Gradle build 29 / 91
30. Avoid Heavy Computation def def gitVersion gitVersion() { if if (!System System.getenv('CI_BUILD')) { // don't care return return 1 } def def cmd cmd = 'git rev-list HEAD HEAD --first-parent --count' cmd.execute().text.trim().toInteger() } android { defaultConfig { versionCode gitVersion() } } 30 / 91
31. Dependencies dependencies { compile 'com.google.code.gson:gson:2.+' } 31 / 91
32. Dependencies dependencies { compile 'com.google.code.gson:gson:2.+' } $ date Tue Oct 27 09:22:29 EET 2015 $ ./gradlew :android:help help BUILD SUCCESSFUL Total time: 0.693 secs 32 / 91
33. Dependencies dependencies { compile 'com.google.code.gson:gson:2.+' } $ date Tue Oct 27 09:22:29 EET 2015 $ ./gradlew :android:help help BUILD SUCCESSFUL Total time: 0.693 secs $ date Wed Oct 28 09:23:19 EET 2015 $ ./gradlew :android:help help BUILD SUCCESSFUL Total time: 1.84 secs 33 / 91
34. Dependencies dependencies { compile 'com.google.code.gson:gson:2.+' } $ date Thu Oct 29 09:29:06 EET 2015 $ ./gradlew :android:help help --offline BUILD SUCCESSFUL Total time: 0.756 secs 34 / 91
35. Dependencies dependencies { compile 'com.google.code.gson:gson:2.3' } 35 / 91
36. Dependencies dependencies { compile 'com.google.code.gson:gson:2.3' } Reproducible builds, yaaay \o/ 36 / 91
37. Continuous Builds $ time ./gradlew :android:test testDebugUnitTest BUILD SUCCESSFUL Total time: 0.859 secs 1.06s user 0.11s system 89% cpu 1.305 total 37 / 91
38. Continuous Builds $ time ./gradlew :android:test testDebugUnitTest --continuous BUILD SUCCESSFUL Total time: 0.858 secs Waiting for for changes to input files of tasks... (ctrl-d to exit exit 38 / 91
39. TL;DR Configuration on Demand 39 / 91
40. TL;DR Configuration on Demand Gradle Daemon (and/or the in-process build in AS) 40 / 91
41. TL;DR Configuration on Demand Gradle Daemon (and/or the in-process build in AS) Newer versions of Gradle are faster 41 / 91
42. TL;DR Configuration on Demand Gradle Daemon (and/or the in-process build in AS) Newer versions of Gradle are faster JVM 1.8 is more performant than 1.6 42 / 91
43. TL;DR Configuration on Demand Gradle Daemon (and/or the in-process build in AS) Newer versions of Gradle are faster JVM 1.8 is more performant than 1.6 Avoid doing expensive things during the configuration phase 43 / 91
44. TL;DR Configuration on Demand Gradle Daemon (and/or the in-process build in AS) Newer versions of Gradle are faster JVM 1.8 is more performant than 1.6 Avoid doing expensive things during the configuration phase Don't use dynamic dependencies (x.y.+) 44 / 91
45. Execution Phase All of this is cool but my builds are still slow? 45 / 91
46. Task Graph $ ./gradlew :jar 46 / 91
47. Task Graph $ ./gradlew :jar :jar 47 / 91
48. Task Graph $ ./gradlew :jar :jar :classes 48 / 91
49. Task Graph $ ./gradlew :jar :jar :classes :compileJava 49 / 91
50. Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 50 / 91
51. Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 51 / 91
52. Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 52 / 91
53. Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 53 / 91
54. Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 54 / 91
55. Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 55 / 91
56. Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 56 / 91
57. Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 57 / 91
58. Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 58 / 91
59. Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 59 / 91
60. Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 60 / 91
61. Task Graph $ ./gradlew :jar :jar :classes :compileJava :processResources 61 / 91
62. Task Graph 62 / 91
63. Task Graph :package :dex :compileJava :processResources res.ap_ R.java *.class classes.dex 63 / 91
64. Build Breakdown - Iosched New file src/main/res/values/new_file.xml: <resources resources> <string string name="new_string">Hello!</string string> </resources resources> $ ./gradlew :android:assembleDebug --profile BUILD SUCCESSFUL Total time: 7.664 secs 64 / 91
65. Build Breakdown - Iosched 65 / 91
66. Build Breakdown - Iosched Package cost: 1.297s Resource change cost: 0.939s merge resources - 0.208s process resources (aapt) - 0.731s Java change cost: 4.462s dex - 3.766s javac - 0.876s 66 / 91
67. Changing Values android { defaultConfig { buildConfigField('String', 'BUILD_DATE', "\"${new Date()} ${new Date()}\"") } } 67 / 91
68. Changing Values android { defaultConfig { resValue 'string', 'build_date', "${new Date()} ${new Date()}" } } 68 / 91
69. Changing Values android { buildTypes { debug { // don't care resValue 'string', 'build_date', 'dev-build' } release { resValue 'string', 'build_date', "${new Date()}" } } } 69 / 91
70. The Dexer Predexing For each library dependency, dex the classes into a .dex file and cache it. E.g.: gson:2.3 -> gson-2.3-9f28...2e3c.jar 70 / 91
71. The Dexer Predexing For each library dependency, dex the classes into a .dex file and cache it. E.g.: gson:2.3 -> gson-2.3-9f28...2e3c.jar Dexing Dex the application .class files, merge the resulting dex with the pre-dexed libraries. 71 / 91
72. The Dexer :dex :javaCompile :preDex *.class lib1.dex, lib2.dex, etc 72 / 91
73. The minSdk 21 Trick android { productFlavors { dev { multiDexEnabled true minSdkVersion 21 } prod {} } } 73 / 91
74. The minSdk 21 Trick $ ./gradlew :android:assembleDevDebug BUILD SUCCESSFUL Total time: 4.633 secs $ ./gradlew :android:assembleProdDebug BUILD SUCCESSFUL Total time: 6.599 secs 74 / 91
75. The minSdk 21 Trick android { productFlavors { dev { multiDexEnabled true minSdkVersion 21 } prod { multiDexEnabled true minSdkVersion 15 } } } 75 / 91
76. The minSdk 21 Trick $ ./gradlew :android:assembleDevDebug BUILD SUCCESSFUL Total time: 4.416 secs $ ./gradlew :android:assembleProdDebug BUILD SUCCESSFUL Total time: 20.703 secs 76 / 91
77. Really Large Projects Build times in the order of minutes 77 / 91
78. Really Large Projects Test subject: android-large-project (artificial!) Depends on 45 libraries Source contains 3 copies of Guava Produces 3 dex files and 150640 method references One line Java change takes 50 seconds $ ./gradlew :app:assembleDebug BUILD SUCCESSFUL Total time: 49.695 secs 78 / 91
79. Really Large Projects One line Java change with minSdkVersion 21 takes 19 seconds android { defaultConfig { minSdkVersion 21 multiDexEnabled true } } $ ./gradlew :app:assembleDebug BUILD SUCCESSFUL Total time: 18.759 secs 79 / 91
80. Really Large Projects Test subject: android-large-project (artificial!) Single line Java change --profile breakdown: 8.115s dex 6.662s javac 2.886s package 80 / 91
81. Utilizing Predexing We have 3 guavas in app/src/main/java, lets move two of them out settings.gradle: app/build.gradle: dependencies { compile project(':guava1') compile project(':guava2') } include include ':app', ':guava1', ':guava2' // guava3 still in :app! 81 / 91
82. Utilizing Predexing Single line Java change: $ ./gradlew :app:assembleDebug --profile BUILD SUCCESSFUL Total time: 9.592 secs Profile output: 2.745s dex 2.362s javac 2.953s package 82 / 91
83. Multi-Project Builds Let's make a change in both guava1 and guava2 libraries. $ echo echo 'class A {}' >> guava1/src/main/java/A.java $ echo echo 'class B {}' >> guava2/src/main/java/B.java $ ./gradlew :app:assembleDebug BUILD SUCCESSFUL Total time: 16.722 secs Wait, what? Shouldn't this be around 19 seconds? 83 / 91
84. Multi-Project Builds javac in both guava1 and guava2 javac in app predex in app runs for guava1.jar and guava2.jar dex in app is UP-TO-DATE since the app classes haven't changed 84 / 91
85. Parallel Task Execution Speeding up the build by using the --parallel flag. $ echo echo 'class A {}' >> guava1/src/main/java/A.java $ echo echo 'class B {}' >> guava2/src/main/java/B.java $ ./gradlew :app:assembleDebug --parallel BUILD SUCCESSFUL Total time: 14.303 secs Requires decoupled projects! 85 / 91
86. Incremental Javac compileJava { options.incremental = true } $ echo echo 'class A {}' >> guava1/src/main/java/A.java $ echo echo 'class B {}' >> guava2/src/main/java/B.java $ ./gradlew :app:assembleDebug --parallel BUILD SUCCESSFUL Total time: 11.37 secs 86 / 91
87. Incremental Javac Works only for pure Java projects (for now) http://b.android.com/82411 87 / 91
88. The Future Looks Brighter The gradle-experimental plugin will help with configuration times The new Jack & Jill toolchain promises an incremental java->dex path 88 / 91
89. Alternative Build Systems BUCK, Bazel, Pants The builds are really fast IDE and community support not there yet Requires rather substantial restructuring of the project 89 / 91
90. JRebel for Android Relies on Gradle for resource processing and javac Makes the dex, package and install steps incremental Changes are applied into the already running VM http://bit.ly/jra-droidcon-gradle 90 / 91
91. Squeezing the Last Drop of Performance Out of Your Gradle Builds madis.pink@zeroturnaround.com http://twitter.com/madisp http://speakerdeck.com/madisp 91 / 91