Squeezing the Last Drop of Performance Out of Your Gradle Builds (Droidcon Paris 2015)

B5b43736709fb47edc3ee649618d84f7?s=47 Madis Pink
November 09, 2015

Squeezing the Last Drop of Performance Out of Your Gradle Builds (Droidcon Paris 2015)

This talk was given at Droidcon Paris 2015.

Recording on YouTube: https://www.youtube.com/watch?v=AbNhen_zn-c

As developers we spend way too much time waiting for things. We all know waiting is no fun and we’d rather spend time developing and testing our app. Sometimes we’re waiting on our integration test suite, sometimes it’s our app, but quite often Gradle is the time waster. The latter is often regarded as a black box that’s way slower than Ant/Eclipse ever was, and that’s saying something!

In this session we’ll see how having a deeper understanding of your Gradle build can help you identify major bottlenecks. We’ll look at how minimizing the configuration and dependency resolution overhead helps to reduce your cycle times when using host-based tools, like the new unit testing support, Robolectric or JRebel for Android. We’ll also explore ways of structuring your project in a way that takes advantage of Gradle incremental builds.

B5b43736709fb47edc3ee649618d84f7?s=128

Madis Pink

November 09, 2015
Tweet

Transcript

  1. 9.

    Gradle Build Lifecycle Execution - the useful part where Gradle

    builds your app Initialization, Configuration - the time Gradle spends on figuring out how to build your app, the overhead 9 / 91
  2. 11.

    Measuring the Overhead Test subject: Google IO app 28 libraries

    53149 method references http://github.com/google/iosched 11 / 91
  3. 12.

    Measuring the Overhead The --dry-run command line flag skips running

    all tasks $ ./gradlew :android:assembleDebug --dry-run ... :android:dexDebug SKIPPED :android:validateDebugSigning SKIPPED :android:packageDebug SKIPPED :android:zipalignDebug SKIPPED :android:assembleDebug SKIPPED BUILD SUCCESSFUL Total time: 4.256 secs 12 / 91
  4. 13.

    Measuring the Overhead You can use the --profile flag to

    get more information $ ./gradlew :android:assembleDebug --dry-run --profile $ open build/reports/profile/profile-2015-10-27-08-06-31.html 13 / 91
  5. 14.

    Configure on Demand Without --configure-on-demand --configure-on-demand: $ ./gradlew :android:assembleDebug --dry-run

    ... BUILD SUCCESSFUL Total time: 4.286 secs With --configure-on-demand --configure-on-demand: $ ./gradlew :android:assembleDebug --dry-run --configure-on-demand ... BUILD SUCCESSFUL Total time: 3.574 secs 14 / 91
  6. 15.

    Configure on Demand Enable configure on demand when invoking from

    the command line: echo echo 'org.gradle.configureondemand=true' >> \ ~/.gradle/gradle.properties 15 / 91
  7. 16.

    Configure on Demand Enable configure on demand when invoking from

    the command line: echo echo 'org.gradle.configureondemand=true' >> \ ~/.gradle/gradle.properties Enable for Studio in Preferences -> Build, Execution, Deployment -> Compiler 16 / 91
  8. 17.

    The Win So Far We've gone from 4.3 seconds to

    3.6 seconds A 17% win! 17 / 91
  9. 18.

    Gradle Daemon Without daemon: $ ./gradlew :android:assembleDebug --dry-run --no-daemon ...

    BUILD SUCCESSFUL Total time: 3.63 secs With daemon: $ ./gradlew :android:assembleDebug --dry-run --daemon ... BUILD SUCCESSFUL Total time: 1.709 secs 18 / 91
  10. 19.

    Gradle Daemon Always use the Gradle daemon for command line:

    echo echo 'org.gradle.daemon=true' >> ~/.gradle/gradle.properties 19 / 91
  11. 20.

    Gradle Daemon Always use the Gradle daemon for command line:

    echo echo 'org.gradle.daemon=true' >> ~/.gradle/gradle.properties For Android Studio, make sure that the Use in- process build option is set. You can find it in Preferences -> Build, Execution, Deployment -> Compiler 20 / 91
  12. 21.

    The Win So Far We've gone from 4.3 seconds to

    1.7 seconds A 60% win! 21 / 91
  13. 22.

    Gradle Version Gradle 2.2.1: $ ./gradlew :android:assembleDebug --dry-run ... BUILD

    SUCCESSFUL Total time: 1.536 secs Gradle 2.8: $ ./gradlew :android:assembleDebug --dry-run ... BUILD SUCCESSFUL Total time: 0.931 secs 22 / 91
  14. 23.

    Gradle Version Update gradle in gradle/wrapper/gradle- wrapper.properties #Tue Dec 02

    14:11:25 EST 2014 distributionBase=GRADLE_USER_HOME distributionPath=wrapper/dists zipStoreBase=GRADLE_USER_HOME zipStorePath=wrapper/dists distributionUrl=https\://services.gradle.org/distributions/gradle-2.8-all.zip 23 / 91
  15. 24.

    The Win So Far We've gone from 4.3 seconds to

    0.9 seconds A 78% win! 24 / 91
  16. 25.

    JVM Version Java 1.6 Java 1.8 $ export export JAVA_HOME=`/usr/libexec/java_home

    -v '1.6'` $ ./gradlew :android:assembleDebug --dry-run --configure-on-demand ... BUILD SUCCESSFUL Total time: 1.29 secs $ export export JAVA_HOME=`/usr/libexec/java_home -v '1.8'` $ ./gradlew :android:assembleDebug --dry-run --configure-on-demand ... BUILD SUCCESSFUL Total time: 0.871 secs 25 / 91
  17. 26.

    JVM Version Change the JDK used for Gradle in Android

    Studio. In File -> Project Structure: 26 / 91
  18. 27.

    I/O Gradle does tons of IO! $ ./gradlew :android:assembleDebug $

    strace -ff -o assemble.trace ./gradlew :android:assembleDebug $ grep -E "^(open|stat|fstat)" assemble.trace.* | wc -l 34479 27 / 91
  19. 28.

    Avoid Heavy Computation def def cmd cmd = 'git rev-list

    HEAD HEAD --first-parent --count' def def gitVersion gitVersion = cmd.execute().text.trim().toInteger() android { defaultConfig { versionCode gitVersion } } 28 / 91
  20. 29.

    Avoid Heavy Computation def def cmd cmd = 'git rev-list

    HEAD HEAD --first-parent --count' def def gitVersion gitVersion = cmd.execute().text.trim().toInteger() android { defaultConfig { versionCode gitVersion } } $ time git rev-list master --first-parent --count 23030 hub rev-list master --first-parent --count 0.21s user 0.05s system 53% cpu 0.470 total That's a 470ms overhead for every Gradle build 29 / 91
  21. 30.

    Avoid Heavy Computation def def gitVersion gitVersion() { if if

    (!System System.getenv('CI_BUILD')) { // don't care return return 1 } def def cmd cmd = 'git rev-list HEAD HEAD --first-parent --count' cmd.execute().text.trim().toInteger() } android { defaultConfig { versionCode gitVersion() } } 30 / 91
  22. 32.

    Dependencies dependencies { compile 'com.google.code.gson:gson:2.+' } $ date Tue Oct

    27 09:22:29 EET 2015 $ ./gradlew :android:help help BUILD SUCCESSFUL Total time: 0.693 secs 32 / 91
  23. 33.

    Dependencies dependencies { compile 'com.google.code.gson:gson:2.+' } $ date Tue Oct

    27 09:22:29 EET 2015 $ ./gradlew :android:help help BUILD SUCCESSFUL Total time: 0.693 secs $ date Wed Oct 28 09:23:19 EET 2015 $ ./gradlew :android:help help BUILD SUCCESSFUL Total time: 1.84 secs 33 / 91
  24. 34.

    Dependencies dependencies { compile 'com.google.code.gson:gson:2.+' } $ date Thu Oct

    29 09:29:06 EET 2015 $ ./gradlew :android:help help --offline BUILD SUCCESSFUL Total time: 0.756 secs 34 / 91
  25. 37.

    Continuous Builds $ time ./gradlew :android:test testDebugUnitTest BUILD SUCCESSFUL Total

    time: 0.859 secs 1.06s user 0.11s system 89% cpu 1.305 total 37 / 91
  26. 38.

    Continuous Builds $ time ./gradlew :android:test testDebugUnitTest --continuous BUILD SUCCESSFUL

    Total time: 0.858 secs Waiting for for changes to input files of tasks... (ctrl-d to exit exit 38 / 91
  27. 41.

    TL;DR Configuration on Demand Gradle Daemon (and/or the in-process build

    in AS) Newer versions of Gradle are faster 41 / 91
  28. 42.

    TL;DR Configuration on Demand Gradle Daemon (and/or the in-process build

    in AS) Newer versions of Gradle are faster JVM 1.8 is more performant than 1.6 42 / 91
  29. 43.

    TL;DR Configuration on Demand Gradle Daemon (and/or the in-process build

    in AS) Newer versions of Gradle are faster JVM 1.8 is more performant than 1.6 Avoid doing expensive things during the configuration phase 43 / 91
  30. 44.

    TL;DR Configuration on Demand Gradle Daemon (and/or the in-process build

    in AS) Newer versions of Gradle are faster JVM 1.8 is more performant than 1.6 Avoid doing expensive things during the configuration phase Don't use dynamic dependencies (x.y.+) 44 / 91
  31. 64.

    Build Breakdown - Iosched New file src/main/res/values/new_file.xml: <resources resources> <string

    string name="new_string">Hello!</string string> </resources resources> $ ./gradlew :android:assembleDebug --profile BUILD SUCCESSFUL Total time: 7.664 secs 64 / 91
  32. 66.

    Build Breakdown - Iosched Package cost: 1.297s Resource change cost:

    0.939s merge resources - 0.208s process resources (aapt) - 0.731s Java change cost: 4.462s dex - 3.766s javac - 0.876s 66 / 91
  33. 69.

    Changing Values android { buildTypes { debug { // don't

    care resValue 'string', 'build_date', 'dev-build' } release { resValue 'string', 'build_date', "${new Date()}" } } } 69 / 91
  34. 70.

    The Dexer Predexing For each library dependency, dex the classes

    into a .dex file and cache it. E.g.: gson:2.3 -> gson-2.3-9f28...2e3c.jar 70 / 91
  35. 71.

    The Dexer Predexing For each library dependency, dex the classes

    into a .dex file and cache it. E.g.: gson:2.3 -> gson-2.3-9f28...2e3c.jar Dexing Dex the application .class files, merge the resulting dex with the pre-dexed libraries. 71 / 91
  36. 73.

    The minSdk 21 Trick android { productFlavors { dev {

    multiDexEnabled true minSdkVersion 21 } prod {} } } 73 / 91
  37. 74.

    The minSdk 21 Trick $ ./gradlew :android:assembleDevDebug BUILD SUCCESSFUL Total

    time: 4.633 secs $ ./gradlew :android:assembleProdDebug BUILD SUCCESSFUL Total time: 6.599 secs 74 / 91
  38. 75.

    The minSdk 21 Trick android { productFlavors { dev {

    multiDexEnabled true minSdkVersion 21 } prod { multiDexEnabled true minSdkVersion 15 } } } 75 / 91
  39. 76.

    The minSdk 21 Trick $ ./gradlew :android:assembleDevDebug BUILD SUCCESSFUL Total

    time: 4.416 secs $ ./gradlew :android:assembleProdDebug BUILD SUCCESSFUL Total time: 20.703 secs 76 / 91
  40. 78.

    Really Large Projects Test subject: android-large-project (artificial!) Depends on 45

    libraries Source contains 3 copies of Guava Produces 3 dex files and 150640 method references One line Java change takes 50 seconds $ ./gradlew :app:assembleDebug BUILD SUCCESSFUL Total time: 49.695 secs 78 / 91
  41. 79.

    Really Large Projects One line Java change with minSdkVersion 21

    takes 19 seconds android { defaultConfig { minSdkVersion 21 multiDexEnabled true } } $ ./gradlew :app:assembleDebug BUILD SUCCESSFUL Total time: 18.759 secs 79 / 91
  42. 80.

    Really Large Projects Test subject: android-large-project (artificial!) Single line Java

    change --profile breakdown: 8.115s dex 6.662s javac 2.886s package 80 / 91
  43. 81.

    Utilizing Predexing We have 3 guavas in app/src/main/java, lets move

    two of them out settings.gradle: app/build.gradle: dependencies { compile project(':guava1') compile project(':guava2') } include include ':app', ':guava1', ':guava2' // guava3 still in :app! 81 / 91
  44. 82.

    Utilizing Predexing Single line Java change: $ ./gradlew :app:assembleDebug --profile

    BUILD SUCCESSFUL Total time: 9.592 secs Profile output: 2.745s dex 2.362s javac 2.953s package 82 / 91
  45. 83.

    Multi-Project Builds Let's make a change in both guava1 and

    guava2 libraries. $ echo echo 'class A {}' >> guava1/src/main/java/A.java $ echo echo 'class B {}' >> guava2/src/main/java/B.java $ ./gradlew :app:assembleDebug BUILD SUCCESSFUL Total time: 16.722 secs Wait, what? Shouldn't this be around 19 seconds? 83 / 91
  46. 84.

    Multi-Project Builds javac in both guava1 and guava2 javac in

    app predex in app runs for guava1.jar and guava2.jar dex in app is UP-TO-DATE since the app classes haven't changed 84 / 91
  47. 85.

    Parallel Task Execution Speeding up the build by using the

    --parallel flag. $ echo echo 'class A {}' >> guava1/src/main/java/A.java $ echo echo 'class B {}' >> guava2/src/main/java/B.java $ ./gradlew :app:assembleDebug --parallel BUILD SUCCESSFUL Total time: 14.303 secs Requires decoupled projects! 85 / 91
  48. 86.

    Incremental Javac compileJava { options.incremental = true } $ echo

    echo 'class A {}' >> guava1/src/main/java/A.java $ echo echo 'class B {}' >> guava2/src/main/java/B.java $ ./gradlew :app:assembleDebug --parallel BUILD SUCCESSFUL Total time: 11.37 secs 86 / 91
  49. 87.
  50. 88.

    The Future Looks Brighter The gradle-experimental plugin will help with

    configuration times The new Jack & Jill toolchain promises an incremental java->dex path 88 / 91
  51. 89.

    Alternative Build Systems BUCK, Bazel, Pants The builds are really

    fast IDE and community support not there yet Requires rather substantial restructuring of the project 89 / 91
  52. 90.

    JRebel for Android Relies on Gradle for resource processing and

    javac Makes the dex, package and install steps incremental Changes are applied into the already running VM http://bit.ly/jra-droidcon-gradle 90 / 91
  53. 91.

    Squeezing the Last Drop of Performance Out of Your Gradle

    Builds madis.pink@zeroturnaround.com http://twitter.com/madisp http://speakerdeck.com/madisp 91 / 91