#AndroidBuilds at Twitter

https://www.youtube.com/watch?v=rG6nzrRPMRI

As codebases become larger and more complex, build performance is impacted. At Twitter, we’ve seen the existing tools struggle in a project with more than 100 modules and 1 million lines of code. This has forced us to explore a number of solutions to maintain good build speeds, and minimize resource usage.

We’ll share strategies that we’ve employed that were successful, and things that weren’t - to avoid hurting developer productivity.

These tips include:

- Strategies to effectively break down larger modules
- How to best utilize build tooling
- Profiling techniques to help find bottlenecks

Transcript

1. #AndroidBuilds @m_evans10 @CesarDielo
2. Who is happy with build times? Problem?
3. Don't think of them as Gradle build times. Think of them as job security for your Twitter account 13 72 2:41 PM - 17 Jun 2017 Lisa Wray @lisawrayz
4. 54 Modules • Mostly inside large app module, and one large library module • AGP 2.2.2 • Gradle 3.1 (In October 2016) Our build Seconds 0 75 150 225 300 Clean Incremental
5. 35 50 30 Seconds/build Builds/day Developers Running the numbers X X
6. >14.5 Hours Spent Waiting On Gradle Every Single Day
7. "I always see you just staring at your screen. You Android devs must have it easy." I AM WAITING FOR GRADLE OKAY 66 144 3:03 AM - 28 Apr 2017 Zarah Dominguez 🦉 @zarahjutz
8. (and OkBuck) But…Buck?
9. Very excited about fi nally switching entirely to Buck soon and never going back to Gradle. Subsecond builds are coming 😂 1 6 2:31 PM - 4 Nov 2016 Felipe Lima @felipecsl
10. Buck • Super fast • OkBuck plugin makes it super easy to get started • Exopackage is like Instant Run (but works)
11. Seconds 0 75 150 225 300 Gradle 2.3.3 Buck Clean Incremental
12. But…
13. Buck • Less fl exible • Not o ff i cially supported • High maintenance cost • No Kotlin support
14. Easy Improvements Speeding Up Gradle
15. Follow best practices
16. Avoid Legacy Multidex 1 Some Tips from James Lau Google IO 2017
17. Avoid Legacy Multidex Disable Multi-APK 1 2 Some Tips from James Lau Google IO 2017
18. Avoid Legacy Multidex Disable Multi-APK Disable PNG Crunching 1 2 3 Some Tips from James Lau Google IO 2017
19. Avoid Legacy Multidex Disable Multi-APK Disable PNG Crunching 1 2 3 Avoid inadvertent changes 4 Some Tips from James Lau Google IO 2017
20. Avoid Legacy Multidex Disable Multi-APK Disable PNG Crunching 1 2 3 Avoid inadvertent changes Watch the memory 4 5 Some Tips from James Lau Google IO 2017
21. Avoid Legacy Multidex Disable Multi-APK Disable PNG Crunching 1 2 3 Avoid inadvertent changes Watch the memory Enable Gradle caching 4 5 6 Some Tips from James Lau Google IO 2017
22. More simple improvements Other Tips
23. More simple improvements Other Tips • Enable Parallel Builds • --parallel
24. More simple improvements Other Tips • Enable Parallel Builds • --parallel • Con fi gure on demand • --configure-on-demand • org.gradle.configureondemand=true
25. More simple improvements Other Tips • Enable Parallel Builds • --parallel • Con fi gure on demand • --configure-on-demand • org.gradle.configureondemand=true • Partial builds (-a) • Ignore all dependent projects and only execute tasks in the target project
26. Leveraging Gradle features
27. Leveraging Gradle features 3.0 Daemon 3.1 Faster Dependency Resolution 3.4 Compile Avoidance Incremental Compilation 4.0 Build Cache
28. 3.0 Daemon 3.1 Faster Dependency Resolution 3.4 Compile Avoidance Incremental Compilation 4.0 Build Cache Leveraging Gradle features
29. 3.0 Daemon 3.1 Faster Dependency Resolution 3.4 Compile Avoidance Incremental Compilation 4.0 Build Cache Leveraging Gradle features
30. Compiler avoidance In Java projects… • Replace java with java-library • Use the api and implementation configurations • Declare annotation processors in the processor classpath
31. Compiler avoidance In Java projects… • Replace java with java-library • Use the api and implementation configurations • Declare annotation processors in the processor classpath
32. Compiler avoidance In Java projects… • Replace java with java-library • Use the api and implementation configurations • Declare annotation processors in the processor classpath
33. Compiler avoidance In Java projects… compile ➔ api/implementation testCompile ➔ testImplementation
34. Compiler avoidance In Java projects… • Replace java with java-library • Use the api and implementation configurations • Declare annotation processors in the processor classpath
35. Compiler avoidance In Java projects… configurations { // A configuration that collects annotation processing dependencies // to add to the annotation processsor path. annotationProcessor } compileJava { options.annotationProcessorPath = configurations.annotationProcessor } dependencies { annotationProcessor 'com.google.dagger:dagger-compiler:2.11' }
36. Compiler avoidance In Java projects… configurations { // A configuration that collects annotation processing dependencies // to add to the annotation processsor path. annotationProcessor } compileJava { options.annotationProcessorPath = configurations.annotationProcessor } dependencies { annotationProcessor 'com.google.dagger:dagger-compiler:2.11' }
37. Compiler avoidance In Java projects… configurations { // A configuration that collects annotation processing dependencies // to add to the annotation processsor path. annotationProcessor } compileJava { options.annotationProcessorPath = configurations.annotationProcessor } dependencies { annotationProcessor 'com.google.dagger:dagger-compiler:2.11' }
38. Compiler avoidance In Java projects… configurations { // A configuration that collects annotation processing dependencies // to add to the annotation processsor path. annotationProcessor } compileJava { options.annotationProcessorPath = configurations.annotationProcessor } dependencies { annotationProcessor 'com.google.dagger:dagger-compiler:2.11' }
39. Incremental compilation In Java projects… tasks.withType(JavaCompile) { options.incremental = true }
40. 3.0 Daemon 3.1 Faster Dependency Resolution 3.4 Compile Avoidance Incremental Compilation 4.0 Build Cache Leveraging Gradle features
41. 3.0 Daemon 3.1 Faster Dependency Resolution 3.4 Compile Avoidance Incremental Compilation 4.0 Build Cache Leveraging Gradle features
42. Enabling the build cache Enable cache in gradle.properties org.gradle.caching=true Con fi gure cache in settings.gradle buildCache { local(DirectoryBuildCache) { directory = System.env.GRADLE_USER_HOME != null ? new File(System.env.GRADLE_USER_HOME, '.buildcache') : new File(rootDir, '.buildcache') enabled = true } remote(HttpBuildCache) { url = 'https://ge.twitter.biz/cache/' push = true enabled = true } }
43. Enable cache in gradle.properties org.gradle.caching=true Con fi gure cache in settings.gradle buildCache { local(DirectoryBuildCache) { directory = System.env.GRADLE_USER_HOME != null ? new File(System.env.GRADLE_USER_HOME, '.buildcache') : new File(rootDir, '.buildcache') enabled = true } remote(HttpBuildCache) { url = 'https://ge.twitter.biz/cache/' push = true enabled = true } } Enabling the build cache
44. Enabling the build cache Enable cache in gradle.properties org.gradle.caching=true Con fi gure cache in settings.gradle buildCache { local(DirectoryBuildCache) { directory = System.env.GRADLE_USER_HOME != null ? new File(System.env.GRADLE_USER_HOME, '.buildcache') : new File(rootDir, '.buildcache') enabled = true } remote(HttpBuildCache) { url = 'https://ge.twitter.biz/cache/' push = true enabled = true } }
45. The right tool for the right job Learn the tools
46. Give me six hours to chop down a tree and I will spend the fi rst four sharpening the axe 16 68 2:48 PM – 15 Jul 1860 Abraham Lincoln @POTUS16
47. --profile Is Your Friend
48. --profile is your friend • Generates a report in build/reports/profile • Lists timing for con fi guration and task execution • Indicates if tasks were skipped/did no work
53. Gradle Enterprise (aka Gradle Build Scans)
54. Gradle Enterprise apply plugin: ‘com.gradle.build-scan' buildScan { licenseAgreementUrl = 'https://gradle.com/terms-of-service' licenseAgree = 'yes' }
55. Gradle Enterprise gradle build --scan
56. Gradle Enterprise
57. Gradle Enterprise
58. Gradle Enterprise
61. Gradle Enterprise
62. Gradle Enterprise
63. The Gradle Profiler
64. The Gradle profiler https://github.com/gradle/gradle-profiler
65. The Gradle profiler ./gradle-profiler \
 --profile chrome-trace \
 --project-dir ~/src/twitter-android/ \
 --scenario-file tfa.scenarios \
 nonAbiChange
66. The Gradle profiler ./gradle-profiler \
 --profile chrome-trace \
 --project-dir ~/src/twitter-android/ \
 --scenario-file tfa.scenarios \
 nonAbiChange
67. The Gradle profiler ./gradle-profiler \
 --profile chrome-trace \
 --project-dir ~/src/twitter-android/ \
 --scenario-file tfa.scenarios \
 nonAbiChange
68. The Gradle profiler nonAbiChange { tasks = [":app:twitter:assembleDefaultDebug"] apply-non-abi-change-to = "lib/twitter/Example.java" gradle-args = ["-PminSdkVersion=23"] warm-ups = 3 } tfa.scenarios
69. The Gradle profiler nonAbiChange { tasks = [":app:twitter:assembleDefaultDebug"] apply-non-abi-change-to = "lib/twitter/Example.java" gradle-args = ["-PminSdkVersion=23"] warm-ups = 3 } tfa.scenarios
70. The Gradle profiler nonAbiChange { tasks = [":app:twitter:assembleDefaultDebug"] apply-non-abi-change-to = "lib/twitter/Example.java" gradle-args = ["-PminSdkVersion=23"] warm-ups = 3 } tfa.scenarios
71. The Gradle profiler nonAbiChange { tasks = [":app:twitter:assembleDefaultDebug"] apply-non-abi-change-to = "lib/twitter/Example.java" gradle-args = ["-PminSdkVersion=23"] warm-ups = 3 } tfa.scenarios
72. The Gradle profiler nonAbiChange { tasks = [":app:twitter:assembleDefaultDebug"] apply-non-abi-change-to = "lib/twitter/Example.java" gradle-args = ["-PminSdkVersion=23"] warm-ups = 3 } tfa.scenarios
73. The Gradle profiler
74. The Gradle profiler
75. The Gradle profiler
76. The Gradle profiler
77. YourKit
78. YourKit changed my life. 4:12 PM - 18 Apr 2017 Gwen (Chen) Shapira @gwenshap
79. YourKit
80. YourKit
81. YourKit
82. YourKit
83. YourKit https://www.yourkit.com/docs/java/help/agent.jsp https://www.yourkit.com/docs/java/help/startup_options.jsp org.gradle.jvmargs=-Xms512m -Xmx4g -Dfile.encoding=utf-8 -Djava.awt.headless=true \ -agentpath:/Applications/YourKit.app/Contents/Resources/bin/mac/libyjpagent.jnilib=sampling,onexit=memory org.gradle.daemon=false org.gradle.parallel=true org.gradle.configureondemand=true gradle.properties
84. YourKit https://www.yourkit.com/docs/java/help/agent.jsp https://www.yourkit.com/docs/java/help/startup_options.jsp org.gradle.jvmargs=-Xms512m -Xmx4g -Dfile.encoding=utf-8 -Djava.awt.headless=true \ -agentpath:/Applications/YourKit.app/Contents/Resources/bin/mac/libyjpagent.jnilib=sampling,onexit=memory org.gradle.daemon=false org.gradle.parallel=true org.gradle.configureondemand=true gradle.properties
85. YourKit https://www.yourkit.com/docs/java/help/agent.jsp https://www.yourkit.com/docs/java/help/startup_options.jsp org.gradle.jvmargs=-Xms512m -Xmx4g -Dfile.encoding=utf-8 -Djava.awt.headless=true \ -agentpath:/Applications/YourKit.app/Contents/Resources/bin/mac/libyjpagent.jnilib=sampling,onexit=memory org.gradle.daemon=false org.gradle.parallel=true org.gradle.configureondemand=true gradle.properties
86. YourKit Or use the Gradle profiler! ./gradle-profiler \
 --profile yourkit --yourkit-sampling \
 --project-dir ~/src/twitter-android/ \
 --scenario-file tfa.scenarios \
 nonAbiChange
87. YourKit Or use the Gradle profiler! ./gradle-profiler \
 --profile yourkit --yourkit-sampling \
 --project-dir ~/src/twitter-android/ \
 --scenario-file tfa.scenarios \
 nonAbiChange
88. Use the logs!
89. Use the logs! Why is your compilation not doing the right thing? ➡ check out the command-line parameters and the classpath ./gradlew assemble --info
90. Use the logs! Why is your compilation not doing the right thing? ➡ check out the command-line parameters and the classpath Compiler arguments: -source 1.7 -target 1.7 -d /Users/ mevans/src/twitter-android/lib/core/database/lru/build/ intermediates/classes/release -encoding UTF-8 -bootclasspath /opt/twitter/Caskroom/android-sdk/25.2.3/ platforms/android-25/android.jar -g -sourcepath -processorpath /Users/mevans/.gradle/caches/modules-2/ files-2.1/javax.inject/javax.inject/ 1/6975da39a7040257bd51d21a231b76c915872d38/ javax.inject-1.jar… -classpath /Users/mevans/src/twitter-android/lib/core/util/ android/common/build/intermediates/bundles/default/ classes.jar…
91. Use the logs! Why is your compilation not doing the right thing? ➡ check out the command-line parameters and the classpath Compiler arguments: -source 1.7 -target 1.7 -d /Users/ mevans/src/twitter-android/lib/core/database/lru/build/ intermediates/classes/release -encoding UTF-8 -bootclasspath /opt/twitter/Caskroom/android-sdk/25.2.3/ platforms/android-25/android.jar -g -sourcepath -processorpath /Users/mevans/.gradle/caches/modules-2/ files-2.1/javax.inject/javax.inject/ 1/6975da39a7040257bd51d21a231b76c915872d38/ javax.inject-1.jar… -classpath /Users/mevans/src/twitter-android/lib/core/util/ android/common/build/intermediates/bundles/default/ classes.jar…
92. Putting the tools to use Beyond the basics
93. Upgrading to AGP 3.0
94. Android Gradle Plugin optimizations 2.2 Pre-dexing 2.3 Gradle Build Cache 3.0 Compile Avoidance Variant-aware dependency Management …
95. Android Gradle Plugin optimizations 2.2 Pre-dexing 2.3 Gradle Build Cache 3.0 Compile Avoidance Variant-aware dependency Management …
96. AGP 3.0 Trying the plugin early • Enable with a flag • Define flavor dimensions • Disable aapt2 and new resource processing • Define configurations for 2.3
97. AGP 3.0 Trying the plugin early • Enable with a flag • Define flavor dimensions • Disable aapt2 and new resource processing • Define configurations for 2.3
98. AGP 3.0 Trying the plugin early ext.isNewBuild = System.properties['build.new'] != null buildscript { repositories { google() } dependencies { classpath isNewBuild ? 'com.android.tools.build:gradle:3.0.0-alpha4' : 'com.android.tools.build:gradle:2.3.0' } }
99. AGP 3.0 Trying the plugin early ext.isNewBuild = System.properties['build.new'] != null buildscript { repositories { google() } dependencies { classpath isNewBuild ? 'com.android.tools.build:gradle:3.0.0-alpha4' : 'com.android.tools.build:gradle:2.3.0' } }
100. AGP 3.0 Trying the plugin early ext.isNewBuild = System.properties['build.new'] != null buildscript { repositories { google() } dependencies { classpath isNewBuild ? 'com.android.tools.build:gradle:3.0.0-alpha4' : 'com.android.tools.build:gradle:2.3.0' } }
101. AGP 3.0 Trying the plugin early • Enable with a flag • Define flavor dimensions • Disable aapt2 and new resource processing • Define configurations for 2.3
102. AGP 3.0 Trying the plugin early android { if (isNewBuild) { flavorDimensions 'type' } productFlavors { 'default' { if (isNewBuild) { dimension 'type' } } } }
103. AGP 3.0 Trying the plugin early • Enable with a flag • Define flavor dimensions • Disable aapt2 and new resource processing • Define configurations for 2.3
104. AGP 3.0 Trying the plugin early Disable aapt2 and resource processing in gradle.properties android.enableNewResourceProcessing=false android.enableAapt2=false
105. AGP 3.0 Trying the plugin early • Enable with a flag • Define flavor dimensions • Disable aapt2 and new resource processing • Define configurations for 2.3
106. AGP 3.0 if (!isNewBuild) { configurations { // Aliases for the new configurations in 3.0.0. api implementation compile.extendsFrom(api, implementation) debugImplementation debugCompile.extendsFrom(debugImplementation) dogfoodImplementation dogfoodCompile.extendsFrom(dogfoodImplementation) testImplementation testCompile.extendsFrom(testImplementation) androidTestImplementation androidTestCompile.extendsFrom(androidTestImplementation) compileOnly provided.extendsFrom(compileOnly) } } Trying the plugin early
107. AGP 3.0 if (!isNewBuild) { configurations { // Aliases for the new configurations in 3.0.0. api implementation compile.extendsFrom(api, implementation) debugImplementation debugCompile.extendsFrom(debugImplementation) dogfoodImplementation dogfoodCompile.extendsFrom(dogfoodImplementation) testImplementation testCompile.extendsFrom(testImplementation) androidTestImplementation androidTestCompile.extendsFrom(androidTestImplementation) compileOnly provided.extendsFrom(compileOnly) } } Trying the plugin early
108. AGP 3.0 if (!isNewBuild) { configurations { // Aliases for the new configurations in 3.0.0. api implementation compile.extendsFrom(api, implementation) debugImplementation debugCompile.extendsFrom(debugImplementation) dogfoodImplementation dogfoodCompile.extendsFrom(dogfoodImplementation) testImplementation testCompile.extendsFrom(testImplementation) androidTestImplementation androidTestCompile.extendsFrom(androidTestImplementation) compileOnly provided.extendsFrom(compileOnly) } } Trying the plugin early
109. Up-to-date Check Issues
110. Up-to-date check issues Why is this task running if nothing changed?
111. Up-to-date check issues SkipUpToDateTaskExecuter in the logs Determining if task ':lib:core:util:android:common:extractReleaseAnnotations' is up-to-date Executing task ':lib:core:util:android:common:extractReleaseAnnotations' (up-to-date check took 0.012 secs) due to: Input property 'classDir' file /Users/cpuerta/workspace/twitter-android/lib/core/util/android/common/ build/intermediates/classes/release/android/support/v14/preference/R$id.class has changed. Input property 'classDir' file /Users/cpuerta/workspace/twitter-android/lib/core/util/android/common/ build/intermediates/classes/release/android/support/v7/appcompat/R$id.class has changed. Input property 'classDir' file /Users/cpuerta/workspace/twitter-android/lib/core/util/android/common/ build/intermediates/classes/release/android/support/v7/preference/R$id.class has changed. Determining if task ':lib:core:util:android:common:mergeReleaseAssets' is up-to-date Skipping task ':lib:core:util:android:common:mergeReleaseAssets' as it is up-to-date (took 0.003 secs). Determining if task ':lib:core:util:android:common:packageReleaseResources' is up-to-date Executing task ':lib:core:util:android:common:packageReleaseResources' (up-to-date check took 0.035 secs) due to: Value of input property 'minSdk' has changed for task ':lib:core:util:android:common:packageReleaseResources'
112. Up-to-date check issues SkipUpToDateTaskExecuter in the logs Determining if task ':lib:core:util:android:common:extractReleaseAnnotations' is up-to-date Executing task ':lib:core:util:android:common:extractReleaseAnnotations' (up-to-date check took 0.012 secs) due to: Input property 'classDir' file /Users/cpuerta/workspace/twitter-android/lib/core/util/android/common/ build/intermediates/classes/release/android/support/v14/preference/R$id.class has changed. Input property 'classDir' file /Users/cpuerta/workspace/twitter-android/lib/core/util/android/common/ build/intermediates/classes/release/android/support/v7/appcompat/R$id.class has changed. Input property 'classDir' file /Users/cpuerta/workspace/twitter-android/lib/core/util/android/common/ build/intermediates/classes/release/android/support/v7/preference/R$id.class has changed. Determining if task ':lib:core:util:android:common:mergeReleaseAssets' is up-to-date Skipping task ':lib:core:util:android:common:mergeReleaseAssets' as it is up-to-date (took 0.003 secs). Determining if task ':lib:core:util:android:common:packageReleaseResources' is up-to-date Executing task ':lib:core:util:android:common:packageReleaseResources' (up-to-date check took 0.035 secs) due to: Value of input property 'minSdk' has changed for task ':lib:core:util:android:common:packageReleaseResources'
113. Up-to-date check issues SkipUpToDateTaskExecuter in the logs Determining if task ':lib:core:util:android:common:extractReleaseAnnotations' is up-to-date Executing task ':lib:core:util:android:common:extractReleaseAnnotations' (up-to-date check took 0.012 secs) due to: Input property 'classDir' file /Users/cpuerta/workspace/twitter-android/lib/core/util/android/common/ build/intermediates/classes/release/android/support/v14/preference/R$id.class has changed. Input property 'classDir' file /Users/cpuerta/workspace/twitter-android/lib/core/util/android/common/ build/intermediates/classes/release/android/support/v7/appcompat/R$id.class has changed. Input property 'classDir' file /Users/cpuerta/workspace/twitter-android/lib/core/util/android/common/ build/intermediates/classes/release/android/support/v7/preference/R$id.class has changed. Determining if task ':lib:core:util:android:common:mergeReleaseAssets' is up-to-date Skipping task ':lib:core:util:android:common:mergeReleaseAssets' as it is up-to-date (took 0.003 secs). Determining if task ':lib:core:util:android:common:packageReleaseResources' is up-to-date Executing task ':lib:core:util:android:common:packageReleaseResources' (up-to-date check took 0.035 secs) due to: Value of input property 'minSdk' has changed for task ':lib:core:util:android:common:packageReleaseResources'
114. Up-to-date check issues SkipUpToDateTaskExecuter in the logs Determining if task ':lib:core:util:android:common:extractReleaseAnnotations' is up-to-date Executing task ':lib:core:util:android:common:extractReleaseAnnotations' (up-to-date check took 0.012 secs) due to: Input property 'classDir' file /Users/cpuerta/workspace/twitter-android/lib/core/util/android/common/ build/intermediates/classes/release/android/support/v14/preference/R$id.class has changed. Input property 'classDir' file /Users/cpuerta/workspace/twitter-android/lib/core/util/android/common/ build/intermediates/classes/release/android/support/v7/appcompat/R$id.class has changed. Input property 'classDir' file /Users/cpuerta/workspace/twitter-android/lib/core/util/android/common/ build/intermediates/classes/release/android/support/v7/preference/R$id.class has changed. Determining if task ':lib:core:util:android:common:mergeReleaseAssets' is up-to-date Skipping task ':lib:core:util:android:common:mergeReleaseAssets' as it is up-to-date (took 0.003 secs). Determining if task ':lib:core:util:android:common:packageReleaseResources' is up-to-date Executing task ':lib:core:util:android:common:packageReleaseResources' (up-to-date check took 0.035 secs) due to: Value of input property 'minSdk' has changed for task ':lib:core:util:android:common:packageReleaseResources'
115. Up-to-date check issues Or use Gradle Enterprise 2017.4!
116. Up-to-date check issues Or use Gradle Enterprise 2017.4!
117. Up-to-date check issues • Android Studio invalidating caches • Annotation processors that don’t generate consistent code • Missing dependencies among tasks • MinSDK Some examples
118. Up-to-date check issues • Android Studio invalidating caches • Annotation processors that don’t generate consistent code • Missing dependencies among tasks • MinSDK Some examples
119. Up-to-date check issues buildscript { // Remove Android Studio's local Maven repository, since it pollutes // the classpath and causes rebuilds. allprojects { buildscript { repositories.clear() } } } Some examples
120. Up-to-date check issues • Android Studio invalidating caches • Annotation processors that don’t generate consistent code • Missing dependencies among tasks • MinSDK Some examples
121. Up-to-date check issues • Android Studio invalidating caches • Annotation processors that don’t generate consistent code • Missing dependencies among tasks • MinSDK Some examples
122. Up-to-date check issues afterEvaluate { // Work around missing task dependency getProjectVariants(project).each { variant -> def safeguardTask = tasks.getByName( "incremental${variant.name.capitalize()}JavaCompilationSafeguard") variant.variantData.outputs.each { output -> safeguardTask.dependsOn(output.scope.processResourcesTask.name) } } } Some examples
123. Up-to-date check issues • Android Studio invalidating caches • Annotation processors that don’t generate consistent code • Missing dependencies among tasks • MinSDK Some examples
124. Debugging Gradle
125. Debugging Gradle git clone https://github.com/gradle/gradle.git ./gradlew idea
126. Debugging Gradle git clone https://github.com/gradle/gradle.git ./gradlew idea
127. Debugging Gradle gr --no-daemon -Dorg.gradle.debug=true task
128. Debugging Gradle gr --no-daemon -Dorg.gradle.debug=true task
129. Debugging Gradle gr --no-daemon -Dorg.gradle.debug=true task
130. Debugging AGP
131. Debugging AGP
132. Debugging AGP
133. Debugging AGP
134. Debugging AGP
135. Debugging AGP
136. Debugging AGP
137. Debugging AGP Under ~/.gradle/caches/modules-2/files-2.1: com.android.tools.build/gradle-core/3.0.0-alpha4/ 67d7c235d088233a5406859348f3886ac32d7c4e/ gradle-core-3.0.0-alpha4.jar com.android.tools.build/builder/3.0.0-alpha4/ c7afbb32e2acd23b9e39fb6d411cc1b3d59e6555/ builder-3.0.0-alpha4.jar
138. Debugging AGP Under ~/.gradle/caches/modules-2/files-2.1: com.android.tools.build/gradle-core/3.0.0-alpha4/ 67d7c235d088233a5406859348f3886ac32d7c4e/ gradle-core-3.0.0-alpha4.jar com.android.tools.build/builder/3.0.0-alpha4/ c7afbb32e2acd23b9e39fb6d411cc1b3d59e6555/ builder-3.0.0-alpha4.jar
139. Debugging AGP
140. Debugging AGP
141. Debugging AGP
142. Debugging AGP
143. Debugging AGP
144. Debugging AGP
145. Debugging AGP
146. Debugging AGP
147. Debugging AGP Downloading the sources…
148. Debugging AGP Downloading the sources… wget https://maven.google.com/com/android/tools/build/ gradle-core/3.0.0-alpha4/gradle-core-3.0.0-alpha4-sources.jar wget https://maven.google.com/com/android/tools/build/ builder/3.0.0-alpha4/builder-3.0.0-alpha4-sources.jar
149. Debugging AGP Downloading the sources… wget https://maven.google.com/com/android/tools/build/ gradle-core/3.0.0-alpha4/gradle-core-3.0.0-alpha4-sources.jar wget https://maven.google.com/com/android/tools/build/ builder/3.0.0-alpha4/builder-3.0.0-alpha4-sources.jar
150. Debugging AGP
151. Patching AGP
152. Awesome Hacks Ahead
153. Patching AGP
154. Patching AGP
155. Patching AGP Copy your patched files to buildSrc…
156. Patching AGP Copy your patched files to buildSrc…
157. Patching AGP … and it just works
158. Patching AGP … and it just works
159. Patching AGP But don’t forget to file an issue!
160. Addressing memory issues
161. Addressing memory issues Too many garbage collection pauses!
162. Addressing memory issues Too many garbage collection pauses!
163. Addressing memory issues Also in a build scan
164. Addressing memory issues Also in a build scan
165. Addressing memory issues Adjust daemon heap size in gradle.properties org.gradle.jvmargs=-Xms512m -Xmx4g -Dfile.encoding=utf-8 Fork compilation to a separate process tasks.withType(JavaCompile) { options.fork = true } Leaks? org.gradle.jvmargs=-Xms512m -Xmx4g -Dfile.encoding=utf-8 \ -agentpath:/Applications/YourKit.app/Contents/Resources/bin/mac/libyjpagent.jnilib=sampling,onexit=memory org.gradle.daemon=false
166. Addressing memory issues Adjust daemon heap size in gradle.properties org.gradle.jvmargs=-Xms512m -Xmx4g -Dfile.encoding=utf-8 Fork compilation to a separate process tasks.withType(JavaCompile) { options.fork = true } Leaks? org.gradle.jvmargs=-Xms512m -Xmx4g -Dfile.encoding=utf-8 \ -agentpath:/Applications/YourKit.app/Contents/Resources/bin/mac/libyjpagent.jnilib=sampling,onexit=memory org.gradle.daemon=false
167. Addressing memory issues Adjust daemon heap size in gradle.properties org.gradle.jvmargs=-Xms512m -Xmx4g -Dfile.encoding=utf-8 Fork compilation to a separate process tasks.withType(JavaCompile) { options.fork = true } Leaks? org.gradle.jvmargs=-Xms512m -Xmx4g -Dfile.encoding=utf-8 \ -agentpath:/Applications/YourKit.app/Contents/Resources/bin/mac/libyjpagent.jnilib=sampling,onexit=memory org.gradle.daemon=false
168. Addressing memory issues Adjust daemon heap size in gradle.properties org.gradle.jvmargs=-Xms512m -Xmx4g -Dfile.encoding=utf-8 Fork compilation to a separate process tasks.withType(JavaCompile) { options.fork = true } Leaks? org.gradle.jvmargs=-Xms512m -Xmx4g -Dfile.encoding=utf-8 \ -agentpath:/Applications/YourKit.app/Contents/Resources/bin/mac/libyjpagent.jnilib=sampling,onexit=memory org.gradle.daemon=false
169. Addressing memory issues Adjust daemon heap size in gradle.properties org.gradle.jvmargs=-Xms512m -Xmx4g -Dfile.encoding=utf-8 Fork compilation to a separate process tasks.withType(JavaCompile) { options.fork = true } Leaks? org.gradle.jvmargs=-Xms512m -Xmx4g -Dfile.encoding=utf-8 \ -agentpath:/Applications/YourKit.app/Contents/Resources/bin/mac/libyjpagent.jnilib=sampling,onexit=memory org.gradle.daemon=false
170. Detecting a memory leak in the build Addressing memory issues
171. Detecting a memory leak in the build Addressing memory issues
172. Detecting a memory leak in the build Addressing memory issues
173. Detecting a memory leak in the build Addressing memory issues
174. Detecting a memory leak in the build Addressing memory issues
175. Detecting a memory leak in the build Addressing memory issues
176. And “ fi xing” the leak Addressing memory issues tasks.withType(MergeResources).each { task -> task.doLast { Utils.setPrivateFieldValue(task, MergeResources, 'processedInputs', null) task.sourceFolderInputs = { [] } } } And report an issue!
177. Fixing low parallelism
178. Fixing parallelism Are you using all your workers?
179. Fixing parallelism Are you using all your workers? 2.36x with 8 workers!
180. Fixing parallelism Also in a build scan…
181. Fixing parallelism Also in a build scan…
182. Fixing parallelism A Chrome trace tells a similar story
183. Fixing parallelism A Chrome trace tells a similar story
184. Because tools can’t fix it all Modularize
185. Why?
186. Why? • Improve compiler avoidance • Optimize caching of build artifacts • More e ffi cient parallelization • Better code architecture • Finer job granularity in CI
187. Why? • Improve compiler avoidance • Optimize caching of build artifacts • More e ffi cient parallelization • Better code architecture • Finer job granularity in CI
188. Why? • Improve compiler avoidance • Optimize caching of build artifacts • More e ffi cient parallelization • Better code architecture • Finer job granularity in CI
189. Why? • Improve compiler avoidance • Optimize caching of build artifacts • More e ffi cient parallelization • Better code architecture • Finer job granularity in CI
190. Why? • Improve compiler avoidance • Optimize caching of build artifacts • More e ffi cient parallelization • Better code architecture • Finer job granularity in CI
191. Why? • Improve compiler avoidance • Optimize caching of build artifacts • More e ffi cient parallelization • Better code architecture • Finer job granularity in CI
192. Configuring Gradle
193. Configuring Gradle • Shared gradle con fi guration fi les • Making sense of build errors • Select projects to run your tasks on
194. Configuring Gradle • Shared gradle con fi guration fi les • Making sense of build errors • Select projects to run your tasks on
195. Configuring Gradle • common-properties.gradle • checkstyle.gradle • lint.gradle • java-library-config.gradle • java-annotation-processor-config.gradle • android-common-config.gradle • android-library-config.gradle • android-application-config.gradle
196. Configuring Gradle • common-properties.gradle • checkstyle.gradle • lint.gradle • java-library-config.gradle • java-annotation-processor-config.gradle • android-common-config.gradle • android-library-config.gradle • android-application-config.gradle
197. Configuring Gradle • common-properties.gradle • checkstyle.gradle • lint.gradle • java-library-config.gradle • java-annotation-processor-config.gradle • android-common-config.gradle • android-library-config.gradle • android-application-config.gradle
198. Configuring Gradle apply plugin: 'checkstyle' apply from: "${project.rootDir}/config/gradle-common/common-properties.gradle" checkstyle { configProperties = [ configDir:"${rootDir}/config/checkstyle" ] configFile = file("${rootDir}/config/checkstyle/checkstyle.xml") toolVersion = checkstyleVersion showViolations = false } dependencies { checkstyle project(':tools:checkstyle-rules') } task checkstyle(type: Checkstyle) { source 'src' include '**/*.java' exclude '*/resources/**' classpath = files() group = 'Verification' description = 'Run all checkstyle tasks.' } checkstyle.gradle
199. Configuring Gradle • common-properties.gradle • checkstyle.gradle • lint.gradle • java-library-config.gradle • java-annotation-processor-config.gradle • android-common-config.gradle • android-library-config.gradle • android-application-config.gradle
200. Configuring Gradle • common-properties.gradle • checkstyle.gradle • lint.gradle • java-library-config.gradle • java-annotation-processor-config.gradle • android-common-config.gradle • android-library-config.gradle • android-application-config.gradle
201. Configuring Gradle apply plugin: 'java-library' apply from: "${project.rootDir}/config/gradle-common/common-properties.gradle" apply from: "${project.rootDir}/config/gradle-common/checkstyle.gradle" apply from: "${project.rootDir}/config/gradle-common/robolectric-test-config.gradle" sourceCompatibility = JavaVersion.VERSION_1_7 targetCompatibility = JavaVersion.VERSION_1_7 dependencies { api project.jetbrainsAnnotations } configurations { // A configuration that collects annotation processing dependencies to add to the // annotation processsor path. annotationProcessor } compileJava { options.annotationProcessorPath = configurations.annotationProcessor } java-library-config.gradle
202. Configuring Gradle apply plugin: 'java-library' apply from: "${project.rootDir}/config/gradle-common/common-properties.gradle" apply from: "${project.rootDir}/config/gradle-common/checkstyle.gradle" apply from: "${project.rootDir}/config/gradle-common/robolectric-test-config.gradle" sourceCompatibility = JavaVersion.VERSION_1_7 targetCompatibility = JavaVersion.VERSION_1_7 dependencies { api project.jetbrainsAnnotations } configurations { // A configuration that collects annotation processing dependencies to add to the // annotation processsor path. annotationProcessor } compileJava { options.annotationProcessorPath = configurations.annotationProcessor } java-library-config.gradle
203. Configuring Gradle apply plugin: 'java-library' apply from: "${project.rootDir}/config/gradle-common/common-properties.gradle" apply from: "${project.rootDir}/config/gradle-common/checkstyle.gradle" apply from: "${project.rootDir}/config/gradle-common/robolectric-test-config.gradle" sourceCompatibility = JavaVersion.VERSION_1_7 targetCompatibility = JavaVersion.VERSION_1_7 dependencies { api project.jetbrainsAnnotations } configurations { // A configuration that collects annotation processing dependencies to add to the // annotation processsor path. annotationProcessor } compileJava { options.annotationProcessorPath = configurations.annotationProcessor } java-library-config.gradle
204. Configuring Gradle • common-properties.gradle • checkstyle.gradle • lint.gradle • java-library-config.gradle • java-annotation-processor-config.gradle • android-common-config.gradle • android-library-config.gradle • android-application-config.gradle
205. Configuring Gradle • common-properties.gradle • checkstyle.gradle • lint.gradle • java-library-config.gradle • java-annotation-processor-config.gradle • android-common-config.gradle • android-library-config.gradle • android-application-config.gradle
206. Configuring Gradle • common-properties.gradle • checkstyle.gradle • lint.gradle • java-library-config.gradle • java-annotation-processor-config.gradle • android-common-config.gradle • android-library-config.gradle • android-application-config.gradle
207. Configuring Gradle apply plugin: 'com.android.library' apply from: "${project.rootDir}/config/gradle-common/common-properties.gradle" apply from: "${project.rootDir}/config/gradle-common/android-common-config.gradle" apply from: "${project.rootDir}/config/gradle-common/checkstyle.gradle" apply from: "${project.rootDir}/config/gradle-common/lint.gradle" android-library-config.gradle
208. Configuring Gradle • common-properties.gradle • checkstyle.gradle • lint.gradle • java-library-config.gradle • java-annotation-processor-config.gradle • android-common-config.gradle • android-library-config.gradle • android-application-config.gradle
209. Configuring Gradle apply from: "${project.rootDir}/config/gradle-common/android-library-config.gradle" dependencies { api 'com.twitter.android:media:1.22.1' api "com.android.support:exifinterface:${supportLibraryVersion}" implementation project(':lib:core:async:common') implementation project(':lib:core:util:android:io') } /lib/core/media/common/build.gradle
210. Configuring Gradle apply from: "${project.rootDir}/config/gradle-common/android-library-config.gradle" dependencies { api 'com.twitter.android:media:1.22.1' api "com.android.support:exifinterface:${supportLibraryVersion}" implementation project(':lib:core:async:common') implementation project(':lib:core:util:android:io') } /lib/core/media/common/build.gradle
211. Configuring Gradle apply from: "${project.rootDir}/config/gradle-common/android-library-config.gradle" dependencies { api 'com.twitter.android:media:1.22.1' api "com.android.support:exifinterface:${supportLibraryVersion}" implementation project(':lib:core:async:common') implementation project(':lib:core:util:android:io') } /lib/core/media/common/build.gradle
212. Configuring Gradle • Shared gradle con fi guration fi les • Making sense of build errors • Select projects to run your tasks on
213. Configuring Gradle
214. Configuring Gradle
215. Configuring Gradle
216. Configuring gradle • Shared gradle con fi guration fi les • Making sense of build errors • Select projects to run your tasks on
217. Configuring Gradle Select projects to run your tasks on
218. Some advice
219. Some advice • Use api dependencies sparingly • Isolate annotation processing • Remove bottlenecks in your graph • Use Java modules when possible • Use the new refactors in Android Studio to move resources
220. • Use api dependencies sparingly • Isolate annotation processing • Remove bottlenecks in your graph • Use Java modules when possible • Use the new refactors in Android Studio to move resources Some advice
221. • Use api dependencies sparingly • Isolate annotation processing • Remove bottlenecks in your graph • Use Java modules when possible • Use the new refactors in Android Studio to move resources Some advice
222. • Use api dependencies sparingly • Isolate annotation processing • Remove bottlenecks in your graph • Use Java modules when possible • Use the new refactors in Android Studio to move resources Some advice
223. • Use api dependencies sparingly • Isolate annotation processing • Remove bottlenecks in your graph • Use Java modules when possible • Use the new refactors in Android Studio to move resources Some advice
224. • Use api dependencies sparingly • Isolate annotation processing • Remove bottlenecks in your graph • Use Java modules when possible • Use the new refactors in Android Studio to move resources Some advice
225. Wrap Up
226. 54 Modules • Mostly inside large app module, and one large library module • AGP 2.2.2 • Gradle 3.1 (In October 2016) Our build
227. 93 Modules • Still have 2 larger modules, but considerably smaller than months ago • AGP 2.3.3 - using 3.0.0 also • Gradle 4.0 (In June 2017) Our build
228. Seconds 0 75 150 225 300 Gradle 2.3.3 Buck Gradle 3.0.0 Clean Incremental
229. Seconds 0 75 150 225 300 Gradle 2.3.3 Buck Gradle 3.0.0 Clean Incremental
230. Profile, Profile, Profile Fix Issues Breakdown Large Modules
231. Questions? @CesarDielo @m_evans10