Build bigger, better: Gradle for large projects

Transcript

2. Gradle for Large Projects Build Bigger, Better:
3. Every project starts simple ...
4. … but quickly gets more complicated.
5. Android Projects are the same • Start as simple monolith builds. • Organically grow in size over time. • Each change incrementally makes sense.
6. AndroidX build growth • 5 (2013) → 240 (today) Gradle projects • 91,415 (2013) → 427,746 (today) lines of source code • 1,965 (2013) → 11,033 (last 12 months) git commits
7. Modularize Smaller is better
8. Why? • More parallelism • Compilation avoidance • Fewer cache invalidations • Monolithic builds can lead to tangled dependencies. • Per module tests
9. Java or Kotlin only libraries • Simpler Gradle projects (no variants, resources, manifests) • Forced separation from Android APIs • Test can run on the host
10. Apply only needed plugins • Keep root project light • Avoid applying every Gradle plugin to every project (e.g. using allprojects {})
11. Scope annotation processors • Separate code that requires annotation processors into a library • Run annotation processors only on libraries that need it
12. Lint final app module Avoid repeated work by running Android Lint on final app module only
13. Use api and implementation Increase compilation avoidance in your project
14. Modularization side-effects • Increased cost of computing dependency graph • Repeated build configuration code which is error-prone and hard to maintain
15. Do Not Repeat Yourself Build configuration sharing
16. Handling dependencies
17. Root build.gradle (Option 1) ext { androidx = [ appcompat : "androidx.appcompat:appcompat:1.0.2", constraintlayout : "androidx.constraintlayout:constraintlayout:1.1.3", test : [ runner : "androidx.test:runner:1.1.1", espresso : "androidx.test.espresso:espresso-core:3.1.1" ] ] firebase = [ appindexing : "com.google.firebase:firebase-appindexing:17.1.0" ] junit = "junit.junit:4.12" }
18. Project build.gradle (Option 1) dependencies { implementation androidx.appcompat implementation androidx.constraintlayout implementation firebase.appindexing testImplementation junit androidTestImplementation androidx.test.runner androidTestImplementation androidx.test.espresso }
19. Embrace buildSrc/
20. Dependencies.kt in buildSrc/ (Option 2) package my.example const val ANDROIDX_APPCOMPAT = "androidx.appcompat:appcompat:1.0.2" const val ANDROIDX_TEST_RUNNER = "androidx.test:runner:1.1.1" const val ANDROIDX_TEST_ESPRESSO = "androidx.test.espresso:espresso-core:3.1.1" const val ANDROIDX_CONSTRAINTLAYOUT = "androidx.constraintlayout:constraintlayout:1.1.3" const val FIREBASE_APPINDEXING = "com.google.firebase:firebase-appindexing:17.1.0" const val JUNIT = "junit.junit:4.12"
21. Project build.gradle (Option 2) import static my.example.DependenciesKt.* dependencies { implementation ANDROIDX_APPCOMPAT implementation ANDROIDX_CONSTRAINTLAYOUT implementation FIREBASE_APPINDEXING testImplementation JUNIT androidTestImplementation ANDROIDX_TEST_RUNNER androidTestImplementation ANDROIDX_TEST_ESPRESSO }
22. Handling other configuration
23. MyPlugin.kt in buildSrc/ class MyPlugin : Plugin<Project> { override fun apply(project: Project) { project.plugins.all { when(it) { is LibraryPlugin -> { val extension = project.extensions.getByType( LibraryExtension::class.java) extension.configureLibrary() } is AppPlugin -> { ... } } } } }
24. MyPlugin.kt in buildSrc/ class MyPlugin : Plugin<Project> { override fun apply(project: Project) { project.plugins.all { when(it) { is LibraryPlugin -> { val extension = project.extensions.getByType( LibraryExtension::class.java) extension.configureLibrary() } is AppPlugin -> { ... } } } } }
25. MyPlugin.kt in buildSrc/ private fun LibraryExtension.configureLibrary() { setCompileSdkVersion(28) defaultConfig.apply { minSdkVersion(21) versionCode = 1 versionName = "1.0" } compileOptions.apply { setSourceCompatibility(VERSION_1_8) setTargetCompatibility(VERSION_1_8) } }
26. MyPlugin.kt in buildSrc/ project.plugins.all { when (it) { is JavaPlugin, is JavaLibraryPlugin -> { project.convention.getPlugin<JavaPluginConvention>().apply { sourceCompatibility = VERSION_1_8 targetCompatibility = VERSION_1_8 } } } }
27. Each build.gradle apply plugin: my.example.MyPlugin apply plugin: 'com.android.application'
28. Per-module configuration • Keep custom configuration in build.gradle • Create your own DSL (Gradle Extension)
29. Custom configuration in each build.gradle // Module build.gradle apply plugin: my.example.MyPlugin apply plugin: 'com.android.application' android { defaultConfig { minSdkVersion 24 } compileOptions { sourceCompatibility = JavaVersion.VERSION_1_8 targetCompatibility = JavaVersion.VERSION_1_8 } }
30. Example build.gradle in androidx plugins { id("AndroidXPlugin") id("com.android.library") } dependencies { ... } android { buildTypes.all { consumerProguardFiles "proguard-rules.pro" } lintOptions.fatal("UnknownNullness") } ...
31. Example build.gradle in androidx ... androidx { name = 'Android Lifecycle Runtime' publish = true mavenVersion = LibraryVersions.LIFECYCLE mavenGroup = LibraryGroups.LIFECYCLE inceptionYear = '2017' description = "Android Lifecycle Runtime" url = AndroidXExtension.ARCHITECTURE_URL failOnDeprecationWarnings = false }
32. Configuration using custom extensions // Module build.gradle apply plugin: my.example.MyPlugin apply plugin: 'com.android.application' myConfig { suppressLintWarnings.set(true) }
33. Configuration using custom extensions // MyPluginExtension.kt abstract class MyPluginExtension() { abstract val suppressLintWarnings: Property<Boolean> }
34. Configuration using custom extensions // MyPlugin.kt class MyPlugin : Plugin<Project> { override fun apply(project: Project) { val myConfig = project.extensions.create("myConfig", MyPluginExtension::class.java) ...
35. Configuration using custom extensions abstract class SomeTask : DefaultTask() { @get:Input abstract val suppressLintWarnings: Property<Boolean> } tasks.register("myTask", SomeTask::class.java) { suppressLintWarnings.set(myConfig.suppressLintWarnings) }
36. Configs done right Common tasks
37. General Advice: Configuration Do as little as possible
38. General Advice Keep debug build type light. For example: create no-op tasks for debug that are full tasks for release builds CI should only build relevant variant(s)
39. Lazy task configuration Especially important for Android when you can have variants you are not building.
40. Task Wiring Task.dependsOn() → can lead to messy unnecessary dependencies.
41. open class Task1 : DefaultTask() { @get:OutputDirectory lateinit var outputDir: File } open class Task2 : DefaultTask() { @get:InputDirectory @get:PathSensitive(PathSensitivity.RELATIVE) lateinit var inputDir: File }
42. val task1: TaskProvider<Task1> = tasks.register<Task1>("task1") { outputDir = File(project.buildDir, "foo") } tasks.register<Task2>("task2") { inputDir = File(project.buildDir, "foo") dependsOn(task1) } meh... !!
43. val task1: TaskProvider<Task1> = tasks.register<Task1>("task1") { outputDir = File(project.buildDir, "foo") } tasks.register<Task2>("task2") { inputDir = File(project.buildDir, "foo") } task2.configure { dependsOn(task1) }
44. abstract class Task1 : DefaultTask() { @get:OutputDirectory abstract val outputDir: DirectoryProperty } abstract class Task2 : DefaultTask() { @get:InputDirectory @get:PathSensitive(PathSensitivity.RELATIVE) abstract val inputDir: DirectoryProperty }
45. val task1: TaskProvider<Task1> = tasks.register<Task1>("task1") { outputDir.set(File(project.buildDir, "foo")) } tasks.register<Task2>("task2") { inputDir.set(task1.get().outputDir) }
46. val task1: TaskProvider<Task1> = tasks.register<Task1>("task1") { outputDir.set(File(project.buildDir, "foo")) } tasks.register<Task2>("task2") { inputDir.set(task1.flatMap { it.outputDir }) }
47. val task1: TaskProvider<Task1> = tasks.register<Task1>("task1") { outputDir.set(File(project.buildDir, "foo")) } val artifact: Provider<Directory> = task1.flatMap { it.outputDir } tasks.register<Task2>("task2") { inputDir.set(artifact) }
48. Use Gradle Worker API New intra-module parallelism
49. Always Create Custom Task Types Default easy path will lead to the inefficient, or wrong, results
50. task("sourceCopy") { expensiveCall() }
51. task<Copy>("sourceCopy") { from(file("srcDir")) into(buildDir) expensiveCall() }
52. task("sourceCopy") { doLast { expensiveCall() } }
53. Computation during Configuration Do NOT do it!
54. git information in version code/name?
55. android { defaultConfig { versionCode = computeVersionCode() } }
56. defaultConfig { versionCode = 99999 applicationVariants.all { outputs.all { processManifestProvider.configure { doLast { val manifest = File( this@configure.manifestOutputDirectory.get().asFile, "AndroidManifest.xml") val content = manifest.readText() manifest.writeText(content.replace("99999", computeVersionCode())) } } } } }
57. defaultConfig { versionCode = 99999 applicationVariants.all { outputs.all { processManifestProvider.configure { doLast { val manifest = File( this@configure.manifestOutputDirectory.get().asFile, "AndroidManifest.xml") val content = manifest.readText() manifest.writeText(content.replace("99999", computeVersionCode())) } } } } }
58. defaultConfig { versionCode = 99999 applicationVariants.all { outputs.all { processManifestProvider.configure { doLast { val manifest = File( this@configure.manifestOutputDirectory.get().asFile, "AndroidManifest.xml") val content = manifest.readText() manifest.writeText(content.replace("99999", computeVersionCode())) } } } } }
59. defaultConfig { versionCode = 99999 applicationVariants.all { outputs.all { processManifestProvider.configure { doLast { val manifest = File( this@configure.manifestOutputDirectory.get().asFile, "AndroidManifest.xml") val content = manifest.readText() manifest.writeText(content.replace("99999", computeVersionCode())) } } } } }
60. defaultConfig { versionCode = 99999 applicationVariants.all { outputs.all { processManifestProvider.configure { doLast { val manifest = File( this@configure.manifestOutputDirectory.get().asFile, "AndroidManifest.xml") val content = manifest.readText() manifest.writeText(content.replace("99999", computeVersionCode())) } } } } }
61. defaultConfig { versionCode = 99999 applicationVariants.all { outputs.all { processManifestProvider.configure { doLast { val manifest = File( this@configure.manifestOutputDirectory.get().asFile, "AndroidManifest.xml") val content = manifest.readText() manifest.writeText(content.replace("99999", computeVersionCode())) } } } } } UP-TO-DATE? CACHING?
62. defaultConfig { versionCode = 99999 applicationVariants.all { outputs.all { processManifestProvider.configure { inputs.property("my_input", Callable{ computeVersionCode()}) doLast { ... manifest.writeText(content.replace("99999", computeVersionCode())) } } } } }
63. defaultConfig { versionCode = 99999 applicationVariants.all { outputs.all { processManifestProvider.configure { val callable = memoize(Callable{ computeVersionCode()}) inputs.property("my_input", callable) doLast { ... manifest.writeText(content.replace("99999", callable.get())) } } } } }
64. Value in android Model is not to update → Can impact other plugins Value vs File as input? doLast don’t use workers. Caveats
65. Task A doLast Thread-1 Workers + doLast
66. doLast Worker For Task A Thread-1 Workers + doLast T A
67. doLast Task B Worker For Task A Thread-1 Thread-2 Workers + doLast T A T1 T2
68. Passing some data via BuildConfig?
69. android { defaultConfig { buildConfigField "long", "something", computeValue() } }
70. abstract class GenerateCode : DefaultTask() { @get:Input abstract val packageName: Property<String> @get:Input abstract val className: Property<String> @get:OutputDirectory abstract val outputDir: DirectoryProperty @get:Input abstract val value: Property<String> }
71. abstract class GenerateCode : DefaultTask() { @TaskAction fun doSomeWork() { computeOutputFile().writeText(""" package ${packageName.get()}; public class ${className.get()} { public final static long Something = ${value.get()}; } """.trimIndent()) } }
72. android.applicationVariants.all { variant -> val taskProvider = project.tasks.register( "generate${variant.name}Code", GenerateCode::class.java ) { task -> task.packageName.set("my.package.name.here") task.className.set("CustomConfig") task.outputDir.set( File(project.buildDir, "intermediates/generate-code/${task.name}")) task.value.set(project.provider { computeValue() }) } }
73. public interface BaseVariant { void registerJavaGeneratingTask( @NonNull Task task, @NonNull File... sourceFolders); void registerJavaGeneratingTask( @NonNull Task task, @NonNull Collection<File> sourceFolders); }
74. android.applicationVariants.all { variant -> val taskProvider = project.tasks.register( "generate${variant.name}Code", GenerateCode::class.java ) { task -> ... } val task = taskProvider.get() variant.registerJavaGeneratingTask(task, task.outputDir.get().asFile) }
75. Guide to Investigations Profile and Measure
76. Build Scans • ./gradlew <my-tasks> --scan ◦ Accept license ◦ Build information is sent to Gradle’s server ◦ You can delete it after • Will help with: ◦ Non lazy task configuration (or lazy tasks that are force configured) ◦ Dependencies resolved are configuration time
77. 18.68ms / module 24.10ms / module
78. Gradle Memory Limits Default value might not be right for your project org.gradle.jvmargs=-Xmx????
79. Gradle Memory Limits Experiment with the value for optimal speed / memory usage • build with --rerun-tasks • kill Gradle daemon in between runs (gradlew --stop) • run the most common “big” tasks • use typical developer computer (CPU / RAM)
80. Gradle build scans Execution vs garbage collection
81. -Xmx1g for androidx
82. Investigating it for androidx
83. Limit Gradle workers Set sensible org.gradle.workers.max in gradle.properties Limits parallelism → reduces memory usage
84. Gradle Memory Limits Consider using different values (memory and --max-workers) for CI • It runs different tasks • Might have more cores / memory
85. https://android.googlesource.com/platform/tools/base/+/refs/heads/mi rror-goog-studio-master-dev/tracer/ See README.md to build / use Tracer Agent
89. Aurimas Liutikas Thank you! Software Engineer, Google Xavier Ducrohet Software Engineer, Google https://developers.google.com/ google.com/io Helpful resources