Gradle: The Build System That Loves To Hate You

Slide 1

Slide 1 text

🐘 Gradle: The Build System That Loves To Hate You The Survival Tips Aurimas Liutikas / AndroidX @ Google @[email protected]

Slide 2

Slide 2 text

🐘 Who’s Aurimas? 12 years at Google 8.5 years on AndroidX Work closely with Android Studio, Gradle, and Jetbrains

Slide 3

Slide 3 text

🐘 Is Gradle a bad build tool?

Slide 4

Slide 4 text

🐘 When it works, it is pretty amazing 🚀

Slide 5

Slide 5 text

🐘 androidx one of the largest OSS Gradle projects

Slide 6

Slide 6 text

🐘

Slide 7

Slide 7 text

🐘

Slide 8

Slide 8 text

🐘

Slide 9

Slide 9 text

🐘 local assembleDebug (90 days)

Slide 10

Slide 10 text

🐘 The end? 🎉

Slide 11

Slide 11 text

🐘 As project grows, you need a build engineer(s) 󰞳

Slide 12

Slide 12 text

🐘 Sadly, the path to becoming a Gradle build engineer is treacherous

Slide 13

Slide 13 text

🐘 Let’s look at some examples

Slide 14

Slide 14 text

🐘 Let’s create a task

Slide 15

Slide 15 text

🐘 tasks.create("myTask") { long start = System.currentTimeMillis() Thread.sleep(5000) long end = System.currentTimeMillis() println( "Spent ${end - start} ms" ) }

Slide 16

Slide 16 text

🐘 $ ./gradlew lib:myTask > Configure project :lib Spent 5000 ms BUILD SUCCESSFUL in 6s

Slide 17

Slide 17 text

🐘 $ ./gradlew lib:jar > Configure project :lib Spent 5000 ms BUILD SUCCESSFUL in 6s

Slide 18

Slide 18 text

🐘 tasks.create("myTask") { // task configuration }

Slide 19

Slide 19 text

🐘

Slide 20

Slide 20 text

🐘 class MyBetterTask extends DefaultTask { @TaskAction void doTheWork() { long start = System.currentTimeMillis() Thread.sleep(5000) long end = System.currentTimeMillis() println("Spent ${end - start} ms") } } tasks.create("myBetterTask", MyBetterTask)

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

🐘 We have one more issue

Slide 24

Slide 24 text

🐘 tasks.create("myBetterTask", MyBetterTask) { // some expensive configuration }

Slide 25

Slide 25 text

🐘 tasks.register("myBetterTask", MyBetterTask) { // some expensive configuration // only runs if task will be executed }

Slide 26

Slide 26 text

🐘 Lazy configuration / tasks

Slide 27

Slide 27 text

🐘 - Gradle should deprecate TaskContainer.create and friends https://github.com/gradle/gradle/issues/17705 (open since 2021) Lessons

Slide 28

Slide 28 text

🐘 - Gradle should deprecate TaskContainer.create and friends https://github.com/gradle/gradle/issues/17705 (open since 2021) - Build engineers need to understand configuration vs execution phase - avoid doing expensive work in configuration phase Lessons

Slide 29

Slide 29 text

Slide 30

Slide 30 text

🐘 Let’s configure some tasks

Slide 31

Slide 31 text

🐘 tasks.withType(MyBetterTask) { configureTask() }

Slide 32

Slide 32 text

🐘 tasks.withType(MyBetterTask) { configureTask() } ❌

Slide 33

Slide 33 text

🐘 tasks.withType(MyBetterTask) .configureEach { configureTask() }

Slide 34

Slide 34 text

🐘 tasks.create tasks.getByPath tasks.findByPath tasks.findByName tasks.withType tasks.getByName tasks.all tasks.matching Eager APIs to Avoid tasks.whenTaskAdded tasks.whenObjectAdded tasks.findAll tasks.iterator, thus all of the kotlin-stdlib collection extension functions, e.g. tasks.any { }

Slide 35

Slide 35 text

Slide 36

Slide 36 text

🐘

Slide 37

Slide 37 text

🐘 - Gradle should deprecate all TaskContainer, TaskCollection, DomainObjectCollection methods that are not register, named, or configureEach Lessons

Slide 38

Slide 38 text

🐘 - Gradle should deprecate all TaskContainer, TaskCollection, DomainObjectCollection methods that are not register, named, or configureEach - Build engineers should pay attention to eagerly created tasks tasks.register("eagerCanary") { throw Exception("Eagerly configured tasks!") } Lessons

Slide 39

Slide 39 text

Slide 40

Slide 40 text

🐘 Let’s wire two tasks together

Slide 41

Slide 41 text

🐘 abstract class FooWriter : DefaultTask() { @get:OutputFile abstract val outputFile : RegularFileProperty @TaskAction fun doTheWork() { outputFile.get().asFile.writeText("Hello") } }

Slide 42

Slide 42 text

🐘 abstract class FooReader : DefaultTask() { @get:InputFile abstract val inputFile : RegularFileProperty @TaskAction fun doTheWork() { println(inputFile.get().asFile.readText()) } }

Slide 43

Slide 43 text

🐘 val foo = layout.buildDirectory.file("foo.txt") val writer = tasks.register("fooWriter") { outputFile.set(foo) } tasks.register("fooReader") { inputFile.set(foo) }

Slide 44

Slide 44 text

🐘 $ ./gradlew fooReader > Task fooReader FAILED FAILURE: Build failed with an exception. * What went wrong: A problem was found with the configuration of task 'fooReader' (type 'FooReader'). - Type 'FooReader' property 'inputFile' specifies file 'foo.txt' which doesn't exist.

Slide 45

Slide 45 text

🐘 val foo = layout.buildDirectory.file("foo.txt") val writer = tasks.register("fooWriter") { outputFile.set(foo) } tasks.register("fooReader") { inputFile.set(foo) dependsOn(writer) }

Slide 46

Slide 46 text

🐘 val foo = layout.buildDirectory.file("foo.txt") val writer = tasks.register("fooWriter") { outputFile.set(foo) } tasks.register("betterFooReader") { inputFile.set(writer.flatMap { it.outputFile }) }

Slide 47

Slide 47 text

🐘 - Gradle should deprecate Task.dependsOn Lessons

Slide 48

Slide 48 text

🐘 - Gradle should deprecate Task.dependsOn - Build engineers should use Provider.flatmap to establish dependencies Lessons

Slide 49

Slide 49 text

🐘 Let’s run the build again

Slide 50

Slide 50 text

🐘 $ ./gradlew lib2:betterFooReader > Task :lib2:betterFooReader Hello BUILD SUCCESSFUL in 1s 2 actionable tasks: 1 executed, 1 up-to-date

Slide 51

Slide 51 text

🐘 Gradle will always rerun tasks if they don’t declare any outputs

Slide 52

Slide 52 text

🐘 - Gradle should force task authors to mark tasks as @AlwaysRerunTask or declare an output Lessons

Slide 53

Slide 53 text

🐘 - Gradle should force task authors to mark tasks as @AlwaysRerunTask or declare an output - Build engineers should always add outputs to tasks tasks.register("betterFooReader") { inputFile.set(writer.flatMap { it.outputFile }) cacheEvenIfNoOutputs() } Lessons

Slide 54

Slide 54 text

🐘 fun Task.cacheEvenIfNoOutputs() { this.outputs.file(getPlaceholderOutput()) } private fun Task.getPlaceholderOutput(): Provider { return project.layout.buildDirectory.file( "placeholder/" + this.name.replace(":", "-") ) }

Slide 55

Slide 55 text

🐘 Let’s investigate what does deleting build/ do to the build

Slide 56

Slide 56 text

🐘 abstract class FooEnhancer : DefaultTask() { @get:InputFile abstract val inputFile : RegularFileProperty @get:OutputFile abstract val outputFile : RegularFileProperty @TaskAction fun doTheWork() { outputFile.get().asFile.writeText( inputFile.get().asFile.readText() + "!" ) } }

Slide 57

Slide 57 text

🐘 $ ./gradlew fooEnhancer BUILD SUCCESSFUL in 965ms 2 actionable tasks: 2 executed

Slide 58

Slide 58 text

🐘 $ ./gradlew fooEnhancer --info > Task :lib2:fooEnhancer Caching disabled for task ':lib2:fooEnhancer' because: Build cache is disabled Caching has not been enabled for the task Task ':lib2:fooEnhancer' is not up-to-date because: Output property 'outputFile' file build/enhancedFoo.txt has been removed.

Slide 59

Slide 59 text

🐘 UP-TO-DATE (default on) Task’s inputs and outputs did not change. FROM-CACHE (default off) Task’s outputs could be found from a previous execution.

Slide 60

Slide 60 text

🐘 org.gradle.caching=true

Slide 61

Slide 61 text

🐘 $ ./gradlew fooEnhancer --info > Task :lib2:fooEnhancer Caching disabled for task ':lib2:fooEnhancer' because: Caching has not been enabled for the task Task ':lib2:fooEnhancer' is not up-to-date because: Output property 'outputFile' file build/enhancedFoo.txt has been removed.

Slide 62

Slide 62 text

🐘 @CacheableTask abstract class FooEnhancer : DefaultTask() { @get:[InputFile PathSensitive(PathSensitivity.NONE)] abstract val inputFile : RegularFileProperty @get:OutputFile abstract val outputFile : RegularFileProperty @TaskAction fun doTheWork() { outputFile.get().asFile.writeText(inputFile.get().asFile.readText() + "!") } }

Slide 63

Slide 63 text

🐘 $ ./gradlew fooEnhancer --info > Task :lib2:fooEnhancer FROM-CACHE Build cache key for task ':lib2:fooEnhancer' is eb57efc9cb2dd2b6587672bac94200b4 Task ':lib2:fooEnhancer' is not up-to-date because: Output property 'outputFile' file build/enhancedFoo.txt has been removed. Loaded cache entry for task ':lib2:fooEnhancer' with cache key eb57efc9cb2dd2b6587672bac94200b4

Slide 64

Slide 64 text

🐘 - Gradle should enable build caching by default and force task authors to mark tasks as @DisableCachingByDefault or @CacheableTask Lessons

Slide 65

Slide 65 text

🐘 - Gradle should enable build caching by default and force task authors to mark tasks as @DisableCachingByDefault or @CacheableTask - Build engineers should enable build cache and always mark their tasks @CacheableTask, unless it is an I/O operation such as copy. Lessons

Slide 66

Slide 66 text

Slide 67

Slide 67 text

🐘 Let’s share artifacts between projects

Slide 68

Slide 68 text

🐘 // projectA tasks.register("zip", Zip) { it.from(layout.projectDirectory.file("build.gradle.kts")) destinationDirectory.set(layout.buildDirectory.dir("zip")) archiveFileName.set("my.zip") } // projectB val zipTask = project(":projectA").tasks.named("zip") tasks.register("zipConsumer") { inputFile.set(zipTask.flatMap { it.archiveFile }) }

Slide 69

Slide 69 text

🐘 // projectA tasks.register("zip", Zip) { it.from(layout.projectDirectory.file("build.gradle")) destinationDirectory.set(layout.buildDirectory.dir("zip")) archiveFileName.set("my.zip") } // projectB val zipTask = project(":projectA").tasks.named("zip") tasks.register("zipConsumer") { inputFile.set(zipTask.flatMap { it.archiveFile }) }

Slide 70

Slide 70 text

🐘 $ ./gradlew projectB:zipConsumer --dry-run

Slide 71

Slide 71 text

🐘 $ ./gradlew projectB:zipConsumer --dry-run :projectA:zip SKIPPED :projectB:zipConsumer SKIPPED

Slide 72

Slide 72 text

🐘 // projectA val zipTask = tasks.register("zip") { … } configurations.register("zipPublish") { isCanBeConsumed = true isCanBeResolved = false attributes { attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, objects.named("special-zip"))} } artifacts { add("zipPublish", zipTask) }

Slide 73

Slide 73 text

🐘 // projectB val zipConfiguration = configurations.create("zipConsumer") { attributes { attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, objects.named("special-zip")) } } zipConfiguration.dependencies.add(dependencies.create(project(":projectA"))) val zipFile : FileCollection = zipConfiguration.incoming.artifactView { }.files tasks.register("zipConsumer") { inputFile.from(zipFile) }

Slide 74

Slide 74 text

Slide 75

Slide 75 text

Slide 76

Slide 76 text

🐘 - By default Gradle should flag cross-project task access and make cross project artifact sharing easier Lessons

Slide 77

Slide 77 text

🐘 - By default Gradle should flag cross-project task access and make cross project artifact sharing easier - Build engineers should test their builds with org.gradle.unsafe.isolated-projects=true to catch such violations Lessons

Slide 78

Slide 78 text

🐘 Let’s reuse expensive work between projects

Slide 79

Slide 79 text

🐘 abstract class MyBuildService : BuildService { val expensiveValue: String by lazy { Thread.sleep(5000) "hello" } } abstract class MyBuildServiceParameters : BuildServiceParameters

Slide 80

Slide 80 text

🐘 class MyPlugin : Plugin { override fun apply(target: Project) { val serviceProvider = target.gradle.sharedServices.registerIfAbsent( "myService", MyBuildService::class.java) target.tasks.register("myTask", MyTask::class.java) { it.myService.set(serviceProvider) it.usesService(serviceProvider) } } }

Slide 81

Slide 81 text

🐘 // projectA plugins { id("myPlugin") }

Slide 82

Slide 82 text

🐘 $ ./gradlew myTask BUILD SUCCESSFUL in 7s

Slide 83

Slide 83 text

🐘 // projectB plugins { id("myPlugin") alias(libs.plugins.ktfmt) }

Slide 84

Slide 84 text

🐘 $ ./gradlew myTask FAILURE: Build failed with an exception. * What went wrong: A problem occurred configuring project ':projectB. > Could not create task ':projectB:myTask'. > Cannot set the value of task ':projectB:myTask' property 'myService' of type org.example.MyBuildService using a provider of type org.example.MyBuildService.

Slide 85

Slide 85 text

🐘 // projectA plugins { id("myPlugin") } // projectB plugins { id("myPlugin") alias(libs.plugins.ktfmt) } ● myPlugin + its dependencies ● myPlugin + its dependencies ● ktmft + its dependencies build classloaders

Slide 86

Slide 86 text

🐘 // root build.gradle.kts plugins { alias(libs.plugins.ktfmt) apply false id("myPlugin") apply false }

Slide 87

Slide 87 text

🐘 $ ./gradlew myTask BUILD SUCCESSFUL in 7s

Slide 88

Slide 88 text

🐘 - By default Gradle should forbid build classpath difference between projects and if they don’t, catch this failure with a better message https://github.com/gradle/gradle/issues/17559 Lessons

Slide 89

Slide 89 text

🐘 - By default Gradle should forbid build classpath difference between projects and if they don’t, catch this failure with a better message https://github.com/gradle/gradle/issues/17559 - Build engineers should add all their plugins in root build.gradle.kts while setting apply false Lessons

Slide 90

Slide 90 text

🐘 and there are many more examples 😭

Slide 91

Slide 91 text

🐘 Takeaways - If you work at Gradle - deprecate APIs that have better alternatives - enable helpful features by default (build cache & configuration cache) - enforce best build practices by default - If you own a build - enable build cache, configuration cache, and test with project isolation - use well known Gradle builds as inspiration - https://github.com/android/gradle-recipes - https://github.com/slackhq/foundry - https://github.com/androidx/androidx - If you own a Gradle plugin - use androidx.lint:lint-gradle on it and enable stricter validation