Improving developer productivity with Gradle Enterprise in 2023

This year has been quite an exciting year for Gradle Enterprise features and Gradle Enterprise customers. With Julio representing the Gradle Enterprise Developer Platform Team and Nelson representing the Solutions team and Gradle Enterprise customers, we’ll cover a broad range of developer productivity tips and topics.

We will start by discussing some of the common Gradle issues facing Android developers. We’ll cover some build cache misses and configuration cache issues as well as how to debug them and solve them. After that we’ll dive into the newly released Gradle Enterprise feature most commonly requested Android devs: Artifact Transforms in Build Scans. Then we’ll wrap it up by previewing some upcoming Gradle Enterprise APIs and how to use them in your day to day workflow.

Transcript

1. Improving developer productivity with Gradle Enterprise in 2023 Nelson Osacky and Julio Zynger
2. Julio • Previously Android Engineer • Large projects • SoundCloud, Autodesk • Developer Produc ti vity Enthusiast • Leading Gradle’s (and previously SoundCloud’s) interdisciplinary Pla tf orm teams • Backstage contributor https://github.com/backstage/backstage Lead Dev Platform Engineer
3. Nelson • Previously Android Engineer • Large projects • SoundCloud • Square • Small startups • Gradle Plugin Maintainer • Fladle - Easily Scale Instrumenta ti on Tests on Firebase https://github.com/runningcode/ fl adle • Gradle Doctor - Ac ti onable Insights for your build https://github.com/runningcode/gradle-doctor Lead Solutions Engineer
4. We see a lot of builds from 10 person teams to 1000+ Android, Gradle and Maven
5. • Optimize and understand builds • Increase build stability • Preach Developer Productivity
6. • Improve Gradle ecosystem • Fix issues in open source plugin affecting customers
7. • Meet with Google and Jetbrains to help prioritize issues impacting customers
9. Android Development Feature Development Tech Debt (There are other things too)
10. Tech Debt Refactoring Improving builds
11. Slow builds are Tech Debt And it always pays off
12. Cost of Builds 60s waste * 50 builds / day * 50 devs 
 = 42 hours lost / day
13. Cost of Builds 60s waste * 50 builds / day * 50 devs 
 = 42 hours lost / day not including lost focus h tt ps:/ /gradle.com/roi-calculator
14. Fast Builds Matter
15. Cost of Builds 60s waste * 50 builds / day * 50 devs 
 = 42 hours lost / day hire 5 new people without paying them! no recruiting h tt ps:/ /gradle.com/roi-calculator
16. Slow builds are Tech Debt And it always pays off And is easy to justify working on it
18. https://gradle.com/enterprise/trial/
19. How many have seen or used a Build Scan?
20. How many are Gradle Enterprise customers?
21. https:/ /gradle.com/enterprise-customers/oss-projects/ Gradle Enterprise OSS Projects
22. What is build caching? https:/ /docs.gradle.org/current/userguide/build_cache.html
23. JavaCompile task Source Files Compiler Args Dependencies Class files any task can be made cacheable
24. Cache key Source Files Compiler Args Dependencies
25. Cache key Source Files Compiler Args Dependencies Build cache Class files
26. What is a cache miss?
27. What is a cache miss? •Expect a cache hit but cache key is di ff erent •Task outcome is the same •Only inputs that a ff ect outcome should be tracked
29. Task input comparison
31. Common Cache misses
32. Build Configuration Differences
33. Build Configuration Differences
34. Build Configuration Differences kotlin.incremental=false kotlin.incremental=true CI Local
35. Build Configuration Differences kotlin.incremental=false kotlin.incremental=true CI Local Incremental has no effect on outcome of task
36. Build Configuration Differences kotlin.incremental=true kotlin.incremental=true CI Local Incremental has no effect on outcome of task h tt ps:/ /youtrack.jetbrains.com/issue/KT-40974/isIncrementalCompila ti onEnabled-should-not-be-marked-as-Input
37. LiveLiterals
38. LiveLiterals
39. LiveLiterals
40. •LiveLiterals$AnalyticsEvent.kt contains fully qualified path •Fix is to disable LiveLiterals composeOptions { useLiveLiterals = false } LiveLiterals https://issuetracker.google.com/issues/233716574
41. KSP Cache misses
42. KSP Cache misses
43. Debugging task inputs KspTaskJvm.pluginOptions.$0.asTaskInputArgs
44. Debugging task inputs KspTaskJvm.pluginOptions.$0.asTaskInputArgs tasks.withType<KspTaskJvm>().configureEach { }
45. Debugging task inputs tasks.withType<KspTaskJvm>().configureEach { } KspTaskJvm.pluginOptions.$0.asTaskInputArgs
46. Debugging task inputs KspTaskJvm.pluginOptions.$0.asTaskInputArgs tasks.withType<KspTaskJvm>().configureEach { doLast { } }
47. Debugging task inputs KspTaskJvm.pluginOptions.$0.asTaskInputArgs tasks.withType<KspTaskJvm>().configureEach { doLast { } }
48. Debugging task inputs KspTaskJvm.pluginOptions.$0.asTaskInputArgs tasks.withType<KspTaskJvm>().configureEach { doLast { } }
49. Debugging task inputs KspTaskJvm.pluginOptions.$0.asTaskInputArgs tasks.withType<KspTaskJvm>().configureEach { doLast { pluginOptions.get().forEach { } } }
50. Debugging task inputs KspTaskJvm.pluginOptions.$0.asTaskInputArgs tasks.withType<KspTaskJvm>().configureEach { doLast { pluginOptions.get().forEach { } } }
51. Debugging task inputs KspTaskJvm.pluginOptions.$0.asTaskInputArgs tasks.withType<KspTaskJvm>().configureEach { doLast { pluginOptions.get().forEach { } } }
52. Debugging task inputs KspTaskJvm.pluginOptions.$0.asTaskInputArgs tasks.withType<KspTaskJvm>().configureEach { doLast { pluginOptions.get().forEach { it.getAsTaskInputArgs().forEach { } } } }
53. Debugging task inputs KspTaskJvm.pluginOptions.$0.asTaskInputArgs tasks.withType<KspTaskJvm>().configureEach { doLast { pluginOptions.get().forEach { it.getAsTaskInputArgs().forEach { } } } }
54. Debugging task inputs KspTaskJvm.pluginOptions.$0.asTaskInputArgs tasks.withType<KspTaskJvm>().configureEach { doLast { pluginOptions.get().forEach { it.getAsTaskInputArgs().forEach { buildScan.value( "${this@doLast.name}-${it.key}", it.value, ) } } } }
55. Debugging task inputs KspTaskJvm.pluginOptions.$0.asTaskInputArgs tasks.withType<KspTaskJvm>().configureEach { doLast { pluginOptions.get().forEach { it.getAsTaskInputArgs().forEach { rootProject.buildScan.value( "${this@doLast.name}-${it.key}", it.value, ) } } } }
56. Debugging task inputs KspTaskJvm.pluginOptions.$0.asTaskInputArgs tasks.withType<KspTaskJvm>().configureEach { doLast { pluginOptions.get().forEach { it.getAsTaskInputArgs().forEach { rootProject.buildScan.value( "${this@doLast.name}-${it.key}", it.value, ) } } } }
57. Debugging task inputs KspTaskJvm.pluginOptions.$0.asTaskInputArgs
58. KSP Cache misses build.gradle.kts class RoomSchemaArgProvider( @get:InputDirectory @get:PathSensitive(PathSensitivity.RELATIVE) val schemaDir: File, ) : CommandLineArgumentProvider { override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}") } extensions.configure<KspExtension> { arg(RoomSchemaArgProvider(File(projectDir, "schemas"))) } h tt ps:/ /developer.android.com/training/data-storage/room/migra ti ng-db-versions#test
59. KSP Cache misses build.gradle.kts class RoomSchemaArgProvider( @get:InputDirectory @get:PathSensitive(PathSensitivity.RELATIVE) val schemaDir: File, ) : CommandLineArgumentProvider { override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}") } extensions.configure<KspExtension> { arg(RoomSchemaArgProvider(File(projectDir, "schemas"))) } h tt ps:/ /developer.android.com/training/data-storage/room/migra ti ng-db-versions#test
60. KSP Cache misses build.gradle.kts class RoomSchemaArgProvider( @get:InputDirectory @get:PathSensitive(PathSensitivity.RELATIVE) val schemaDir: File, ) : CommandLineArgumentProvider { override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}") } extensions.configure<KspExtension> { arg(RoomSchemaArgProvider(File(projectDir, "schemas"))) } h tt ps:/ /developer.android.com/training/data-storage/room/migra ti ng-db-versions#test
61. KSP Cache misses build.gradle.kts class RoomSchemaArgProvider( @get:InputDirectory @get:PathSensitive(PathSensitivity.RELATIVE) val schemaDir: File, ) : CommandLineArgumentProvider { override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}") } extensions.configure<KspExtension> { arg(RoomSchemaArgProvider(File(projectDir, "schemas"))) } h tt ps:/ /developer.android.com/training/data-storage/room/migra ti ng-db-versions#test
62. KSP Cache misses KspTaskJvm.pluginOptions.$0.asTaskInputArgs
63. KSP Cache misses build.gradle.kts class RoomSchemaArgProvider( @get:InputDirectory @get:PathSensitive(PathSensitivity.RELATIVE) val schemaDir: File, ) : CommandLineArgumentProvider { override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}") }
64. KSP Cache misses build.gradle.kts class RoomSchemaArgProvider( @get:InputDirectory @get:PathSensitive(PathSensitivity.RELATIVE) val schemaDir: File, ) : CommandLineArgumentProvider { override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}") }
65. KSP Cache misses KspSubplugin.kt commandLineArgumentProviders.get().forEach { it.asArguments().forEach { argument -> options += SubpluginOption("apoption", argument) } }
66. KSP Cache misses KspSubplugin.kt commandLineArgumentProviders.get().forEach { it.asArguments().forEach { argument -> options += SubpluginOption("apoption", argument) } }
67. KSP Cache misses KspSubplugin.kt commandLineArgumentProviders.get().forEach { it.asArguments().forEach { argument -> options += InternalSubpluginOption("apoption", argument) } }
68. KSP Cache misses KspSubplugin.kt commandLineArgumentProviders.get().forEach { it.asArguments().forEach { argument -> options += InternalSubpluginOption("apoption", argument) } } To be released…
69. Configuration cache misses Calculating task graph as configuration cache cannot be reused because...
70. Configuration cache misses Calculating task graph as configuration cache cannot be reused because an input to plugin 'com.android.internal.library' has changed. • How to debug? • Check con fi gura ti on cache report?
71. Configuration cache misses
72. Configuration cache misses 1. A tt ach Debugger to Gradle Build Process 1../gradlew help -Dorg.gradle.debug=true 2. Set breakpoint in ConfigurationCacheFingerprintWriter and ConfigurationCacheFingerprintChecker 3. Print fi les which are changing build/generated/source/kapt/debug read at con fi gura ti on ti me
73. Con fi gura ti on inputs detec ti on improvements Now detects FileCollec ti ons queried at con fi gura ti on ti me h tt ps:/ /docs.gradle.org/8.1/release-notes.html#con fi gura ti on-inputs-detec ti on-improvements Why only in Gradle 8.1?
74. Why only in Gradle 8.1? h tt ps:/ /youtrack.jetbrains.com/issue/KT-58167/Applying-KAPT-plugin-to-Gradle-8.1-causes-con fi gura ti on-cache-miss Using KAPT Gradle plugin with AGP and Gradle 8.1 or higher causes a con fi gura ti on cache miss h tt ps:/ /issuetracker.google.com/issues/285320724 Lint inputs and registerExternalAptJavaOutput are incompa ti ble with Gradle Con fi gura ti on cache Will be fi xed in AGP 8.1.1 and AGP 8.2🤞
76. So you fixed all cache misses and understood how your build times are distributed…
77. But Gradle Enterprise can help you proactively too!
78. Gradle Enterprise API h tt ps:/ /docs.gradle.com/enterprise/api-manual
79. /api/builds /api/build-cache /api/test-distribution Gradle Enterprise API
80. Introducing… Gradle Enterprise Test API 🥳 BETA
81. Automate all the things!
82. Integrate all the things!
83. Retrieve test suites and test case details, filtering by: - test container (e.g test suite, etc) - test outcome (success, failed, flaky etc) - tags - custom values - much more and a combination of all! Gradle Enterprise Test API BETA
84. Gradle Enterprise Test API BETA
85. Gradle Enterprise Test API BETA
86. Gradle Enterprise Test API BETA What is a fl aky test according to Gradle Enterprise? • Non-determinis ti c outcome • Fails and succeeds within the execu ti on of a single Gradle task W
87. Prevent flaky tests from creeping in
88. Alert on tests that became flaky
89. Cross reference with your repository structure…
90. … and provide a comprehensive report
91. … and provide a comprehensive report Build Scan® aggrega ti on
92. … and provide a comprehensive report Knowledge on module structure
93. … and provide a comprehensive report Rela ti onship to common code snippets
94. … and provide a comprehensive report Direct ownership assignment based on build scan a tt rs
95. … identify issues earlier
96. Cross reference with your CI infrastructure… Exposed in API De fi ne how fl akiness compares across CI jobs, infrastructure and target matrix
97. … and make it smarter
98. … and make it smarter
99. Reach out to us after the talk for access to Gradle Enterprise Test API 🥳 BETA
100. Reach out to us after the talk for Importing Instrumentation Test Results to Gradle Enterprise BETA
101. Artifact transforms
102. What are artifact transforms?
103. What are artifact transforms? • Dependency management concept • Changes dependency from incorrect format to requested format via Ar ti fact Transforms • Example: classes jar to dex fi le • Users do not directly request transforms • Gradle schedules Transforms between units of work • Transforms can depend on tasks or transforms
104. Artifact transforms are now visible in Build Scans! 🥳 Must use: Gradle Enterprise 2023.1+, Gradle Enterprise Gradle Plugin 3.13+, Gradle Build Tool 8.1+ 
 Only visualizes project ar ti fact transforma ti ons
105. Artifact transforms are now visible in Build Scans!
106. Artifact transforms are now visible in Build Scans! Details and dura ti on
107. Why are transforms important in Build Scans? • Understand transform dependencies • Understand gaps in ti meline • Understanding ar ti fact transform errors
108. Understand transform dependencies
109. Transform visibility fills gaps in timeline
110. Transform visibility fills gaps in timeline
111. View transform failures in Build Scans
112. Search for transform failures in dashboard
113. Artifact transforms upcoming work • Understand transform cacheability and up-to-date checks • Improved transform fi lter in ti meline • Support more transform types (je ti fi er transforms are not visible)
115. More Resources Gradle Training https:/ /gradle.com/training/ https:/ /gradle.com/training/introduction-to-gradle https:/ /gradle.com/training/build-cache-deep-dive Gradle Community Slack https:/ /gradle.com/slack-invite
116. Thank you! Gradle Enterprise Trial osacky.com h tt ps:/ /gradle.com/enterprise/trial/ https:/ /gradle.com/careers/ Gradle is hiring @nellyspageli @juliozynger