Slide 1
Slide 1 text
Improving developer productivity with
Gradle Enterprise in 2023
Nelson Osacky and Julio Zynger
Slide 2
Slide 2 text
Julio
• Previously Android Engineer
• Large projects
• SoundCloud, Autodesk
• Developer Produc
ti
vity Enthusiast
• Leading Gradle’s (and previously SoundCloud’s)
interdisciplinary Pla
tf
orm teams
• Backstage contributor
https://github.com/backstage/backstage
Lead Dev Platform Engineer
Slide 3
Slide 3 text
Nelson
• Previously Android Engineer
• Large projects
• SoundCloud
• Square
• Small startups
• Gradle Plugin Maintainer
• Fladle - Easily Scale Instrumenta
ti
on Tests on Firebase
https://github.com/runningcode/
fl
adle
• Gradle Doctor - Ac
ti
onable Insights for your build
https://github.com/runningcode/gradle-doctor
Lead Solutions Engineer
Slide 4
Slide 4 text
We see a lot of builds
from 10 person teams to 1000+
Android, Gradle and Maven
Slide 5
Slide 5 text
• Optimize and understand builds
• Increase build stability
• Preach Developer Productivity
Slide 6
Slide 6 text
• Improve Gradle ecosystem
• Fix issues in open source plugin
affecting customers
Slide 7
Slide 7 text
• Meet with Google and Jetbrains
to help prioritize issues
impacting customers
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
Android Development
Feature Development Tech Debt
(There are other things too)
Slide 10
Slide 10 text
Tech Debt
Refactoring Improving builds
Slide 11
Slide 11 text
Slow builds are Tech Debt
And it always pays off
Slide 12
Slide 12 text
Cost of Builds
60s waste * 50 builds / day * 50 devs
= 42 hours lost / day
Slide 13
Slide 13 text
Cost of Builds
60s waste * 50 builds / day * 50 devs
= 42 hours lost / day
not including lost focus
h
tt
ps:/
/gradle.com/roi-calculator
Slide 14
Slide 14 text
Fast Builds Matter
Slide 15
Slide 15 text
Cost of Builds
60s waste * 50 builds / day * 50 devs
= 42 hours lost / day
hire 5 new people without paying
them! no recruiting
h
tt
ps:/
/gradle.com/roi-calculator
Slide 16
Slide 16 text
Slow builds are Tech Debt
And it always pays off
And is easy to justify working on it
Slide 17
Slide 17 text
No content
Slide 18
Slide 18 text
https://gradle.com/enterprise/trial/
Slide 19
Slide 19 text
How many have seen or used a
Build Scan?
Slide 20
Slide 20 text
How many are Gradle Enterprise
customers?
Slide 21
Slide 21 text
https:/
/gradle.com/enterprise-customers/oss-projects/
Gradle Enterprise OSS Projects
Slide 22
Slide 22 text
What is build caching?
https:/
/docs.gradle.org/current/userguide/build_cache.html
Slide 23
Slide 23 text
JavaCompile task
Source Files
Compiler Args
Dependencies
Class files
any task can be made cacheable
Slide 24
Slide 24 text
Cache key
Source Files
Compiler Args
Dependencies
Slide 25
Slide 25 text
Cache key
Source Files
Compiler Args
Dependencies
Build cache Class files
Slide 26
Slide 26 text
What is a cache miss?
Slide 27
Slide 27 text
What is a cache miss?
•Expect a cache hit but cache key is di
ff
erent
•Task outcome is the same
•Only inputs that a
ff
ect outcome should be tracked
Slide 28
Slide 28 text
No content
Slide 29
Slide 29 text
Task input comparison
Slide 30
Slide 30 text
No content
Slide 31
Slide 31 text
Common Cache misses
Slide 32
Slide 32 text
Build Configuration Differences
Slide 33
Slide 33 text
Build Configuration Differences
Slide 34
Slide 34 text
Build Configuration Differences
kotlin.incremental=false kotlin.incremental=true
CI Local
Slide 35
Slide 35 text
Build Configuration Differences
kotlin.incremental=false kotlin.incremental=true
CI Local
Incremental has no effect on outcome of task
Slide 36
Slide 36 text
Build Configuration Differences
kotlin.incremental=true kotlin.incremental=true
CI Local
Incremental has no effect on outcome of task
h
tt
ps:/
/youtrack.jetbrains.com/issue/KT-40974/isIncrementalCompila
ti
onEnabled-should-not-be-marked-as-Input
Slide 37
Slide 37 text
LiveLiterals
Slide 38
Slide 38 text
LiveLiterals
Slide 39
Slide 39 text
LiveLiterals
Slide 40
Slide 40 text
•LiveLiterals$AnalyticsEvent.kt contains fully
qualified path
•Fix is to disable LiveLiterals
composeOptions {
useLiveLiterals = false
}
LiveLiterals
https://issuetracker.google.com/issues/233716574
Slide 41
Slide 41 text
KSP Cache misses
Slide 42
Slide 42 text
KSP Cache misses
Slide 43
Slide 43 text
Debugging task inputs
KspTaskJvm.pluginOptions.$0.asTaskInputArgs
Slide 44
Slide 44 text
Debugging task inputs
KspTaskJvm.pluginOptions.$0.asTaskInputArgs
tasks.withType().configureEach {
}
Slide 45
Slide 45 text
Debugging task inputs
tasks.withType().configureEach {
}
KspTaskJvm.pluginOptions.$0.asTaskInputArgs
Slide 46
Slide 46 text
Debugging task inputs
KspTaskJvm.pluginOptions.$0.asTaskInputArgs
tasks.withType().configureEach {
doLast {
}
}
Slide 47
Slide 47 text
Debugging task inputs
KspTaskJvm.pluginOptions.$0.asTaskInputArgs
tasks.withType().configureEach {
doLast {
}
}
Slide 48
Slide 48 text
Debugging task inputs
KspTaskJvm.pluginOptions.$0.asTaskInputArgs
tasks.withType().configureEach {
doLast {
}
}
Slide 49
Slide 49 text
Debugging task inputs
KspTaskJvm.pluginOptions.$0.asTaskInputArgs
tasks.withType().configureEach {
doLast {
pluginOptions.get().forEach {
}
}
}
Slide 50
Slide 50 text
Debugging task inputs
KspTaskJvm.pluginOptions.$0.asTaskInputArgs
tasks.withType().configureEach {
doLast {
pluginOptions.get().forEach {
}
}
}
Slide 51
Slide 51 text
Debugging task inputs
KspTaskJvm.pluginOptions.$0.asTaskInputArgs
tasks.withType().configureEach {
doLast {
pluginOptions.get().forEach {
}
}
}
Slide 52
Slide 52 text
Debugging task inputs
KspTaskJvm.pluginOptions.$0.asTaskInputArgs
tasks.withType().configureEach {
doLast {
pluginOptions.get().forEach {
it.getAsTaskInputArgs().forEach {
}
}
}
}
Slide 53
Slide 53 text
Debugging task inputs
KspTaskJvm.pluginOptions.$0.asTaskInputArgs
tasks.withType().configureEach {
doLast {
pluginOptions.get().forEach {
it.getAsTaskInputArgs().forEach {
}
}
}
}
Slide 54
Slide 54 text
Debugging task inputs
KspTaskJvm.pluginOptions.$0.asTaskInputArgs
tasks.withType().configureEach {
doLast {
pluginOptions.get().forEach {
it.getAsTaskInputArgs().forEach {
buildScan.value(
"${[email protected]}-${it.key}", it.value,
)
}
}
}
}
Slide 55
Slide 55 text
Debugging task inputs
KspTaskJvm.pluginOptions.$0.asTaskInputArgs
tasks.withType().configureEach {
doLast {
pluginOptions.get().forEach {
it.getAsTaskInputArgs().forEach {
rootProject.buildScan.value(
"${[email protected]}-${it.key}", it.value,
)
}
}
}
}
Slide 56
Slide 56 text
Debugging task inputs
KspTaskJvm.pluginOptions.$0.asTaskInputArgs
tasks.withType().configureEach {
doLast {
pluginOptions.get().forEach {
it.getAsTaskInputArgs().forEach {
rootProject.buildScan.value(
"${[email protected]}-${it.key}", it.value,
)
}
}
}
}
Slide 57
Slide 57 text
Debugging task inputs
KspTaskJvm.pluginOptions.$0.asTaskInputArgs
Slide 58
Slide 58 text
KSP Cache misses
build.gradle.kts
class RoomSchemaArgProvider(
@get:InputDirectory
@get:PathSensitive(PathSensitivity.RELATIVE)
val schemaDir: File,
) : CommandLineArgumentProvider {
override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}")
}
extensions.configure {
arg(RoomSchemaArgProvider(File(projectDir, "schemas")))
}
h
tt
ps:/
/developer.android.com/training/data-storage/room/migra
ti
ng-db-versions#test
Slide 59
Slide 59 text
KSP Cache misses
build.gradle.kts
class RoomSchemaArgProvider(
@get:InputDirectory
@get:PathSensitive(PathSensitivity.RELATIVE)
val schemaDir: File,
) : CommandLineArgumentProvider {
override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}")
}
extensions.configure {
arg(RoomSchemaArgProvider(File(projectDir, "schemas")))
}
h
tt
ps:/
/developer.android.com/training/data-storage/room/migra
ti
ng-db-versions#test
Slide 60
Slide 60 text
KSP Cache misses
build.gradle.kts
class RoomSchemaArgProvider(
@get:InputDirectory
@get:PathSensitive(PathSensitivity.RELATIVE)
val schemaDir: File,
) : CommandLineArgumentProvider {
override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}")
}
extensions.configure {
arg(RoomSchemaArgProvider(File(projectDir, "schemas")))
}
h
tt
ps:/
/developer.android.com/training/data-storage/room/migra
ti
ng-db-versions#test
Slide 61
Slide 61 text
KSP Cache misses
build.gradle.kts
class RoomSchemaArgProvider(
@get:InputDirectory
@get:PathSensitive(PathSensitivity.RELATIVE)
val schemaDir: File,
) : CommandLineArgumentProvider {
override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}")
}
extensions.configure {
arg(RoomSchemaArgProvider(File(projectDir, "schemas")))
}
h
tt
ps:/
/developer.android.com/training/data-storage/room/migra
ti
ng-db-versions#test
Slide 62
Slide 62 text
KSP Cache misses
KspTaskJvm.pluginOptions.$0.asTaskInputArgs
Slide 63
Slide 63 text
KSP Cache misses
build.gradle.kts
class RoomSchemaArgProvider(
@get:InputDirectory
@get:PathSensitive(PathSensitivity.RELATIVE)
val schemaDir: File,
) : CommandLineArgumentProvider {
override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}")
}
Slide 64
Slide 64 text
KSP Cache misses
build.gradle.kts
class RoomSchemaArgProvider(
@get:InputDirectory
@get:PathSensitive(PathSensitivity.RELATIVE)
val schemaDir: File,
) : CommandLineArgumentProvider {
override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}")
}
Slide 65
Slide 65 text
KSP Cache misses
KspSubplugin.kt
commandLineArgumentProviders.get().forEach {
it.asArguments().forEach { argument ->
options += SubpluginOption("apoption", argument)
}
}
Slide 66
Slide 66 text
KSP Cache misses
KspSubplugin.kt
commandLineArgumentProviders.get().forEach {
it.asArguments().forEach { argument ->
options += SubpluginOption("apoption", argument)
}
}
Slide 67
Slide 67 text
KSP Cache misses
KspSubplugin.kt
commandLineArgumentProviders.get().forEach {
it.asArguments().forEach { argument ->
options += InternalSubpluginOption("apoption", argument)
}
}
Slide 68
Slide 68 text
KSP Cache misses
KspSubplugin.kt
commandLineArgumentProviders.get().forEach {
it.asArguments().forEach { argument ->
options += InternalSubpluginOption("apoption", argument)
}
}
To be released…
Slide 69
Slide 69 text
Configuration cache misses
Calculating task graph as configuration cache cannot be reused
because...
Slide 70
Slide 70 text
Configuration cache misses
Calculating task graph as configuration cache cannot be reused
because an input to plugin 'com.android.internal.library' has
changed.
• How to debug?
• Check con
fi
gura
ti
on cache report?
Slide 71
Slide 71 text
Configuration cache misses
Slide 72
Slide 72 text
Configuration cache misses
1. A
tt
ach Debugger to Gradle Build Process
1../gradlew help -Dorg.gradle.debug=true
2. Set breakpoint in ConfigurationCacheFingerprintWriter
and ConfigurationCacheFingerprintChecker
3. Print
fi
les which are changing
build/generated/source/kapt/debug read at con
fi
gura
ti
on
ti
me
Slide 73
Slide 73 text
Con
fi
gura
ti
on inputs detec
ti
on improvements
Now detects FileCollec
ti
ons queried at con
fi
gura
ti
on
ti
me
h
tt
ps:/
/docs.gradle.org/8.1/release-notes.html#con
fi
gura
ti
on-inputs-detec
ti
on-improvements
Why only in Gradle 8.1?
Slide 74
Slide 74 text
Why only in Gradle 8.1?
h
tt
ps:/
/youtrack.jetbrains.com/issue/KT-58167/Applying-KAPT-plugin-to-Gradle-8.1-causes-con
fi
gura
ti
on-cache-miss
Using KAPT Gradle plugin with AGP and Gradle 8.1 or higher causes a con
fi
gura
ti
on cache miss
h
tt
ps:/
/issuetracker.google.com/issues/285320724
Lint inputs and registerExternalAptJavaOutput are incompa
ti
ble with Gradle Con
fi
gura
ti
on cache
Will be
fi
xed in AGP 8.1.1 and AGP 8.2🤞
Slide 75
Slide 75 text
No content
Slide 76
Slide 76 text
So you fixed all cache misses and
understood how your build times
are distributed…
Slide 77
Slide 77 text
But Gradle Enterprise can help
you proactively too!
Slide 78
Slide 78 text
Gradle Enterprise API
h
tt
ps:/
/docs.gradle.com/enterprise/api-manual
Slide 79
Slide 79 text
/api/builds
/api/build-cache
/api/test-distribution
Gradle Enterprise API
Slide 80
Slide 80 text
Introducing…
Gradle Enterprise Test API
🥳
BETA
Slide 81
Slide 81 text
Automate
all the things!
Slide 82
Slide 82 text
Integrate
all the things!
Slide 83
Slide 83 text
Retrieve test suites and test case details,
filtering by:
- test container (e.g test suite, etc)
- test outcome (success, failed, flaky etc)
- tags
- custom values
- much more and a combination of all!
Gradle Enterprise Test API
BETA
Slide 84
Slide 84 text
Gradle Enterprise Test API
BETA
Slide 85
Slide 85 text
Gradle Enterprise Test API
BETA
Slide 86
Slide 86 text
Gradle Enterprise Test API
BETA
What is a
fl
aky test according to Gradle Enterprise?
• Non-determinis
ti
c outcome
• Fails and succeeds within the execu
ti
on of a single Gradle task
W
Slide 87
Slide 87 text
Prevent flaky tests from creeping in
Slide 88
Slide 88 text
Alert on tests that became flaky
Slide 89
Slide 89 text
Cross reference with your repository structure…
Slide 90
Slide 90 text
… and provide a comprehensive report
Slide 91
Slide 91 text
… and provide a comprehensive report
Build Scan® aggrega
ti
on
Slide 92
Slide 92 text
… and provide a comprehensive report
Knowledge on
module structure
Slide 93
Slide 93 text
… and provide a comprehensive report
Rela
ti
onship to
common code
snippets
Slide 94
Slide 94 text
… and provide a comprehensive report
Direct ownership assignment based on build scan a
tt
rs
Slide 95
Slide 95 text
… identify issues earlier
Slide 96
Slide 96 text
Cross reference with your CI infrastructure…
Exposed in API
De
fi
ne how
fl
akiness compares across CI
jobs, infrastructure and target matrix
Slide 97
Slide 97 text
… and make it smarter
Slide 98
Slide 98 text
… and make it smarter
Slide 99
Slide 99 text
Reach out to us after the talk for access to
Gradle Enterprise Test API
🥳
BETA
Slide 100
Slide 100 text
Reach out to us after the talk for
Importing Instrumentation Test
Results to Gradle Enterprise
BETA
Slide 101
Slide 101 text
Artifact transforms
Slide 102
Slide 102 text
What are artifact transforms?
Slide 103
Slide 103 text
What are artifact transforms?
• Dependency management concept
• Changes dependency from incorrect format to
requested format via Ar
ti
fact Transforms
• Example: classes jar to dex
fi
le
• Users do not directly request transforms
• Gradle schedules Transforms between units of work
• Transforms can depend on tasks or transforms
Slide 104
Slide 104 text
Artifact transforms are now visible in Build Scans!
🥳
Must use: Gradle Enterprise 2023.1+, Gradle Enterprise Gradle Plugin 3.13+, Gradle Build Tool 8.1+
Only visualizes project ar
ti
fact transforma
ti
ons
Slide 105
Slide 105 text
Artifact transforms are now visible in Build Scans!
Slide 106
Slide 106 text
Artifact transforms are now visible in Build Scans!
Details and dura
ti
on
Slide 107
Slide 107 text
Why are transforms important in Build Scans?
• Understand transform dependencies
• Understand gaps in
ti
meline
• Understanding ar
ti
fact transform errors
Slide 108
Slide 108 text
Understand transform dependencies
Slide 109
Slide 109 text
Transform visibility fills gaps in timeline
Slide 110
Slide 110 text
Transform visibility fills gaps in timeline
Slide 111
Slide 111 text
View transform failures in Build Scans
Slide 112
Slide 112 text
Search for transform failures in dashboard
Slide 113
Slide 113 text
Artifact transforms upcoming work
• Understand transform cacheability and up-to-date checks
• Improved transform
fi
lter in
ti
meline
• Support more transform types (je
ti fi
er transforms are not visible)
Slide 114
Slide 114 text
No content
Slide 115
Slide 115 text
More Resources
Gradle Training
https:/
/gradle.com/training/
https:/
/gradle.com/training/introduction-to-gradle
https:/
/gradle.com/training/build-cache-deep-dive
Gradle Community Slack
https:/
/gradle.com/slack-invite
Slide 116
Slide 116 text
Thank you!
Gradle Enterprise Trial
osacky.com
h
tt
ps:/
/gradle.com/enterprise/trial/
https:/
/gradle.com/careers/
Gradle is hiring
@nellyspageli
@juliozynger