Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Improve Build Times in Less Time

Improve Build Times in Less Time

Build times are never as fast as we want but there is a lot you can do to improve them today without being a Gradle domain expert or buying everyone new computers! In this talk we'll cover a number of different tools, patterns, and common pitfalls to avoid for improving your build times quickly and take your (team's) developer productivity to the next level.

Zac Sweers

April 26, 2022
Tweet

More Decks by Zac Sweers

Other Decks in Programming

Transcript

  1. Zac Sweers – Slack – @ZacSweers Improve Build Times in

    Less Time https://speakerdeck.com/zacsweers/improve-build-times-in-less-time
  2. Gradle Refresher • Tasks • Projects • Plugins • Conf

    i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing
  3. Tasks @CacheableTask abstract class PrintMessageTask : DefaultTask() { @get:Input abstract

    val inputMessage: Property<String> @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun run() { val message = inputMessage.get() println(message) outputFile.asFile.get().writeText(message) } }
  4. Tasks • Inputs and outputs @CacheableTask abstract class PrintMessageTask :

    DefaultTask() { @get:Input abstract val inputMessage: Property<String> @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun run() { val message = inputMessage.get() println(message) outputFile.asFile.get().writeText(message) } }
  5. Tasks • Inputs and outputs • Parallelizable @CacheableTask abstract class

    PrintMessageTask : DefaultTask() { @get:Input abstract val inputMessage: Property<String> @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun run() { val message = inputMessage.get() println(message) outputFile.asFile.get().writeText(message) } }
  6. Tasks • Inputs and outputs • Parallelizable • Task action

    @CacheableTask abstract class PrintMessageTask : DefaultTask() { @get:Input abstract val inputMessage: Property<String> @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun run() { val message = inputMessage.get() println(message) outputFile.asFile.get().writeText(message) } }
  7. Tasks • Inputs and outputs • Parallelizable • Task action

    • Cacheability @CacheableTask abstract class PrintMessageTask : DefaultTask() { @get:Input abstract val inputMessage: Property<String> @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun run() { val message = inputMessage.get() println(message) outputFile.asFile.get().writeText(message) } }
  8. Tasks • Inputs and outputs • Parallelizable • Task action

    • Cacheability @CacheableTask abstract class PrintMessageTask : DefaultTask() { @get:Input abstract val inputMessage: Property<String> @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun run() { val message = inputMessage.get() println(message) outputFile.asFile.get().writeText(message) } }
  9. Tasks • Inputs and outputs • Parallelizable • Task action

    • Cacheability • Con fi gured at con fi guration-time • Should be lazy! project.tasks.register<PrintMessageTask>("printMessage") { inputMessage.set("Hello World!") outputFile.set( project.layout.buildDirectory. fi le( "outputs/logs/message.txt")) }
  10. Tasks • Inputs and outputs • Parallelizable • Task action

    • Cacheability • Con fi gured at con fi guration-time • Should be lazy! @CacheableTask abstract class PrintMessageTask : DefaultTask() { @get:Input abstract val inputMessage: Property<String> @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun run() { val message = inputMessage.get() println(message) outputFile.asFile.get().writeText(message) } }
  11. Tasks • Inputs and outputs • Parallelizable • Task action

    • Cacheability • Con fi gured at con fi guration-time • Should be lazy! • Some common types (SourceTask, CompileTask, etc) @CacheableTask abstract class PrintMessageTask : DefaultTask() { @get:Input abstract val inputMessage: Property<String> @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun run() { val message = inputMessage.get() println(message) outputFile.asFile.get().writeText(message) } }
  12. Gradle Refresher • Tasks • Projects • Plugins • Conf

    i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing
  13. Projects • Projects contain a set of tasks • Also

    contain things like source sets • Can have subprojects • Every directory in between is technically a project • Can have plugins applied to them
  14. Gradle Refresher • Tasks • Projects • Plugins • Conf

    i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing
  15. Plugins • Reusable implementations of gradle logic • Well known

    examples • Android Gradle Plugin (AGP) • Kotlin Gradle Plugin (KGP) • Many 1st-party plugins • Ours too! github.com/slackhq/slack-gradle-plugin plugins { kotlin("jvm") id("com.android.library") `java-platform` }
  16. Gradle Refresher • Tasks • Projects • Plugins • Conf

    i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing
  17. Conf i gurations • Essentially named fi le collections •

    “implementation”, “api”, etc dependencies { implementation("com.squareup.moshi:moshi:1.13.0") testImplementation("junit:junit:4.13") }
  18. Conf i gurations • Essentially named fi le collections •

    “implementation”, “api”, etc dependencies { implementation("com.squareup.moshi:moshi:1.13.0") testImplementation("junit:junit:4.13") }
  19. Gradle Refresher • Tasks • Projects • Plugins • Conf

    i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing
  20. Conf i guration • Don’t confuse it with con fi

    gurations • The time spent from “go” to tasks actually executing
  21. Gradle Refresher • Tasks • Projects • Plugins • Conf

    i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing
  22. Caching/Incremental Builds • Multiple kinds • build dir • Build

    cache • Local/Remote • Con fi guration cache • GRADLE_USER_HOME • Not Studio caches https://www.liutikas.net/2022/04/19/Caches-Everywhere.html
  23. Gradle Refresher • Tasks • Projects • Plugins • Conf

    i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing
  24. Incremental Compilation (IC) • Multiple kinds • Java – Gradle

    – JavaCompile • Kotlin – JB/Google – KotlinCompile* • kaptGenerateStubs*Kotlin • kapt*Kotlin • compile*Kotlin • compile*JavaWithKotlinC
  25. Incremental Compilation (IC) – Java • Compilation avoidance • “Can

    we not run?” • ABI-based • Incremental Compilation • “Can we only recompile a subset?” https://blog.gradle.org/incremental-compiler-avoidance :project-a :project-b
  26. Incremental Compilation (IC) – Java • Compilation avoidance • “Can

    we not run?” • ABI-based • Incremental Compilation • “Can we only recompile a subset?” https://blog.gradle.org/incremental-compiler-avoidance :project-a :project-b
  27. Incremental Compilation (IC) – Java • Compilation avoidance • “Can

    we not run?” • ABI-based • Incremental Compilation • “Can we only recompile a subset?” https://blog.gradle.org/incremental-compiler-avoidance :project-a :project-b
  28. Incremental Compilation (IC) – Kotlin • Incremental only for now,

    no avoidance • Coming soon though! • Depends on producer-side IC data • This makes it tricky
  29. Incremental Compilation (IC) – Kotlin • Incremental only for now,

    no avoidance • Coming soon though! • Depends on producer-side IC data • This makes it tricky :project-a :project-b
  30. Incremental Compilation (IC) – Kotlin • Incremental only for now,

    no avoidance • Coming soon though! • Depends on producer-side IC data • This makes it tricky :project-a :project-b <IC data> produces consumes
  31. Incremental Compilation (IC) – Kotlin • Incremental only for now,

    no avoidance • Coming soon though! • Depends on producer-side IC data • This makes it tricky :project-a :project-b <IC data> produces consumes
  32. • Multiple kinds • Java – Gradle – JavaCompile •

    Kotlin – JB/Google – KotlinCompile* • kaptGenerateStubs*Kotlin • kapt*Kotlin • compile*Kotlin • compile*JavaWithKotlinC Incremental Compilation (IC) – Kotlin
  33. Incremental Compilation (IC) – Kotlin kotlinc Frontend Backend AnalysisHandler IR

    Lowering https://medium.com/google-developer-experts/crash-course-on-the- kotlin-compiler-1-frontend-parsing-phase-9898490d922b Parse/PSI psi2ir
  34. Incremental Compilation (IC) – Kotlin Frontend Backend AnalysisHandler IR Lowering

    Parse/PSI psi2ir kaptGenerateStubs*Kotlin kapt*Kotlin compile*Kotlin compile*JavaWithKotlinC
  35. Incremental Compilation (IC) – Kotlin Frontend Backend AnalysisHandler IR Lowering

    Parse/PSI psi2ir kaptGenerateStubs*Kotlin kapt*Kotlin compile*Kotlin compile*JavaWithKotlinC
  36. Incremental Compilation (IC) – Kotlin Frontend Backend AnalysisHandler IR Lowering

    Parse/PSI psi2ir kaptGenerateStubs*Kotlin kapt*Kotlin compile*Kotlin compile*JavaWithKotlinC
  37. Incremental Compilation (IC) – Kotlin Frontend Backend AnalysisHandler IR Lowering

    Parse/PSI psi2ir kaptGenerateStubs*Kotlin kapt*Kotlin compile*Kotlin compile*JavaWithKotlinC
  38. Incremental Compilation (IC) – Kotlin Frontend Backend AnalysisHandler IR Lowering

    Parse/PSI psi2ir kaptGenerateStubs*Kotlin kapt*Kotlin compile*Kotlin compile*JavaWithKotlinC
  39. Gradle Refresher • Tasks • Projects • Plugins • Conf

    i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing
  40. Gradle Refresher • Tasks • Projects • Plugins • Conf

    i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing
  41. Annotation Processing • Non - incremental • Two types of

    incremental • ISOLATING (Dagger, Moshi, etc) • AGGREGATING (Glide) javac/kapt KSP
  42. Gradle Refresher • Tasks • Projects • Plugins • Conf

    i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing
  43. Things That Are Hurting Your Build Things That Will Improve

    Your Build Things That Will Keep Your Build Fast
  44. Kapt • Minimum 3 compilation tasks • 4 if you

    generate/have Java sources (hi, Dagger!) • Brittle to classpath changes • Disables new FIR • Keeps appearing in later problems as a multiplier
  45. kotlinc 
 (Frontend only) Stub gen .java f i les

    Kapt (javac) Kapt .java/.kt f i les kotlinc 
 (Frontend and Backend) compileKotlin .class f i les
  46. kotlinc 
 (Frontend only) Stub gen .java f i les

    Kapt (javac) Kapt .java/.kt f i les kotlinc 
 (Frontend and Backend) compileKotlin .class f i les javac compileJava More 
 .class f i les
  47. kotlinc 
 (Frontend only) Stub gen .java f i les

    Kapt (javac) Kapt .java/.kt f i les kotlinc 
 (Frontend and Backend) compileKotlin .class f i les javac compileJava More 
 .class f i les
  48. kotlinc 
 (Frontend only) Stub gen .java f i les

    Kapt (javac) Kapt .java/.kt f i les kotlinc 
 (Frontend and Backend) compileKotlin .class f i les javac compileJava More 
 .class f i les
  49. kotlinc 
 (Frontend only) KSP .java/.kt f i les javac

    javaCompile More .class f i les kotlinc 
 (Frontend and Backend) compileKotlin .class f i les
  50. kotlinc 
 (Frontend only) KSP .kt f i les kotlinc

    
 (Frontend and Backend) compileKotlin .class f i les
  51. kotlinc 
 (Frontend and Backend) compileKotlin .class f i les

    all - open Kotlinx - serialization parcelize moshi etc. https://blog.bnorm.dev/writing-your-second-compiler- plugin-part-1
  52. kotlinc 
 (Frontend only) Stub gen .java f i les

    Kapt (javac) Kapt .java/.kt f i les kotlinc 
 (Frontend and Backend) compileKotlin .class f i les javac compileJava More 
 .class f i les
  53. kotlinc 
 (Frontend only) Stub gen .java f i les

    Kapt (javac) Kapt .java/.kt f i les kotlinc 
 (Frontend and Backend) compileKotlin .class f i les javac compileJava More 
 .class f i les https://github.com/square/anvil
  54. kotlinc 
 (Frontend and Backend) compileKotlin .class f i les

    Anvil runs here https://github.com/square/anvil
  55. buildSrc • Regularly invalidates buildscript classpath • It’s like occasionally

    having - - rerun - tasks • What’s the alternative? • Version catalogs for deps • Separate repo + artifactory for plugins
  56. buildSrc • What about live testing? • Use included builds

    if ("slack.plugin.internal.localPath" in extra.properties) { val path = (extra["slack.plugin.internal.localPath"] as String) includeBuild(path) { dependencySubstitution { substitute(module("slack.internal.gradle:slack-plugin-internal")) .using(project(":slack-plugin-internal")) } } }
  57. Gradle API Surface Area con fi gurations.all { } extensions.get(...)

    tasks.getByName(...) var property: String con fi gurations.con fi gureEach { // Lazy! } extensions.con fi gure(...) { // Lazy! } tasks.named(...) { // Lazy! } val property: Property<String> http://autonomousapps.com/blog/rules-for- gradle-plugin-authors.html
  58. Incremental Compilation Issues • Pure Java projects break IC on

    every project that depends on them (directly or transitively) • Java - only, resources - only, etc • Workaround: add a Kotlin f i le • KT-30980 KT-38622 • Code gen tools generating non - deterministic outputs • Resource ABI changes break IC (KT-40772) • Cleaning https://www.zacsweers.dev/ optimizing-your-kotlin-build/
  59. Literally bugs • Kotlin build cache entries often break incremental

    compilation. KT-34862 • Gradle AbstractCompile tasks (i.e. KotlinCompile) are sensitive to classpath jar ordering. gradle/gradle#15626 • Error - prone freeCompilerArgs use. KT-41985 https://www.zacsweers.dev/ optimizing-your-kotlin-build/
  60. Old tools • JDK • We are on 17, moving

    to 18 • Kotlin • Gradle • Gradle plugins • Compiler plugins • Literally anything on your buildscript classpath
  61. Tune GCs org.gradle.jvmargs=... -XX:+UseG1GC \ -XX:+UnlockExperimentalVMOptions \ -XX:G1NewSizePercent=67 \ -XX:G1MaxNewSizePercent=67

    https://github.com/slackhq/slack-gradle-plugin/blob/main/slack- plugin/src/main/kotlin/slack/gradle/tasks/BootstrapTask.kt
  62. Remote Build caching buildCache { local { isEnabled = true

    } remote<HttpBuildCache> { isEnabled = !gradle.startParameter.isOf fl ine setUrl(“…”) // only write to cache from build server isPush = isCi credentials { username = ... password = ... } } }
  63. android - cache - f i x - gradle -

    plugin https://github.com/gradle/android-cache- fi x-gradle-plugin
  64. Non - transitive R classes • 5.5MB or 8.5% of

    Slack APK size • 14% incremental build time improvement w/ layout change • import slack.l10n.R as L10nR
  65. Disable Android Features # Build features that are disabled by

    default in all projects android.defaults.buildfeatures.aidl=false android.defaults.buildfeatures.buildcon fi g=false android.defaults.buildfeatures.renderscript=false android.defaults.buildfeatures.resvalues=false android.defaults.buildfeatures.shaders=false
  66. Make A Platform Plugin import slack.gradle.Platforms plugins { id("slack.base") `java-platform`

    } val catalog = extensions. fi ndByType<VersionCatalogsExtension>() ?. fi nd("libs") ?: error("Could not fi nd libs catalog!") Platforms.applyFromCatalog(project, catalog.get()) https://github.com/slackhq/slack-gradle-plugin/blob/main/slack- plugin/src/main/kotlin/slack/gradle/Platforms.kt
  67. Other Things • Remove jetif i er (use `checkJetif i

    er` task in AGP 7.1) • api/impl projects • api/impl conf i gurations don’t avoid compilations, just control visibility •
  68. Gradle Enterprise • Remote build cache • Build scans •

    Compare builds • Trends/insight/visualization • Export API • Critical path • Tags/custom values • Debugging (everyone has a scan ready)
  69. Telemetry • Local vs CI (GE tag) • System arch

    (rolling out M1 Macs) • Thermals • https: / / github.com/slackhq/slack - gradle - plugin/blob/main/slack - plugin/src/main/kotlin/slack/gradle/util/ThermalsWatcher.kt • VCS state • Env info (OS, version, etc)
  70. modules - graph - assert https://github.com/jraska/modules-graph-assert moduleGraphAssert { maxHeight =

    4 allowed = [':.* -> :core', ':feature.* -> :lib.*'] // regex to match module names restricted = [':feature-[a-z]* -X> :forbidden-to-depend-on'] // regex to match module names con fi gurations = ['api', 'implementation'] // Dependency confs to look. }
  71. dependency - analysis - android - gradle - plugin https://github.com/slackhq/slack-gradle-plugin/blob/main/slack-plugin/src/main/

    kotlin/slack/dependencyrake/DependencyRake.kt /** * Task that consumes the generated advice report json generated by `AdviceTask` * and applies its advice to the project build fi le. This is usually not run * directly, but rather added as a fi nalizer to the `AdviceTask` it reads from. */ class RakeDependencies(...) : AbstractPostProcessingTask() { // ... }
  72. https://github.com/slackhq/slack-gradle-plugin Convention Plugins plugins { id("com.android.library") kotlin("android") id("com.squareup.anvil") } android

    { namespace = "slack.libraries.emoji.api" compileSdk = 31 defaultCon fi g { minSdk = 26 } buildTypes { debug { matchingFallbacks.add("release") } } compileOptions { sourceCompatibility = JavaVersion.VERSION_1_8 targetCompatibility = JavaVersion.VERSION_1_8 } } androidComponents { beforeVariants { builder -> if (builder.buildType == "debug") { builder.enable = false } else { builder.enableAndroidTest = hasProperty("slack.libraryWithAndroidTest") } } } anvil { generateFactories = true } dependencies { compileOnly(libs.dagger.dagger) implementation(projects.libraries.emoji.data) implementation(projects.libraries.foundation.slackCommons.android) implementation(projects.libraries.imageLoading) implementation(projects.libraries.model) implementation(projects.libraries.telemetry.telemetryPublic) implementation(libs.rxJava.java3) }
  73. https://github.com/slackhq/slack-gradle-plugin Convention Plugins plugins { id("slack.base") id("com.android.library") kotlin("android") } slack

    { features { dagger() } } dependencies { implementation(projects.libraries.emoji.data) implementation(projects.libraries.foundation.slackCommons.android) implementation(projects.libraries.imageLoading) implementation(projects.libraries.model) implementation(projects.libraries.telemetry.telemetryPublic) implementation(libs.rxJava.java3) }
  74. Future New Kotlin IC • Consumer - side IC based

    on classpath snapshot • Unstable preview available in 1.6.20, targeting 1.7 • Solves many producer - side issues (Java projects, caching, etc) • True compilation avoidance in some cases ( ! ! ) kotlin.incremental.useClasspathSnapshot=true https://youtrack.jetbrains.com/issue/KT-45777
  75. Future K2 Compiler • New Kotlin compiler frontend (aka FIR)

    • Promises 2-4x build speed improvements • Preview coming in 1.7 https://blog.jetbrains.com/kotlin/2021/10/the-road-to-the-k2-compiler/
  76. Future Gradle Project Isolation • Conf i guration cache pt.

    2 • True project isolation, will allow parallel conf i guration and tooling model creation (aka Studio sync) • WIP. TL;DR don’t use allprojects/subprojects https://gradle.github.io/con fi guration-cache/#project_isolation systemProp.org.gradle.unsafe.isolated-projects=true
  77. How can I help? • Report bugs! • Open source

    your build stuff • Test/prepare for new features • Use modern APIs • Make your own convention plugin • File feature requests *
  78. What should I do today? • Audit deprecated/eager APIs •

    Update your tools (JDK, Gradle, Kotlin, etc) • Swap KSP for kapt where possible • Single - variant libraries • Disable android features by default • Fix java - only projects breaking IC • Dependency analysis to remove unused deps
  79. What should I do tomorrow? • Learn some Gradle •

    Reduce/remove Kapt • Create gradle - prof i ler scenarios • Modularization • Gradle Enterprise • Apple Silicon • Incorporate new tools prep in planning • OSS your stuff
  80. People to follow for more! • Tony Robalik – @AutonomousApps

    – https: / / dev.to/autonomousapps • Aurimas Liutikas – @_aurimas – https: / / w w w .liutikas.net/blog - posts • Iñaki Villar – @inyaki_mwc • Ryan Harter – @rharter – http: / / ryanharter.com/ • Xavier Ducrohet – @droidxav – https: / / developer.android.com/reference/tools/gradle - api • “Xav says read the docs and stop touching internal APIs” • Ivan Gavrilovic – @gavra0 • Cesar Puerta – @CesarDielo • Nelson Osacky – @nellyspageli – https: / / osacky.com/ • Jendrik Johannes – @jeoj – youtube.com/c/onepieceSoftware – github.com/jjohannes