Save 37% off PRO during our Black Friday Sale! »

What is the Android Cache Fix plugin and why do I need to solve my own cache misses.

What is the Android Cache Fix plugin and why do I need to solve my own cache misses.

This talk covers why cache misses happen, how to diagnose them, and why Gradle can't always automatically fix them for you. We'll also dive under the hood of the Android Cache Fix plugin to figure out some of the dirtiest cache misses in the Android ecosystem and why they still exist.

Give at Droidcon Berlin October 20, 2021

5f69045a2ca496221cfc624405917cdf?s=128

Nelson Osacky

October 20, 2021
Tweet

Transcript

  1. Android Cache Fix Plugin Nelson Osacky and why do I

    need to solve my own cache misses?
  2. Me • Previously Android Engineer • Large projects • SoundCloud

    • Square • Small startups • Gradle Plugin Maintainer • Fladle - Easily Scale Instrumentation Tests on Firebase https://github.com/runningcode/fladle • Gradle Doctor - Actionable Insights for your build https://github.com/runningcode/gradle-doctor Solutions Engineer
  3. I see a lot of builds from 10 person teams

    to 1000+ Android, Gradle and Maven
  4. • Optimize and understand builds • Increase build stability •

    Preach Developer Productivity
  5. • Improve Gradle ecosystem • Fix issues in open source

    plugin affecting customers
  6. • Meet with Google and Jetbrains to help prioritize issues

    impacting customers
  7. None
  8. Android Development Feature Development Tech Debt (There are other things

    too)
  9. Tech Debt Refactoring Improving builds

  10. Slow builds are Tech Debt And it always pays off

  11. Cost of Builds 60s waste * 50 builds / day

    * 50 devs 
 = 42 hours lost / day
  12. Cost of Builds 60s waste * 50 builds / day

    * 50 devs 
 = 42 hours lost / day not including lost focus https://gradle.com/roi-calculator
  13. Fast Builds Matter

  14. Cost of Builds 60s waste * 50 builds / day

    * 50 devs 
 = 42 hours lost / day hire 5 new people without paying them! no recruiting https://gradle.com/roi-calculator
  15. Slow builds are Tech Debt And it always pays off

    And is easy to justify working on it
  16. None
  17. What is build caching? https://docs.gradle.org/current/userguide/build_cache.html

  18. JavaCompile task Source Files Compiler Args Dependencies Class files any

    task can be made cacheable
  19. Cache key Source Files Compiler Args Dependencies

  20. Cache key Source Files Compiler Args Dependencies Build cache Class

    files
  21. None
  22. What is the Android Cache Fix plugin?

  23. Solves cache misses/issues in the Android Gradle Plugin

  24. What is a cache miss? not "broken" cache

  25. "Broken" Cache • Cache miss -> Expected a hit but

    didn't • Cache hit when inputs changed • Missing outputs from cache • etc
  26. How to discover a cache miss?

  27. Run two identical builds and compare ./gradlew assembleDebug ./gradlew assembleDebug

  28. Task input comparison

  29. How does the Android Cache Fix plugin work?

  30. Task input comparison

  31. Path sensitivity Which part of the file path is considered

    a part of the key
  32. CompileLibraryResourcesTask Source Files Compiler Args Dependencies

  33. CompileLibraryResourcesTask build/intermediates/packaged_res/debug Compiler Args Dependencies

  34. CompileLibraryResourcesTask build/intermediates/packaged_res/debug

  35. CompileLibraryResourcesTask /Users/alice/build/intermediates/packaged_res/debug CompileLibraryResourcesTask /Users/jenkins/build/intermediates/packaged_res/debug

  36. CompileLibraryResourcesTask build/intermediates/packaged_res/debug

  37. @CacheableTas k abstract class CompileLibraryResourcesTask : NewIncrementalTask() {
 }

  38. @CacheableTas k abstract class CompileLibraryResourcesTask : NewIncrementalTask() { @get:InputFile s

    abstract val mergedLibraryResourcesDir: DirectoryProperty
 }
  39. @CacheableTas k abstract class CompileLibraryResourcesTask : NewIncrementalTask() { @get:PathSensitive(PathSensitivity.ABSOLUTE )

    @get:InputFile s abstract val mergedLibraryResourcesDir: DirectoryProperty
 }
  40. @CacheableTas k abstract class CompileLibraryResourcesTask : NewIncrementalTask() { @get:PathSensitive(PathSensitivity.RELATIVE )

    @get:InputFile s abstract val mergedLibraryResourcesDir: DirectoryProperty
 }
  41. How does the cache fix plugin workaround this? https://issuetracker.google.com/issues/155218379

  42. @AndroidIssue ( introducedIn = "4.0.0" , fixedIn = ['7.0.0-alpha09'] ,

    link = "https://issuetracker.google.com/issues/155218379" ) class CompileLibraryResourcesWorkaround implements Workaround { // the workaroun d }
  43. @AndroidIssue( introducedIn = "4.0.0" , fixedIn = ['7.0.0-alpha09'] , link

    = "https://issuetracker.google.com/issues/155218379" ) class CompileLibraryResourcesWorkaround implements Workaround { // the workaroun d }
  44. @AndroidIssue ( introducedIn = "4.0.0" , fixedIn = ['7.0.0-alpha09'] ,

    link = "https://issuetracker.google.com/issues/155218379" ) class CompileLibraryResourcesWorkaround implements Workaround { // the workaroun d }
  45. @AndroidIssue ( introducedIn = "4.0.0" , fixedIn = ['7.0.0-alpha09'] ,

    link = "https://issuetracker.google.com/issues/155218379" ) class CompileLibraryResourcesWorkaround implements Workaround { // the workaroun d }
  46. @AndroidIssue ( introducedIn = "4.0.0" , fixedIn = ['7.0.0-alpha09'] ,

    link = "https://issuetracker.google.com/issues/155218379" ) class CompileLibraryResourcesWorkaround implements Workaround { // the workaround }
  47. Three-card Monte https://www.youtube.com/watch?v=QJv44-Ghj_Y aka Find the Lady, das Kümmelblättchen, Jeu

    du Bonneteau, Gioco delle tre carte, Bul Karayı Al Parayı
  48. Con artists enticing people on Potsdamer Platz, Berlin, to play,

    and lose money in the game in 2018. https://en.wikipedia.org/wiki/Three-card_Monte
  49. None
  50. None
  51. Gradle Monte 🎩

  52. Gradle Monte 🎩 don't try this at home

  53. 1.Add new workaround property with RELATIVE path sensitivity to existing

    task 2.Set workaround property with value of original property 3.Set original property to dummy (empty) value 4.Gradle performs input snapshotting on RELATIVE workaround property and dummy original property 5.After input snapshotting, set original property back its original value 6.Task executes on original value of original property Gradle Monte 🎩
  54. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > }

  55. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty(

    ) }
  56. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty()

    // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) }
  57. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty(

    ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { if (it == task) { originalPropertyValue.set(task.getProperty(propertyName) ) } } }
  58. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty(

    ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { if (it == task) { originalPropertyValue.set(task.getProperty(propertyName) ) def dummyProperty = objects.directoryProperty( ) // Dummy file to give the DirectoryProperty a value . dummyProperty.set(project.file('/doesnt-exist') ) setPropertyValue(task, dummyProperty ) } } }
  59. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty(

    ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { if (it == task) { originalPropertyValue.set(task.getProperty(propertyName) ) def dummyProperty = objects.directoryProperty( ) // Non-existent file to give the DirectoryProperty a value . dummyProperty.set(project.file('/doesnt-exist') ) setPropertyValue(task, dummyProperty ) } } // Set the original task property back to its original valu e task.doFirst { setPropertyValue(task, originalPropertyValue ) } }
  60. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty(

    ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { if (it == task) { originalPropertyValue.set(task.getProperty(propertyName) ) def dummyProperty = objects.directoryProperty( ) // Non-existent file to give the DirectoryProperty a value . dummyProperty.set(project.file('/doesnt-exist') ) setPropertyValue(task, dummyProperty ) } } // Set the original task property back to its original valu e task.doFirst { setPropertyValue(task, originalPropertyValue ) } }
  61. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty(

    ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { if (it == task) { originalPropertyValue.set(task.getProperty(propertyName) ) def dummyProperty = objects.directoryProperty( ) // Non-existent file to give the DirectoryProperty a value . dummyProperty.set(project.file('/doesnt-exist') ) setPropertyValue(task, dummyProperty ) } } // Set the original task property back to its original valu e task.doFirst { setPropertyValue(task, originalPropertyValue ) } }
  62. Please don't ever write that code

  63. CompileLibraryResourcesTask is fixed in AGP 7

  64. CompileLibraryResourcesTask is fixed in AGP 7*

  65. * must enable android.experimental.enableSourceSetPathsMap and android.experimental.cacheCompileLibResources

  66. Room cache miss

  67. Room is an annotation processor

  68. implementation("androidx.room:room-runtime:2.3.0" ) kapt("androidx.room:room-compiler:2.3.0")

  69. How many use Room?

  70. None
  71. None
  72. android { defaultConfig { javaCompileOptions { annotationProcessorOptions { arguments +=

    mapOf ( "room.schemaLocation" to "$projectDir/schemas", ) } } } }
  73. annotationProcessorOptions { arguments += mapOf ( "room.schemaLocation" to "$projectDir/schemas", )

    }
  74. annotationProcessorOptions { arguments += mapOf ( "room.schemaLocation" to "$projectDir/schemas", )

    }
  75. "$projectDir/schemas"

  76. None
  77. "$projectDir/schemas"

  78. "User/no/home/workspace/myapp/schemas"

  79. "User/no/home/workspace/myapp/schemas " "User/bob/home/workspace/myapp/schemas"

  80. "User/no/home/workspace/myapp/schemas " "User/bob/home/workspace/myapp/schemas " "User/jenkins/home/myapp/schemas"

  81. "$projectDir/schemas"

  82. None
  83. CommandLineArgumentProvider

  84. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { }

  85. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @Overrid e Iterable<String> asArguments() {

    return [] // TODO } }
  86. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { } @Overrid

    e Iterable<String> asArguments() { return [] // TOD O } }
  87. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory

    schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return [] // TOD O } }
  88. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory

    schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["room.schemaLocation"] } }
  89. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory

    schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath"] } }
  90. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @get:PathSensitive(PathSensitivity.RELATIVE) final Directory schemaLocationDi r

    RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath" ] } b }a
  91. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @OutputDirector y @get:PathSensitive(PathSensitivity.RELATIVE) final Directory

    schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath" ] } b }a
  92. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @OutputDirector y @get:PathSensitive(PathSensitivity.RELATIVE) final Directory

    schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath"] } }
  93. How to use it?

  94. annotationProcessorOptions { arguments += mapOf ( "room.schemaLocation" to "$projectDir/schemas", )

    }
  95. annotationProcessorOptions { compilerArgumentProviders.add(RoomSchemaLocationArgumentProvider("$projectDir/schemas") ) }

  96. And now you get cache hits

  97. And now you get cache hits?

  98. yes, but

  99. None
  100. None
  101. schemas/1.json // exists and is committed to version contro l

    schemas/2.json // written when schema is updated
  102. schemas/1.json // inpu t schemas/2.json // output

  103. schemas // input and output

  104. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @OutputDirector y @get:PathSensitive(PathSensitivity.RELATIVE) final Directory

    schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath"] } }
  105. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @InputDirectory @OutputDirector y @get:PathSensitive(PathSensitivity.RELATIVE) final

    Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath"] } }
  106. - In plugin 'kotlin-android' type 'org.jetbrains.kotlin.gradle.internal.KaptWithoutKotlincTask' property 'annotationProcessorOptionProviders$kotlin_gradle_plugin.$0.$0.schemaLocationDir' has conflicting

    type annotations declared: @OutputDirectory, @InputDirectory . Reason: The different annotations have different semantics and Gradle cannot determine which one to pick . Possible solution: Choose between one of the conflicting annotations. - In plugin 'kotlin-android' type 'org.jetbrains.kotlin.gradle.internal.KaptWithoutKotlincTask' property 'annotationProcessorOptionProviders$kotlin_gradle_plugin.$0.$0.schemaLocationDir' is annotated with @PathSensitive but that is not allowed for 'OutputDirectory' properties . Reason: This modifier is used in conjunction with a property of type 'OutputDirectory' but this doesn't have semantics . Possible solution: Remove the '@PathSensitive' annotation . Please refer to https://docs.gradle.org/7.2/userguide/ validation_problems.html#incompatible_annotations for more details about this problem.
  107. Input and output cannot be overlapping

  108. ?

  109. None
  110. None
  111. Task specific output directory

  112. android { defaultConfig { javaCompileOptions { annotationProcessorOptions { arguments +=

    mapOf ( "room.schemaLocation" to "$projectDir/schemas", ) } } } } kaptDebug and kaptRelease write to the same folder
  113. None
  114. None
  115. •Absolute path sensitivity •Overlapping inputs and outputs •Non exclusive output

    directory •Kapt removes contents of output directories
  116. None
  117. • Annotation Processors are not tools to write arbitrary fi

    les on the fi lesystem. • This is what Gradle tasks and Gradle plugins are for.
  118. This needs to be a Gradle plugin

  119. This really needs to be a Gradle plugin

  120. Until then, the Android Cache Fix plugin will exist forever.

    😭
  121. • Absolute path sensitivity • Overlapping inputs and outputs •

    Non exclusive output directory • Kapt removes contents of output directories RoomSchemaLocationWorkaround fi xes all of these. Use the Android Cache Fix plugin. Shout out to Gary!
  122. https://issuetracker.google.com/issues/132245929 https://issuetracker.google.com/issues/139438151 What is Google doing about this?

  123. Room now has KSP support because performance matters. This doesn't

    solve the cache miss 🥲
  124. What looks better in a promotion case? • Use new

    technology to rewrite Room to improve build speeds • Fix a 41 star bug filed by community with a Gradle plugin
  125. https://issuetracker.google.com/issues/132245929 https://issuetracker.google.com/issues/139438151 What is Google doing about this? please star

    ⭐ #fixroom
  126. None
  127. What else does the Android Cache Fix plugin do?

  128. None
  129. None
  130. What is caching?

  131. JavaCompile task Source Files Compiler Args Dependencies Class files

  132. Copy task Source Files Source Files

  133. Project Directory Up-to-date checking is the file already there?

  134. Gradle build cache is there a zip entry with the

    same build cache key? ~/.gradle/caches Gradle Home
  135. Copy task Source Files Source Files

  136. Cache key Source Files Build cache Source Files Cache Retrieval

  137. Project Directory zip unzip ~/.gradle/caches Build Cache Copy task Source

    Files Source Files Re-execution Cache store and fetch Happens along every execution
  138. Sync, Copy, Zip and Jar should not be cached https://docs.gradle.org/current/userguide/build_cache_concepts.html#non_cacheable_tasks

  139. None
  140. BundleLibraryClassesJar is a Jar task

  141. Disabling caching

  142. outputs.doNotCacheIf {btrueb}b

  143. tasks.withType(BundleLibraryClassesJar).configureEacha{a outputs.doNotCacheIf ("I/O bound tasks do not benefit from caching")

    {btrueb}b }
  144. None
  145. Thanks to the Android Gradle plugin team at Google

  146. https://github.com/gradle/android-cache-fix-gradle-plugin Android Cache Fix plugin plugins { id "org.gradle.android.cache-fix" version

    "2.4.4" apply false } subprojects { plugins.withType(com.android.build.gradle.api.AndroidBasePlugin) { project.apply plugin: "org.gradle.android.cache-fix" } }
  147. Gradle Enterprise Android Features • Find your slowest tests in

    the test dashboard • Learn which tests are flaky with flaky test insights • Android Instrumentation Test results in Build Scans! Check it out! https://gradle.com/enterprise-customers/oss-projects/
  148. https://ge.gradle.org/scans/tests?search.relativeStartTime=P7D&search.timeZoneId=Europe/Berlin&tests.sortField=MEAN_DURATION&tests.unstableOnly=true https://bit.ly/3DUBTu0

  149. None
  150. Tweet, Like, Subscribe, Upvote, Star https://github.com/gradle/android-cache-fix-gradle-plugin https://issuetracker.google.com/issues/132245929 https://issuetracker.google.com/issues/139438151 please star

    ⭐ #fixroom
  151. More Resources Gradle Training https://gradle.com/training/ https://gradle.com/training/introduction-to-gradle https://gradle.com/training/build-cache-deep-dive Gradle Community Slack

    https://gradle.com/slack-invite
  152. Thank you! Gradle Enterprise Trial osacky.com https://gradle.com/enterprise/trial/ https://go.gradle.com/solutions-engineer Gradle Solutions

    is hiring #fixroom @nellyspageli Find us at our booth here at Droidcon