Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What is the Android Cache Fix plugin and why do I need to solve my own cache misses.

What is the Android Cache Fix plugin and why do I need to solve my own cache misses.

This talk covers why cache misses happen, how to diagnose them, and why Gradle can't always automatically fix them for you. We'll also dive under the hood of the Android Cache Fix plugin to figure out some of the dirtiest cache misses in the Android ecosystem and why they still exist.

Nelson Osacky

October 29, 2021
Tweet

More Decks by Nelson Osacky

Other Decks in Technology

Transcript

  1. Android Cache Fix Plugin Nelson Osacky and why do I

    need to solve my own cache misses?
  2. Me • Previously Android Engineer • Large projects • SoundCloud

    • Square • Small startups • Gradle Plugin Maintainer • Fladle - Easily Scale Instrumentation Tests on Firebase https://github.com/runningcode/fladle • Gradle Doctor - Actionable Insights for your build https://github.com/runningcode/gradle-doctor Solutions Engineer
  3. I see a lot of builds from 10 person teams

    to 1000+ Android, Gradle and Maven
  4. Cost of Builds 60s waste * 50 builds / day

    * 50 devs 
 = 42 hours lost / day
  5. Cost of Builds 60s waste * 50 builds / day

    * 50 devs 
 = 42 hours lost / day not including lost focus https://gradle.com/roi-calculator
  6. Cost of Builds 60s waste * 50 builds / day

    * 50 devs 
 = 42 hours lost / day hire 5 new people without paying them! no recruiting https://gradle.com/roi-calculator
  7. Slow builds are Tech Debt And it always pays off

    And is easy to justify working on it
  8. "Broken" Cache • Cache miss -> Expected a hit but

    didn't • Cache hit when inputs changed • Missing outputs from cache • etc
  9. @AndroidIssue ( introducedIn = "4.0.0" , fixedIn = ['7.0.0-alpha09'] ,

    link = "https://issuetracker.google.com/issues/155218379" ) class CompileLibraryResourcesWorkaround implements Workaround { // the workaroun d }
  10. @AndroidIssue ( introducedIn = "4.0.0" , fixedIn = ['7.0.0-alpha09'] ,

    link = "https://issuetracker.google.com/issues/155218379" ) class CompileLibraryResourcesWorkaround implements Workaround { // the workaroun d }
  11. @AndroidIssue ( introducedIn = "4.0.0" , fixedIn = ['7.0.0-alpha09'] ,

    link = "https://issuetracker.google.com/issues/155218379" ) class CompileLibraryResourcesWorkaround implements Workaround { // the workaroun d }
  12. @AndroidIssue ( introducedIn = "4.0.0" , fixedIn = ['7.0.0-alpha09'] ,

    link = "https://issuetracker.google.com/issues/155218379" ) class CompileLibraryResourcesWorkaround implements Workaround { // the workaroun d }
  13. @AndroidIssue ( introducedIn = "4.0.0" , fixedIn = ['7.0.0-alpha09'] ,

    link = "https://issuetracker.google.com/issues/155218379" ) class CompileLibraryResourcesWorkaround implements Workaround { // the workaround }
  14. Con artists enticing people on Potsdamer Platz, Berlin, to play,

    and lose money in the game in 2018. https://en.wikipedia.org/wiki/Three-card_Monte
  15. 1.Add new workaround property with RELATIVE path sensitivity to existing

    task 2.Set workaround property with value of original property 3.Set original property to dummy (empty) value 4.Gradle performs input snapshotting on RELATIVE workaround property and dummy original property 5.After input snapshotting, set original property back its original value 6.Task executes on original value of original property Gradle Monte 🎩
  16. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty(

    ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) }
  17. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty(

    ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) }
  18. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty(

    ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { } }
  19. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty(

    ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { } }
  20. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty(

    ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { if (it == task) { originalPropertyValue.set(task.getProperty(propertyName) ) } } }
  21. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty(

    ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { if (it == task) { originalPropertyValue.set(task.getProperty(propertyName) ) } } }
  22. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty(

    ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { if (it == task) { originalPropertyValue.set(task.getProperty(propertyName) ) def dummyProperty = objects.directoryProperty( ) // Dummy file to give the DirectoryProperty a value . dummyProperty.set(project.file('/dummy') ) setPropertyValue(task, dummyProperty ) } } }
  23. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty(

    ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { if (it == task) { originalPropertyValue.set(task.getProperty(propertyName) ) def dummyProperty = objects.directoryProperty( ) // Dummy file to give the DirectoryProperty a value . dummyProperty.set(project.file('/dummy') ) setPropertyValue(task, dummyProperty ) } } }
  24. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty(

    ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { if (it == task) { originalPropertyValue.set(task.getProperty(propertyName) ) def dummyProperty = objects.directoryProperty( ) // Dummy file to give the DirectoryProperty a value . dummyProperty.set(project.file('/dummy') ) setPropertyValue(task, dummyProperty ) } } // Set the original task property back to its original valu e task.doFirst { setPropertyValue(task, originalPropertyValue ) } }
  25. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty(

    ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { if (it == task) { originalPropertyValue.set(task.getProperty(propertyName) ) def dummyProperty = objects.directoryProperty( ) // Dummy file to give the DirectoryProperty a value . dummyProperty.set(project.file('/dummy') ) setPropertyValue(task, dummyProperty ) } } // Set the original task property back to its original valu e task.doFirst { setPropertyValue(task, originalPropertyValue ) } }
  26. android { defaultConfig { javaCompileOptions { annotationProcessorOptions { arguments +=

    mapOf ( "room.schemaLocation" to "$projectDir/schemas", ) } } } }
  27. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory

    schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return [] // TODO } }
  28. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory

    schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return [] // TODO } }
  29. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory

    schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return [] // TODO } }
  30. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory

    schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["room.schemaLocation"] } }
  31. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory

    schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath"] } }
  32. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory

    schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath"] } }
  33. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @get:PathSensitive(PathSensitivity.RELATIVE) final Directory schemaLocationDi r

    RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath"] } }
  34. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @OutputDirector y @get:PathSensitive(PathSensitivity.RELATIVE) final Directory

    schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath"] } }
  35. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @OutputDirector y @get:PathSensitive(PathSensitivity.RELATIVE) final Directory

    schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath"] } }
  36. schemas/1.json // exists and is committed to version contro l

    schemas/2.json // written when schema is updated
  37. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @OutputDirector y @get:PathSensitive(PathSensitivity.RELATIVE) final Directory

    schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath"] } }
  38. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @InputDirectory @OutputDirector y @get:PathSensitive(PathSensitivity.RELATIVE) final

    Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath"] } }
  39. - In plugin 'kotlin-android' type 'org.jetbrains.kotlin.gradle.internal.KaptWithoutKotlincTask' property 'annotationProcessorOptionProviders$kotlin_gradle_plugin.$0.$0.schemaLocationDir' has conflicting

    type annotations declared: @OutputDirectory, @InputDirectory . Reason: The different annotations have different semantics and Gradle cannot determine which one to pick . Possible solution: Choose between one of the conflicting annotations. - In plugin 'kotlin-android' type 'org.jetbrains.kotlin.gradle.internal.KaptWithoutKotlincTask' property 'annotationProcessorOptionProviders$kotlin_gradle_plugin.$0.$0.schemaLocationDir' is annotated with @PathSensitive but that is not allowed for 'OutputDirectory' properties . Reason: This modifier is used in conjunction with a property of type 'OutputDirectory' but this doesn't have semantics . Possible solution: Remove the '@PathSensitive' annotation . Please refer to https://docs.gradle.org/7.2/userguide/ validation_problems.html#incompatible_annotations for more details about this problem.
  40. ?

  41. android { defaultConfig { javaCompileOptions { annotationProcessorOptions { arguments +=

    mapOf ( "room.schemaLocation" to "$projectDir/schemas", ) } } } } kaptDebug and kaptRelease write to the same folder
  42. •Absolute path sensitivity •Overlapping inputs and outputs •Non exclusive output

    directory •Kapt removes contents of output directories
  43. • Annotation Processors are not tools to write arbitrary fi

    les on the fi lesystem. • This is what Gradle tasks and Gradle plugins are for.
  44. • Absolute path sensitivity • Overlapping inputs and outputs •

    Non exclusive output directory • Kapt removes contents of output directories RoomSchemaLocationWorkaround fi xes all of these. Use the Android Cache Fix plugin. Shout out to Gary!
  45. What looks better in a promotion case? • Use new

    technology to rewrite Room to improve build speeds • Fix a 41 star bug filed by community with a Gradle plugin
  46. Gradle build cache is there a zip entry with the

    same build cache key? ~/.gradle/caches Gradle Home
  47. Project Directory zip unzip ~/.gradle/caches Build Cache Copy task Source

    Files Source Files Re-execution Cache store and fetch Happens along every execution
  48. https://github.com/gradle/android-cache-fix-gradle-plugin Android Cache Fix plugin plugins { id "org.gradle.android.cache-fix" version

    "2.4.4" apply false } subprojects { plugins.withType(com.android.build.gradle.api.AndroidBasePlugin) { project.apply plugin: "org.gradle.android.cache-fix" } }
  49. Gradle Enterprise Android Features • Find your slowest tests in

    the test dashboard • Learn which tests are flaky with flaky test insights • Android Instrumentation Test results in Build Scans! Check it out! https://gradle.com/enterprise-customers/oss-projects/