Remote Build Cache Misses and how to solve them by annoying your colleagues.

Remote Build Cache Misses and how to solve them by annoying your colleagues.

This is a talk about remote build cache misses and how we solved them at SoundCloud.
This was presented and Berlin Gradle Night on April 8, 2019.
Here is a recording of the presentation:
https://youtu.be/2frfDMJwvf4?t=384

5f69045a2ca496221cfc624405917cdf?s=128

Nelson Osacky

April 08, 2019
Tweet

Transcript

  1. None
  2. Gradle Remote Cache Misses and how to solve them by

    annoying your colleagues Nelson Osacky
  3. None
  4. Gradle Task

  5. A Task represents a single atomic piece of work for

    a build.
  6. • Copying files • Compiling Classes • Running Tests

  7. A task has inputs and ouputs.

  8. JavaCompile • Inputs = source files • Outputs = class

    files
  9. Gradle Task Work Input Input Output Output

  10. JavaCompile Task Compile Input1=Audio.java Input2=Music.java Output=Audio.class Output=Music.class

  11. Build Cache

  12. Build Cache • Mechanism to reuse outputs from other builds

    • Allows builds to refetch outputs when it is determined that the inputs haven’t changed • Can be local (filesystem) or remote (http)
  13. A build cache key uniquely defines the task’s outputs based

    on its inputs.
  14. • The task type and its classpath • The names

    of the output properties • The names and values of properties annotated as described in the section called "Custom task types" • The names and values of properties added by the DSL via TaskInputs • The classpath of the Gradle distribution, buildSrc and plugins • The content of the build script when it affects execution of the task
  15. LruCache<BuildCacheKey, BuildOutput>()

  16. Local Cache If you have built it before, skip building

    it again. Remote Cache If anyone else has built it before, skip building it again.
  17. Local Push Pull Pull Remote Cache CI

  18. Local CI In order to re-use task outputs we need

    to have the same build cache key across environments.
  19. Seed build cache task on CI. ./gradlew assembleDevDebug

  20. None
  21. Build Scans

  22. None
  23. Gradle Enterprise

  24. Gradle Enterprise • Aggregates Build Scans • Task Input Comparison

  25. Identify what changed between two executions of a task that

    prevented the output from being reused from a build cache.
  26. Local CI

  27. Gradle treats whitespace as build cache miss.

  28. None
  29. public final class BuildConfig { public static final boolean DEBUG

    = Boolean.parseBoolean("true"); public static final String APPLICATION_ID = "com.soundcloud.android"; public static final String BUILD_TYPE = "debug"; public static final int VERSION_CODE = -1; public static final int TEST_RETRY_COUNT = 0; }
  30. None
  31. public final class BuildConfig { public static final boolean DEBUG

    = Boolean.parseBoolean("true"); public static final String APPLICATION_ID = "com.soundcloud.android"; public static final String BUILD_TYPE = "debug"; public static final int VERSION_CODE = -1; public static final int TEST_RETRY_COUNT = 0; }
  32. afterEvaluate { android.libraryVariants.all { variant -> variant.resValue "bool", "analytics_enabled", "true"

    variant.resValue "bool", "verbose_logging", "false" variant.buildConfigField "int", "TEST_RETRY_COUNT", "${getRetryCount()}" } }
  33. private static int getRetryCount() { if (isCI) { return 1

    } else { return 0 } }
  34. Retry mechanism against flaky tests.

  35. public final class BuildConfig { public static final boolean DEBUG

    = Boolean.parseBoolean("true"); public static final String APPLICATION_ID = "com.soundcloud.android"; public static final String BUILD_TYPE = "debug"; public static final int VERSION_CODE = -1; public static final int TEST_RETRY_COUNT = 0; }
  36. public final class BuildConfig { public static final boolean DEBUG

    = Boolean.parseBoolean("true"); public static final String APPLICATION_ID = "com.soundcloud.android"; public static final String BUILD_TYPE = "debug"; public static final int VERSION_CODE = -1; public static final int TEST_RETRY_COUNT = 1; }
  37. afterEvaluate { android.libraryVariants.all { variant -> variant.resValue "bool", "analytics_enabled", "true"

    variant.resValue "bool", "verbose_logging", "false" variant.buildConfigField "int", "TEST_RETRY_COUNT", "${getRetryCount()}" } }
  38. afterEvaluate { android.libraryVariants.all { variant -> variant.resValue "bool", "analytics_enabled", "true"

    variant.resValue "bool", "verbose_logging", "false" } }
  39. public final class BuildConfig { public static final boolean DEBUG

    = Boolean.parseBoolean("true"); public static final String APPLICATION_ID = "com.soundcloud.android"; public static final String BUILD_TYPE = "debug"; public static final int VERSION_CODE = -1; public static final int TEST_RETRY_COUNT = 1; }
  40. public final class BuildConfig { public static final boolean DEBUG

    = Boolean.parseBoolean("true"); public static final String APPLICATION_ID = "com.soundcloud.android"; public static final String BUILD_TYPE = "debug"; public static final int VERSION_CODE = -1; }
  41. open class ActivityTest<T : Activity> protected constructor(activityClass: Class<T>) { private

    val isRunningOnTestLab = Settings.System.getString(contentResolver, "firebase.test.lab") == "true" @JvmField val retryRule = RetryRule(if (isRunningOnTestLab) 1 else 0) }
  42. None
  43. Local src/main/java/com/soundcloud/Player.java src/main/java/com/soundcloud/Artwork.java src/main/java/com/soundcloud/audio/ CI src/main/java/com/soundcloud/Player.java src/main/java/com/soundcloud/Artwork.java

  44. Empty directories are considered part of a task’s input properties

  45. tasks.withType(SourceTask).configureEach {at -> t.doFirst { t.source.visit { FileVisitDetails d ->

    if (d.file.directory && d.file.listFiles().size() == 0) { throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) } } } }
  46. tasks.withType(SourceTask).configureEach {at ->

  47. tasks.withType(SourceTask).configureEach {at -> t.doFirst {a t.source.visit {aFileVisitDetails d ->a if

    (d.file.directory && d.file.listFiles().size() == 0) { throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) } } } }
  48. t.doFirst {a t.source.visit {aFileVisitDetails d ->a

  49. tasks.withType(SourceTask).configureEach { t -> t.doFirst {a t.source.visit {aFileVisitDetails d ->a

    if (d.file.directory && d.file.listFiles().size() == 0) {b throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) }c }d } }
  50. t.source.visit {aFileVisitDetails d ->a if (d.file.directory && d.file.listFiles().size() == 0)

    {b throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) }c }d
  51. if (d.file.directory && d.file.listFiles().size() == 0) {b throw new IllegalStateException(

    "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) }c
  52. tasks.withType(SourceTask).configureEach { t -> t.doFirst { t.source.visit { FileVisitDetails d

    -> if (d.file.directory && d.file.listFiles().size() == 0) { throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) }c }d } }
  53. None
  54. None
  55. None
  56. buildSrc

  57. A special module where code added to buildscript classpath instead

    of runtime classpath.
  58. !"" build.gradle !"" buildSrc # !"" build.gradle # $"" src

    # !"" main # $"" java # $"" com # $"" enterprise # !"" Deploy.java # $"" DeploymentPlugin.java $"" settings.gradle
  59. None
  60. None
  61. • All classes that are visible to the task class

    are considered part of the implementation. • Usually a transient cache miss when adding or removing a plugin • Also a transient miss happens when modifying buildSrc
  62. Our buildSrc was removed. Or was it?

  63. None
  64. None
  65. if (file("buildSrc").exists()) { throw new IllegalStateException("Please run the following command

    in order to solve your build cache issues: rm -r buildSrc.") }
  66. InAppBillingService.aidl

  67. android.sourceSets.main { aidl.srcDirs java.srcDirs } build.gradle

  68. None
  69. None
  70. None
  71. tasks.named("compileDebugAidl").configure {a doLast {b outputs.files.forEach { directory -> directory.traverse(type: FILES)

    { file -> file.setText((file as String[]).findAll { !it.contains('Original file:') }.join(‚\n'), 'utf-8') } } } }
  72. tasks.named("compileDebugAidl").configure {a doLast {b

  73. tasks.named("compileDebugAidl").configure {a doLast {b outputs.files.forEach {adirectory -> directory.traverse(type: FILES) {bfile

    -> file.setText((file as String[]).findAll { !it.contains('Original file:') }.join(‚\n'), 'utf-8') } } } }
  74. outputs.files.forEach {adirectory -> directory.traverse(type: FILES) {bfile ->

  75. tasks.named("compileDebugAidl").configure {a doLast {b outputs.files.forEach {adirectory -> directory.traverse(type: FILES) {bfile

    -> file.setText((file as String[]).findAll {d !it.contains('Original file:') }.join(‚\n'), 'utf-8') } } } }
  76. file.setText((file as String[]).findAll {d !it.contains('Original file:') }.join(‚\n'), 'utf-8')

  77. tasks.named("compileDebugAidl").configure { doLast { outputs.files.forEach { directory -> directory.traverse(type: FILES)

    { file -> file.setText((file as String[]).findAll {d !it.contains('Original file:') }.join(‚\n'), 'utf-8') } } } }
  78. Fixed future release of AGP https://issuetracker.google.com/issues/121251997

  79. None
  80. android.sourceSets.main { aidl.srcDirs java.srcDirs }

  81. https://github.com/gradle/gradle/issues/8559

  82. Third Party Plugins

  83. None
  84. • PathSensitivity.ABSOLUTE • Consider the full paths of files and

    directories. • PathSensitivity.RELATIVE • Use the location of the file relative to a hierarchy.
  85. • If a Task declares a file property without PathSensivity

    annotation, the default is PathSensivity.ABSOLUTE.
  86. None
  87. None
  88. protobuf-gradle- plugin

  89. None
  90. Results

  91. 10 Dec - 14 Dec

  92. 1 Apr - 5 Apr

  93. 25% faster builds on avg (49.55s - 36.84s) / 49.55s

    = 1000+ builds / week ~12s / build
  94. Questions?