Remote Build Cache Misses and how to solve them by annoying your colleagues.

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Gradle Remote Cache Misses and how to solve them by annoying your colleagues Nelson Osacky

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Gradle Task

Slide 5

Slide 5 text

A Task represents a single atomic piece of work for a build.

Slide 6

Slide 6 text

• Copying ﬁles • Compiling Classes • Running Tests

Slide 7

Slide 7 text

A task has inputs and ouputs.

Slide 8

Slide 8 text

JavaCompile • Inputs = source ﬁles • Outputs = class ﬁles

Slide 9

Slide 9 text

Gradle Task Work Input Input Output Output

Slide 10

Slide 10 text

JavaCompile Task Compile Input1=Audio.java Input2=Music.java Output=Audio.class Output=Music.class

Slide 11

Slide 11 text

Build Cache

Slide 12

Slide 12 text

Build Cache • Mechanism to reuse outputs from other builds • Allows builds to refetch outputs when it is determined that the inputs haven’t changed • Can be local (ﬁlesystem) or remote (http)

Slide 13

Slide 13 text

A build cache key uniquely deﬁnes the task’s outputs based on its inputs.

Slide 14

Slide 14 text

• The task type and its classpath • The names of the output properties • The names and values of properties annotated as described in the section called "Custom task types" • The names and values of properties added by the DSL via TaskInputs • The classpath of the Gradle distribution, buildSrc and plugins • The content of the build script when it aﬀects execution of the task

Slide 15

Slide 15 text

LruCache()

Slide 16

Slide 16 text

Local Cache If you have built it before, skip building it again. Remote Cache If anyone else has built it before, skip building it again.

Slide 17

Slide 17 text

Local Push Pull Pull Remote Cache CI

Slide 18

Slide 18 text

Local CI In order to re-use task outputs we need to have the same build cache key across environments.

Slide 19

Slide 19 text

Seed build cache task on CI. ./gradlew assembleDevDebug

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

Build Scans

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

Gradle Enterprise

Slide 24

Slide 24 text

Gradle Enterprise • Aggregates Build Scans • Task Input Comparison

Slide 25

Slide 25 text

Identify what changed between two executions of a task that prevented the output from being reused from a build cache.

Slide 26

Slide 26 text

Local CI

Slide 27

Slide 27 text

Gradle treats whitespace as build cache miss.

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

public final class BuildConfig { public static final boolean DEBUG = Boolean.parseBoolean("true"); public static final String APPLICATION_ID = "com.soundcloud.android"; public static final String BUILD_TYPE = "debug"; public static final int VERSION_CODE = -1; public static final int TEST_RETRY_COUNT = 0; }

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

Slide 32

Slide 32 text

afterEvaluate { android.libraryVariants.all { variant -> variant.resValue "bool", "analytics_enabled", "true" variant.resValue "bool", "verbose_logging", "false" variant.buildConﬁgField "int", "TEST_RETRY_COUNT", "${getRetryCount()}" } }

Slide 33

Slide 33 text

private static int getRetryCount() { if (isCI) { return 1 } else { return 0 } }

Slide 34

Slide 34 text

Retry mechanism against ﬂaky tests.

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

afterEvaluate { android.libraryVariants.all { variant -> variant.resValue "bool", "analytics_enabled", "true" variant.resValue "bool", "verbose_logging", "false" } }

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Slide 41

Slide 41 text

open class ActivityTest protected constructor(activityClass: Class) { private val isRunningOnTestLab = Settings.System.getString(contentResolver, "firebase.test.lab") == "true" @JvmField val retryRule = RetryRule(if (isRunningOnTestLab) 1 else 0) }

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

Local src/main/java/com/soundcloud/Player.java src/main/java/com/soundcloud/Artwork.java src/main/java/com/soundcloud/audio/ CI src/main/java/com/soundcloud/Player.java src/main/java/com/soundcloud/Artwork.java

Slide 44

Slide 44 text

Empty directories are considered part of a task’s input properties

Slide 45

Slide 45 text

tasks.withType(SourceTask).configureEach {at -> t.doFirst { t.source.visit { FileVisitDetails d -> if (d.file.directory && d.file.listFiles().size() == 0) { throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) } } } }

Slide 46

Slide 46 text

tasks.withType(SourceTask).configureEach {at ->

Slide 47

Slide 47 text

tasks.withType(SourceTask).configureEach {at -> t.doFirst {a t.source.visit {aFileVisitDetails d ->a if (d.file.directory && d.file.listFiles().size() == 0) { throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) } } } }

Slide 48

Slide 48 text

t.doFirst {a t.source.visit {aFileVisitDetails d ->a

Slide 49

Slide 49 text

tasks.withType(SourceTask).configureEach { t -> t.doFirst {a t.source.visit {aFileVisitDetails d ->a if (d.file.directory && d.file.listFiles().size() == 0) {b throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) }c }d } }

Slide 50

Slide 50 text

t.source.visit {aFileVisitDetails d ->a if (d.file.directory && d.file.listFiles().size() == 0) {b throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) }c }d

Slide 51

Slide 51 text

if (d.file.directory && d.file.listFiles().size() == 0) {b throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) }c

Slide 52

Slide 52 text

tasks.withType(SourceTask).configureEach { t -> t.doFirst { t.source.visit { FileVisitDetails d -> if (d.file.directory && d.file.listFiles().size() == 0) { throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) }c }d } }

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

buildSrc

Slide 57

Slide 57 text

A special module where code added to buildscript classpath instead of runtime classpath.

Slide 58

Slide 58 text

!"" build.gradle !"" buildSrc # !"" build.gradle # $"" src # !"" main # $"" java # $"" com # $"" enterprise # !"" Deploy.java # $"" DeploymentPlugin.java $"" settings.gradle

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

• All classes that are visible to the task class are considered part of the implementation. • Usually a transient cache miss when adding or removing a plugin • Also a transient miss happens when modifying buildSrc

Slide 62

Slide 62 text

Our buildSrc was removed. Or was it?

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

No content

Slide 65

Slide 65 text

if (file("buildSrc").exists()) { throw new IllegalStateException("Please run the following command in order to solve your build cache issues: rm -r buildSrc.") }

Slide 66

Slide 66 text

InAppBillingService.aidl

Slide 67

Slide 67 text

android.sourceSets.main { aidl.srcDirs java.srcDirs } build.gradle

Slide 68

Slide 68 text

No content

Slide 69

Slide 69 text

No content

Slide 70

Slide 70 text

No content

Slide 71

Slide 71 text

tasks.named("compileDebugAidl").configure {a doLast {b outputs.files.forEach { directory -> directory.traverse(type: FILES) { file -> file.setText((file as String[]).findAll { !it.contains('Original file:') }.join(‚\n'), 'utf-8') } } } }

Slide 72

Slide 72 text

tasks.named("compileDebugAidl").configure {a doLast {b

Slide 73

Slide 73 text

tasks.named("compileDebugAidl").configure {a doLast {b outputs.files.forEach {adirectory -> directory.traverse(type: FILES) {bfile -> file.setText((file as String[]).findAll { !it.contains('Original file:') }.join(‚\n'), 'utf-8') } } } }

Slide 74

Slide 74 text

outputs.files.forEach {adirectory -> directory.traverse(type: FILES) {bfile ->

Slide 75

Slide 75 text

tasks.named("compileDebugAidl").configure {a doLast {b outputs.files.forEach {adirectory -> directory.traverse(type: FILES) {bfile -> file.setText((file as String[]).findAll {d !it.contains('Original file:') }.join(‚\n'), 'utf-8') } } } }

Slide 76

Slide 76 text

file.setText((file as String[]).findAll {d !it.contains('Original file:') }.join(‚\n'), 'utf-8')

Slide 77

Slide 77 text

tasks.named("compileDebugAidl").configure { doLast { outputs.files.forEach { directory -> directory.traverse(type: FILES) { file -> file.setText((file as String[]).findAll {d !it.contains('Original file:') }.join(‚\n'), 'utf-8') } } } }

Slide 78

Slide 78 text

Fixed future release of AGP https://issuetracker.google.com/issues/121251997

Slide 79

Slide 79 text

No content

Slide 80

Slide 80 text

android.sourceSets.main { aidl.srcDirs java.srcDirs }

Slide 81

Slide 81 text

https://github.com/gradle/gradle/issues/8559

Slide 82

Slide 82 text

Third Party Plugins

Slide 83

Slide 83 text

No content

Slide 84

Slide 84 text

• PathSensitivity.ABSOLUTE • Consider the full paths of ﬁles and directories. • PathSensitivity.RELATIVE • Use the location of the ﬁle relative to a hierarchy.

Slide 85

Slide 85 text

• If a Task declares a ﬁle property without PathSensivity annotation, the default is PathSensivity.ABSOLUTE.

Slide 86

Slide 86 text

No content

Slide 87

Slide 87 text

No content

Slide 88

Slide 88 text

protobuf-gradle- plugin

Slide 89

Slide 89 text

No content

Slide 90

Slide 90 text

Results

Slide 91

Slide 91 text

10 Dec - 14 Dec

Slide 92

Slide 92 text

1 Apr - 5 Apr

Slide 93

Slide 93 text

25% faster builds on avg (49.55s - 36.84s) / 49.55s = 1000+ builds / week ~12s / build

Slide 94

Slide 94 text

Questions?