Tweet
Tweet

Slide 1

Slide 1 text

From Laptop Builds to Advanced CI Oct 27th, 2023 Jason Pearsonโ€จ Senior Staff Engineer Formerly Hinge

Slide 2

Slide 2 text

First, about me Jason Pearson

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

CI experience since 2011. Built Android at Hinge, went from solo dev to 24 Android engineers. Also spent years in backend and fullstack roles across various companies.

Slide 8

Slide 8 text

Agenda Laptop Builds ๐Ÿ’ป Automate All The Things Make it Faster

Slide 9

Slide 9 text

Make it Faster 12 9 6 3 0 Unit Tests Release R8 UI Tests Unit Tests Release R8 UI tests minutes

Slide 10

Slide 10 text

~400 Modules ๐Ÿ’ป 290k lines of Kotlin 24 active contributors Context Dagger Hilt (KAPT) ๐Ÿ—ก Gradle Kotlin Scripts KSP (except Dagger)

Slide 11

Slide 11 text

Problems ๐Ÿคฏ Distracting from Feature Work โŒ Regressions ๐Ÿดโ˜  Security Concerns

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

Our Story

Slide 14

Slide 14 text

How every Android project starts ๐Ÿ’ป

Slide 15

Slide 15 text

How every Android project starts ๐Ÿ’ป

Slide 16

Slide 16 text

How every Android project starts ๐Ÿ’ป

Slide 17

Slide 17 text

๐Ÿ’ป

Slide 18

Slide 18 text

๐Ÿ’ป

Slide 19

Slide 19 text

๐Ÿ’ป

Slide 20

Slide 20 text

๐Ÿ’ป

Slide 21

Slide 21 text

๐Ÿ’ป

Slide 22

Slide 22 text

๐Ÿ’ป

Slide 23

Slide 23 text

๐Ÿ’ป

Slide 24

Slide 24 text

๐Ÿ’ป

Slide 25

Slide 25 text

๐Ÿ’ป

Slide 26

Slide 26 text

๐Ÿ’ป

Slide 27

Slide 27 text

๐Ÿ’ป Problems ๐Ÿคฏ Distracting from Feature Work

Slide 28

Slide 28 text

๐Ÿ’ป Problems ๐Ÿคฏ Distracting from Feature Work

Slide 29

Slide 29 text

๐Ÿ’ป Problems ๐Ÿคฏ Distracting from Feature Work

Slide 30

Slide 30 text

๐Ÿ’ป Problems

Slide 31

Slide 31 text

๐Ÿ’ป

Slide 32

Slide 32 text

๐Ÿ’ป ๐Ÿ’ป

Slide 33

Slide 33 text

๐Ÿ’ป ๐Ÿ’ป ๐Ÿ’ป ๐Ÿ’ป

Slide 34

Slide 34 text

Problems ๐Ÿคฏ Distracting from Feature Work

Slide 35

Slide 35 text

Problems ๐Ÿคฏ Distracting from Feature Work โŒ Regressions

Slide 36

Slide 36 text

Problems ๐Ÿคฏ Distracting from Feature Work โŒ Regressions ๐Ÿดโ˜  Security Concerns

Slide 37

Slide 37 text

Problems ๐Ÿคฏ Distracting from Feature Work โŒ Regressions ๐Ÿดโ˜  Security Concerns CI Solutions ๐Ÿ— Automate Building & Publishing ๐Ÿ‘ฉ๐Ÿ”ฌ Automate Tests ๐Ÿ” Automate Security

Slide 38

Slide 38 text

When do we build? What do we run? Self hosted or Managed? What should CI look like?

Slide 39

Slide 39 text

What should CI look like? This talk will focus on builds on commit push

Slide 40

Slide 40 text

Automating Security

Slide 41

Slide 41 text

Handle secrets appropriately Automating Security

Slide 42

Slide 42 text

Handle secrets appropriately Automating Security ๐Ÿ”

Slide 43

Slide 43 text

Handle secrets appropriately Automating Security ๐Ÿ”

Slide 44

Slide 44 text

Handle secrets appropriately No personal user tokens Automating Security ๐Ÿ”

Slide 45

Slide 45 text

Testing on CI ๐Ÿ‘ฉ๐Ÿ”ฌ

Slide 46

Slide 46 text

Testing on CI ๐Ÿ‘ฉ๐Ÿ”ฌ

Slide 47

Slide 47 text

config.yml ๐Ÿšง

Slide 48

Slide 48 text

version: 2.1 jobs: unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout config.yml ๐Ÿšง

Slide 49

Slide 49 text

version: 2.1 jobs: unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest config.yml ๐Ÿšง

Slide 50

Slide 50 text

๐Ÿšง

Slide 51

Slide 51 text

* What went wrong: Execution failed for task ':features:user-account:fresh- start:testInternalDebugUnitTest'. > There were failing tests. See the report at: file:///mnt/ramdisk/ features/user-account/fresh-start/build/reports/tests/ testInternalDebugUnitTest/index.html * Exception is: org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':features:user-account:fresh-start:testInternalDebugUnitTest'. at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$e xecuteIfValid$1(ExecuteActionsTaskExecuter.java:148) at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeI fValid(ExecuteActionsTaskExecuter.java:146) at ๐Ÿšง

Slide 52

Slide 52 text

* What went wrong: Execution failed for task ':features:user-account:fresh- start:testInternalDebugUnitTest'. > There were failing tests. See the report at: file:///mnt/ramdisk/ features/user-account/fresh-start/build/reports/tests/ testInternalDebugUnitTest/index.html * Exception is: org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':features:user-account:fresh-start:testInternalDebugUnitTest'. at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$e xecuteIfValid$1(ExecuteActionsTaskExecuter.java:148) at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeI fValid(ExecuteActionsTaskExecuter.java:146) at ๐Ÿšง

Slide 53

Slide 53 text

* What went wrong: Execution failed for task ':features:user-account:fresh- start:testInternalDebugUnitTest'. > There were failing tests. See the report at: file:///mnt/ramdisk/ features/user-account/fresh-start/build/reports/tests/ testInternalDebugUnitTest/index.html * Exception is: org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':features:user-account:fresh-start:testInternalDebugUnitTest'. at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$e xecuteIfValid$1(ExecuteActionsTaskExecuter.java:148) at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeI fValid(ExecuteActionsTaskExecuter.java:146) at ๐Ÿšง

Slide 54

Slide 54 text

version: 2.1 jobs: unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest config.yml ๐Ÿšง

Slide 55

Slide 55 text

unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest config.yml ๐Ÿšง

Slide 56

Slide 56 text

unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - store_test_results: path: ~/test-results config.yml ๐Ÿšง

Slide 57

Slide 57 text

unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | when: always - store_test_results: path: ~/test-results config.yml ๐Ÿšง

Slide 58

Slide 58 text

unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | mkdir -p ~/test-results/junit/ when: always - store_test_results: path: ~/test-results config.yml ๐Ÿšง

Slide 59

Slide 59 text

unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | mkdir -p ~/test-results/junit/ find . when: always - store_test_results: path: ~/test-results config.yml ๐Ÿšง

Slide 60

Slide 60 text

unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | mkdir -p ~/test-results/junit/ find . \ -type f when: always - store_test_results: path: ~/test-results config.yml ๐Ÿšง

Slide 61

Slide 61 text

unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | mkdir -p ~/test-results/junit/ find . \ -type f \ -regex โ€œ.*/build/test-results/.*xml when: always - store_test_results: path: ~/test-results config.yml ๐Ÿšง

Slide 62

Slide 62 text

unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | mkdir -p ~/test-results/junit/ find . \ -type f \ -regex โ€œ.*/build/test-results/.*xmlโ€ \โ€จ -exec cp {} when: always - store_test_results: path: ~/test-results config.yml ๐Ÿšง

Slide 63

Slide 63 text

unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | mkdir -p ~/test-results/junit/ find . \ -type f \ -regex โ€œ.*/build/test-results/.*xmlโ€ \โ€จ -exec cp {} ~/test-results/junit/ \; when: always - store_test_results: path: ~/test-results config.yml ๐Ÿšง

Slide 64

Slide 64 text

unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | mkdir -p ~/test-results/junit/ find . \ -type f \ -regex โ€œ.*/build/test-results/.*xmlโ€ \โ€จ -exec cp {} ~/test-results/junit/ \; when: always - store_test_results: path: ~/test-results config.yml ๐Ÿšง

Slide 65

Slide 65 text

unit_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew testDebugUnitTest - run: name: Save Test Results command: | mkdir -p ~/test-results/junit/ find . \ -type f \ -regex โ€œ.*/build/test-results/.*xmlโ€ \โ€จ -exec cp {} ~/test-results/junit/ \; when: always - store_test_results: path: ~/test-results config.yml โœ…

Slide 66

Slide 66 text

version: 2.1 jobs: ui_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew connectedDebugAndroidTest config.yml โœ… ๐Ÿšง

Slide 67

Slide 67 text

version: 2.1 jobs: ui_tests: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew connectedDebugAndroidTest config.yml โŒ โœ… ๐Ÿšง

Slide 68

Slide 68 text

Do not run emulators on CI yourself โŒ Slow Flaky

Slide 69

Slide 69 text

But how should we automate UI tests? Donโ€™t write them to begin with

Slide 70

Slide 70 text

But how should we automate UI tests? Donโ€™t write them to begin with OR

Slide 71

Slide 71 text

But how should we automate UI tests? Donโ€™t write them to begin with Use a fast vendor OR

Slide 72

Slide 72 text

๐Ÿ’ป Manual Publishing

Slide 73

Slide 73 text

Automate Publishing ๐Ÿ’ป

Slide 74

Slide 74 text

แ… BikeShed > ./gradlew tasks > Task :tasks ------------------------------------------------------------ Tasks runnable from root project 'BikeShed' ------------------------------------------------------------ Android tasks ------------- androidDependencies - Displays the Android dependencies of the project. signingReport - Displays the signing info for the base and test modules sourceSets - Prints out all the source sets defined in this project. Build tasks ----------- assemble - Assemble main outputs for all the variants. assembleAndroidTest - Assembles all the Test applications. build - Assembles and tests this project. buildDependents - Assembles and tests this project and all projects that depend on it. buildKotlinToolingMetadata - Build metadata json file containing information about the used Kotlin tooling buildNeeded - Assembles and tests this project and all projects it depends on.

Slide 75

Slide 75 text

Android tasks ------------- androidDependencies - Displays the Android dependencies of the project. signingReport - Displays the signing info for the base and test modules sourceSets - Prints out all the source sets defined in this project. Build tasks ----------- assemble - Assemble main outputs for all the variants. assembleAndroidTest - Assembles all the Test applications. build - Assembles and tests this project. buildDependents - Assembles and tests this project and all projects that depend on it. buildKotlinToolingMetadata - Build metadata json file containing information about the used Kotlin tooling buildNeeded - Assembles and tests this project and all projects it depends on. bundle - Assemble bundles for all the variants. clean - Deletes the build directory. compileDebugAndroidTestSources compileDebugSources compileDebugUnitTestSources compileReleaseSources compileReleaseUnitTestSources Build Setup tasks ----------------- init - Initializes a new Gradle build. wrapper - Generates Gradle wrapper files.

Slide 76

Slide 76 text

Android tasks ------------- androidDependencies - Displays the Android dependencies of the project. signingReport - Displays the signing info for the base and test modules sourceSets - Prints out all the source sets defined in this project. Build tasks ----------- assemble - Assemble main outputs for all the variants. assembleAndroidTest - Assembles all the Test applications. bundle - Assemble bundles for all the variants.

Slide 77

Slide 77 text

version: 2.1 jobs: build: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew assembleDebug - run: ./gradlew assembleDebugAndroidTest - run: scripts/upload_and_test.sh config.yml ๐Ÿ‘ฉ๐Ÿ”ฌ

Slide 78

Slide 78 text

version: 2.1 jobs: publish: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew bundleRelease config.yml

Slide 79

Slide 79 text

version: 2.1 jobs: publish: docker: - image: circleci/android:api-34-node steps: - checkout - run: ./gradlew bundleRelease - run: ./gradlew publishBundle config.yml

Slide 80

Slide 80 text

version: 2.1 jobs: publish: docker: - image: circleci/android:api-34-node steps: - checkout - run: | openssl aes-256-cbc -d \ -in bikeshed-keystore-secret \ -out ./app/bikeshed.keystore \ -k $SECRET_KEY - run: ./gradlew bundleRelease - run: ./gradlew publishBundle config.yml

Slide 81

Slide 81 text

Static Analysis ktlint to ensure consistency in writing detekt for code smells xmlstarlet to validate XML structure

Slide 82

Slide 82 text

๐Ÿ— Automate Building & Publishing ๐Ÿ‘ฉ๐Ÿ”ฌ Automate Tests ๐Ÿ” Automate Security CI Solutions

Slide 83

Slide 83 text

No content

Slide 84

Slide 84 text

โ€œBut why is it taking so long?โ€

Slide 85

Slide 85 text

๐Ÿ— Automate Building & Publishing โ€œBut why is it taking so long?โ€ โฐ

Slide 86

Slide 86 text

๐Ÿ— Automate Building & Publishing ๐Ÿ‘ฉ๐Ÿ”ฌ Automate Tests โ€œBut why is it taking so long?โ€ โฐ

Slide 87

Slide 87 text

๐Ÿ— Automate Building & Publishing ๐Ÿ‘ฉ๐Ÿ”ฌ Automate Tests ๐Ÿ” Automate Security โ€œBut why is it taking so long?โ€ โฐ

Slide 88

Slide 88 text

๐Ÿ— Automate Building & Publishing ๐Ÿ‘ฉ๐Ÿ”ฌ Automate Tests ๐Ÿ” Automate Security โ€œBut why is it taking so long?โ€ โฐ

Slide 89

Slide 89 text

๐Ÿ— Automate Building & Publishing ๐Ÿ‘ฉ๐Ÿ”ฌ Automate Tests ๐Ÿ” Automate Security โฐ ๐Ÿ“ฆ Caching ๐Ÿญ Parallelization Merge Queue ๐Ÿฅท Shadow Jobs Original Problems Scale Problems

Slide 90

Slide 90 text

This is what we want 12 9 6 3 0 Unit Tests Release R8 UI Tests Unit Tests Release R8 UI tests minutes

Slide 91

Slide 91 text

12 9 6 3 0 Unit Tests Release R8 UI Tests Unit Tests Release R8 UI tests minutes How do we get to this?

Slide 92

Slide 92 text

12 9 6 3 0 Unit Tests Release R8 UI Tests Unit Tests Release R8 UI tests minutes How do we get to this?

Slide 93

Slide 93 text

Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker ๐Ÿ‘ฉ๐Ÿ”ฌ Unit Tests on CI

Slide 94

Slide 94 text

Cache Dependencies ๐Ÿ“ฆ

Slide 95

Slide 95 text

Gradle Caches Build Cache Configuration Cache Dependency Cache Script Cache

Slide 96

Slide 96 text

Gradle Caches Build Cache EXECUTING Configuration Cache Dependency Cache Script Cache

Slide 97

Slide 97 text

Gradle Caches Build Cache EXECUTING Configuration Cache Dependency Cache Script Cache

Slide 98

Slide 98 text

Gradle Caches Build Cache EXECUTING Configuration Cache Dependency Cache Script Cache

Slide 99

Slide 99 text

Gradle Caches Build Cache CONFIGURING EXECUTING Configuration Cache Dependency Cache Script Cache

Slide 100

Slide 100 text

Gradle Caches Build Cache Configuration Cache Dependency Cache Script Cache

Slide 101

Slide 101 text

๐Ÿ’ป Local Support Gradle Caches Build Cache ๐Ÿ’ป ๐Ÿ’ป ๐Ÿ’ป Configuration Cache Dependency Cache Script Cache Local Support Local Support Local Support

Slide 102

Slide 102 text

๐Ÿ’ป Local Support Gradle Caches Build Cache ๐Ÿ’ป ๐Ÿ’ป ๐Ÿ’ป Configuration Cache Dependency Cache Script Cache Local Support Local Support Local Support

Slide 103

Slide 103 text

Gradle Cache directories ~/.gradle โ”œโ”€โ”€ caches โ”‚ โ”œโ”€โ”€ 8.1 โ”‚ โ”œโ”€โ”€ 8.2 โ”‚ โ”œโ”€โ”€ โ‹ฎ โ”‚ โ”œโ”€โ”€ buildcacheโ€”1 โ”‚ โ”œโ”€โ”€ jars-9 โ”‚ โ”œโ”€โ”€ journal-1 โ”‚ โ”œโ”€โ”€ modules-2 โ”‚ โ””โ”€โ”€ transforms-3 โ”‚ โ”œโ”€โ”€ โ‹ฎ

Slide 104

Slide 104 text

Gradle Cache directories ~/.gradle โ”œโ”€โ”€ caches โ”‚ โ”œโ”€โ”€ 8.1 โ”‚ โ”œโ”€โ”€ 8.2 โ”‚ โ”œโ”€โ”€ โ‹ฎ โ”‚ โ”œโ”€โ”€ buildcacheโ€”1 โ”‚ โ”œโ”€โ”€ jars-9 โ”‚ โ”œโ”€โ”€ journal-1 โ”‚ โ”œโ”€โ”€ modules-2 โ”‚ โ””โ”€โ”€ transforms-3 โ”‚ โ”œโ”€โ”€ โ‹ฎ Dependency Cache

Slide 105

Slide 105 text

Gradle Cache directories ~/.gradle โ”œโ”€โ”€ caches โ”‚ โ”œโ”€โ”€ 8.1 โ”‚ โ”œโ”€โ”€ 8.2 โ”‚ โ”‚ โ”œโ”€โ”€ generated-gradle-jars โ”‚ โ”‚ โ””โ”€โ”€ kotlin-dsl โ”‚ โ”œโ”€โ”€ โ‹ฎ โ”‚ โ”œโ”€โ”€ buildcacheโ€”1 โ”‚ โ”œโ”€โ”€ jars-9 โ”‚ โ”œโ”€โ”€ journal-1 โ”‚ โ”œโ”€โ”€ modules-2 โ”‚ โ””โ”€โ”€ transforms-3 โ”‚ Dependency Cache Script Cache

Slide 106

Slide 106 text

~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/modules-2 ci-cache.yml

Slide 107

Slide 107 text

๐Ÿ’ป Local Support Gradle Caches Build Cache ๐Ÿ’ป ๐Ÿ’ป ๐Ÿ’ป Configuration Cache Dependency Cache Script Cache Local Support Local Support Local Support

Slide 108

Slide 108 text

Gradle Caches Build Cache ๐Ÿ’ป Configuration Cache Dependency Cache Script Cache Local Support ๐Ÿ’ป Local Support Remote Support Remote Support

Slide 109

Slide 109 text

Control Cache build dependencies 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker

Slide 110

Slide 110 text

Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change Cache build dependencies minutes cache save gradle execution gradle configuration cache restore git Docker

Slide 111

Slide 111 text

Gradle Properties Prioritizing quick one-liner quick wins None of these are defaults

Slide 112

Slide 112 text

# If you don't turn these on you aren't # getting any performance wins from # Gradle Build Cache or modularization. org.gradle.caching=true org.gradle.parallel=true gradle.properties

Slide 113

Slide 113 text

# Kotlin 1.7+ only # Good reason to upgrade if you are on # an older version. kotlin.incremental.useClasspathSnapshot =true gradle.properties

Slide 114

Slide 114 text

# Incremental compilation with KAPT. # Only if you still use KAPT. kapt.incremental.apt =true gradle.properties

Slide 115

Slide 115 text

# Stop generating BuildConfig Java stubs. # If this is the last Java in your codebase # you could see a decent performance win. android.enableBuildConfigAsBytecode =true gradle.properties

Slide 116

Slide 116 text

# Stop generating BuildConfig files # if you don't use them. android.defaults.buildfeatures.buildconfig =false gradle.properties

Slide 117

Slide 117 text

# Experimental flag for parallelizing # R8 work - only useful if you have # multiple R8 tasks. android.r8.maxWorkers =2 gradle.properties

Slide 118

Slide 118 text

# Summary org.gradle.caching=true org.gradle.parallel=true kotlin.incremental.useClasspathSnapshot=true kapt.incremental.apt=true android.enableBuildConfigAsBytecode=true android.defaults.buildfeatures.buildconfig=false android.r8.maxWorkers=2 gradle.properties

Slide 119

Slide 119 text

Gradle Properties

Slide 120

Slide 120 text

Gradle Properties Other engineers ask me, "Where do you find these?" github.com/androidx/androidx and other OSS projects

Slide 121

Slide 121 text

Gradle Plugins & Tweaks

Slide 122

Slide 122 text

Gradle Plugins & Tweaks Gradle Cache Fix Plugin Disable LiveLiterals if you use Compose

Slide 123

Slide 123 text

# Re-renables Android Cache fix plugin for currently disabled defaults we believe are safe for a speed boost. systemProp.org.gradle.android.cache- fix.MergeSourceSetFolders.caching.enabled=true systemProp.org.gradle.android.cache- fix.ZipMergingTask.caching.enabled=true gradle.properties

Slide 124

Slide 124 text

android { composeOptions { useLiveLiterals = project.providers.gradleProperty("useLiveLiterals") .get() .toBooleanStrict() } } build.gradle.kts

Slide 125

Slide 125 text

android { composeOptions { useLiveLiterals = false } } build.gradle.kts

Slide 126

Slide 126 text

Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change Cache build dependencies minutes cache save gradle execution gradle configuration cache restore git Docker

Slide 127

Slide 127 text

Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change Cache build dependencies minutes cache save gradle execution gradle configuration cache restore git Docker

Slide 128

Slide 128 text

๐Ÿญ Parallelization

Slide 129

Slide 129 text

๐Ÿญ Parallelization

Slide 130

Slide 130 text

Overhead Cost ๐Ÿ“ฆ ๐Ÿ“ฆ ๐Ÿ“ฆ ๐Ÿ“ฆ ๐Ÿญ Parallelization

Slide 131

Slide 131 text

Overhead Cost ๐Ÿ“ฆ ๐Ÿ“ฆ ๐Ÿ“ฆ ๐Ÿ“ฆ Depending on how you organize your CI jobs these costs can become significant. ๐Ÿญ Parallelization

Slide 132

Slide 132 text

No content

Slide 133

Slide 133 text

โ€œMain is failing even though we passed CIโ€

Slide 134

Slide 134 text

No content

Slide 135

Slide 135 text

Changed Shared Behavior Changed CI Checks Added โ€œregressionโ€ when run in future builds

Slide 136

Slide 136 text

Merge Queue

Slide 137

Slide 137 text

Cache Build Steps ๐Ÿ“ฆ

Slide 138

Slide 138 text

Gradle Caches Build Cache ๐Ÿ’ป Configuration Cache Dependency Cache Script Cache Local Support ๐Ÿ’ป Local Support Remote Support Remote Support

Slide 139

Slide 139 text

Gradle Caches Build Cache ๐Ÿ’ป Configuration Cache Dependency Cache Script Cache Local Support ๐Ÿ’ป Local Support Remote Support Remote Support

Slide 140

Slide 140 text

Gradle Build Cache - Cacheable Tasks

Slide 141

Slide 141 text

Gradle Build Cache - Modularize โ€ข Reduce the number of inputs per task and make them more stable โ€ข Watch Siggiโ€™s talk about graph theory โ€ข Gradle build scans to observe performance

Slide 142

Slide 142 text

Gradle Remote Cache โ€ข Do it: https://docs.gradle.com/build-cache-node/ โ€ข 90 Days, 300GB, use validation scripts โ€ข Most local and CI runs will benefit from build cache hits โ€ข Free

Slide 143

Slide 143 text

Gradle Enterprise

Slide 144

Slide 144 text

No content

Slide 145

Slide 145 text

โ€ข Provides support for features beyond remote cache โ€ข Really helps figure out failures, cache misses, and performance issues across a team โ€ข Looking forward to using test prediction

Slide 146

Slide 146 text

Gradle Caches Build Cache ๐Ÿ’ป Configuration Cache Dependency Cache Script Cache Local Support ๐Ÿ’ป Local Support Remote Support Remote Support

Slide 147

Slide 147 text

Gradle Caches Build Cache ๐Ÿ’ป Configuration Cache Dependency Cache Script Cache Remote Support Local Support Remote Support Remote Support

Slide 148

Slide 148 text

Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker Gradle Build Cache Optimization

Slide 149

Slide 149 text

Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker Gradle Build Cache Optimization

Slide 150

Slide 150 text

Is it worth it to use the latest JDK

Slide 151

Slide 151 text

Is it worth it to use the latest JDK Yes, 19+

Slide 152

Slide 152 text

Control JDK 11 -> JDK 19 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker

Slide 153

Slide 153 text

Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker JDK 11 -> JDK 19

Slide 154

Slide 154 text

Upgrade Tooling

Slide 155

Slide 155 text

Upgrade Tooling โ€œWill JDK 19โ€จ work on our project?โ€

Slide 156

Slide 156 text

Upgrade Tooling โ€œWill version X of dependency Y work on our project?โ€

Slide 157

Slide 157 text

๐Ÿฅท Shadow Jobs โ€œWill version X of dependency Y work on our project?โ€

Slide 158

Slide 158 text

๐Ÿฅท Shadow Jobs Run nightly checks against your project with latest snapshots of new versions. โ€œWill version X of dependency Y work on our project?โ€ Zac Sweers

Slide 159

Slide 159 text

๐Ÿฅท Shadow Jobs Run nightly checks against your project with latest snapshots of new versions. โ€œWill version X of dependency Y work on our project?โ€ Zac Sweers

Slide 160

Slide 160 text

JVM Argument Tuning

Slide 161

Slide 161 text

JVM Argument Tuning DISCLAIMER: I am not accountable for any of these settings. If they donโ€™t work for you there are so many reasons why your setup would handle them differently.

Slide 162

Slide 162 text

Garbage Collection Algo Tradeoff ๐Ÿ—‘

Slide 163

Slide 163 text

Garbage Collection Algo Tradeoff Each JVM GC algorithm makes a choice about what they optimize for and how much.

Slide 164

Slide 164 text

Garbage Collection Algo Tradeoff

Slide 165

Slide 165 text

Garbage Collection Algo Tradeoff Throughput Transactions per second

Slide 166

Slide 166 text

Garbage Collection Algo Tradeoff Throughput Latency Transactions per second Time spent in GC pauses

Slide 167

Slide 167 text

Garbage Collection Algo Tradeoff Throughput Footprint Latency Transactions per second Time spent in GC pauses Rough memory footprint size

Slide 168

Slide 168 text

Garbage Collection Algo Tradeoff Throughput Footprint Latency

Slide 169

Slide 169 text

Garbage Collection Algo Tradeoff Throughput Footprint Latency

Slide 170

Slide 170 text

Garbage Collection Algo Tradeoff Throughput Footprint Latency

Slide 171

Slide 171 text

Garbage Collection Algo Tradeoff Throughput Footprint Latency ParallelGC G1GC

Slide 172

Slide 172 text

Garbage Collection Algo Tradeoff Throughput Footprint Latency ParallelGC G1GC

Slide 173

Slide 173 text

Garbage Collection Algo Tradeoff Throughput Footprint Latency ParallelGC G1GC

Slide 174

Slide 174 text

ParallelGC Garbage Collection Algo Tradeoff G1GC

Slide 175

Slide 175 text

ParallelGC Garbage Collection Algo Tradeoff G1GC โ€ข Optimize for speed โ€ข Maximum footprint โ€ข Might have longer GC pauses โ€ข Balance throughput โ€ข Lower footprint โ€ข Shorter GC pauses

Slide 176

Slide 176 text

ParallelGC Garbage Collection Algo Tradeoff G1GC โ€ข Optimize for speed โ€ข Maximum footprint โ€ข Might have longer GC pauses โ€ข Balance throughput โ€ข Lower footprint โ€ข Shorter GC pauses Use when resource constrained Use when resources allow

Slide 177

Slide 177 text

kotlin.daemon.jvm.options When passing arguments, follow these rules: Use the minus sign - only before the arguments Xmx, XX:MaxMetaspaceSize, and XX:ReservedCodeCacheSize. Separate arguments with commas (,) without spaces. Arguments that come after a space will be used for the Gradle daemon, not for the Kotlin daemon. Kotlin JVM Arguments

Slide 178

Slide 178 text

kotlin.daemon.jvm.options When passing arguments, follow these rules: Use the minus sign - only before the arguments Xmx, XX:MaxMetaspaceSize, and XX:ReservedCodeCacheSize. Separate arguments with commas (,) without spaces. Arguments that come after a space will be used for the Gradle daemon, not for the Kotlin daemon. Kotlin JVM Arguments

Slide 179

Slide 179 text

kotlin.daemon.jvm.options When passing arguments, follow these rules: Use the minus sign - only before the arguments Xmx, XX:MaxMetaspaceSize, and XX:ReservedCodeCacheSize. Separate arguments with commas (,) without spaces. Arguments that come after a space will be used for the Gradle daemon, not for the Kotlin daemon. Kotlin JVM Arguments

Slide 180

Slide 180 text

kotlin.daemon.jvmargs You can add the kotlin.daemon.jvmargs property in the gradle.properties fi le: kotlin.daemon.jvmargs=-Xmx1500m -Xms=500m Kotlin JVM Arguments

Slide 181

Slide 181 text

kotlin.daemon.jvmargs You can add the kotlin.daemon.jvmargs property in the gradle.properties fi le: kotlin.daemon.jvmargs=-Xmx1500m -Xms=500m Using spaces at the beginning will cause all arguments to be ignored Kotlin JVM Arguments

Slide 182

Slide 182 text

Xmx (max) JVM Heap Size Xms (min) โ€ข Amount of memory it takes to compile your biggest task / module. โ€ข For most projects this is going to be your main app moduleโ€™s minification step. โ€ข Somewhere between 80 and 100% of Xmx. โ€ข Heap only grows when GC happens, so increasing it avoids this penalty.

Slide 183

Slide 183 text

JVM Heap Size & Modularization are related

Slide 184

Slide 184 text

MaxMetaspaceSize Metaspace CompressedClassSpaceSize โ€ข class & non-class space โ€ข unlimited by default โ€ข virtual size of compressed classes โ€ข defaults to 1GB

Slide 185

Slide 185 text

Metaspace https://stuefe.de/posts/metaspace/analyze-metaspace-with-jcmd/

Slide 186

Slide 186 text

CompressedOops Metaspace CompressedClassPointers โ€ข Defaults on < 30GB heap โ€ข Uses 4 fewer bytes โ€ข Restricts memory heap to 32GB โ€ข Available if CompressedOops is on

Slide 187

Slide 187 text

What if we turn off CompressedClassSpace + CompressedOops? 20 15 10 5 0 On Off On Off minutes

Slide 188

Slide 188 text

Metaspace CompressedOops & unlimited MaxMetaspace is going to be better for majority of Android projects.

Slide 189

Slide 189 text

Metaspace CompressedOops & unlimited MaxMetaspace is going to be better for majority of Android projects. (so stick with the defaults)

Slide 190

Slide 190 text

Metaspace ~1GB Metaspace helps avoid early Metaspace-related GCs

Slide 191

Slide 191 text

Measuring Metaspace jcmp VM.metaspace

Slide 192

Slide 192 text

CodeCache Where the JVM stores its bytecode into compiled native code

Slide 193

Slide 193 text

CodeCache Where the JVM stores its bytecode into compiled native code

Slide 194

Slide 194 text

CodeCache ReservedCodeCacheSize is the maximum amount for cached bytecode

Slide 195

Slide 195 text

CodeCache So we gave ReservedCodeCacheSize 512mb

Slide 196

Slide 196 text

CodeCache To determine what number you should use, measure via โ€“XX:+PrintCodeCache

Slide 197

Slide 197 text

Measuring CodeCache In JVM args: โ€“XX:+PrintCodeCache

Slide 198

Slide 198 text

SoftRefLRUPolicyMSPerMB The number of milliseconds to hold onto soft references per MB of heap size, default 1000

Slide 199

Slide 199 text

SoftRefLRUPolicyMSPerMB The number of milliseconds to hold onto soft references per MB of heap size, default 1000 = 2h 16m 32s 8GB Heap @ 1000 MS Policy

Slide 200

Slide 200 text

SoftRefLRUPolicyMSPerMB The number of milliseconds to hold onto soft references per MB of heap size, default 1000 = 2h 16m 32s 8GB Heap @ 1000 MS Policy 8GB Heap @ 10 MS Policy = 0h 1m 22s

Slide 201

Slide 201 text

SoftRefLRUPolicyMSPerMB The number of milliseconds to hold onto soft references per MB of heap size, default 1000 = 2h 16m 32s 8GB Heap @ 1 MS Policy 8GB Heap @ 1000 MS Policy = 0h 0m 8s 8GB Heap @ 10 MS Policy = 0h 1m 22s

Slide 202

Slide 202 text

Gradle Max Workers --max-workers The maximum number of active workers to allow Should probably be number of performance cores, possibly 1-2 less Will not spawn more workers is memory heap requirements cannot be met

Slide 203

Slide 203 text

jemalloc / tcmalloc Memory fragmentation slows down builds, use a better malloc algorithm

Slide 204

Slide 204 text

malloc alternatives Memory fragmentation slows down builds, use a better malloc algorithm

Slide 205

Slide 205 text

malloc alternatives We saw no speed difference with jemalloc, but it did notice 10% less memory usage

Slide 206

Slide 206 text

malloc alternatives Therefore use jemalloc (or tcmalloc) to be able to run more daemons in parallel with less memory pressure.

Slide 207

Slide 207 text

What do we actually pick for JVM args?

Slide 208

Slide 208 text

๐Ÿฅท Shadow Jobs 1. Instrument your build with JVM args as parameters 2. Run parameterized tests 3. Profit What do we actually pick for JVM args?

Slide 209

Slide 209 text

โ€ข GC Algo = It Depends (try for Parallel) โ€ข SoftRefLRUPolicyMSPerMB = 1 โ€ข CodeCache = 256MB - 512MB โ€ข MaxMetaspace = Some GB or unlimited โ€ข Metaspace = 64MB - 1GB โ€ข Xmx = 4GB - 8GB โ€ข Xms = 3GB - 7GB โ€ข Max Workers=Performance Cores - 2 What do we actually pick?

Slide 210

Slide 210 text

โ€ข jcmp for Metaspace โ€ข โ€“XX:+PrintCodeCache for CodeCache โ€ข Measure CPU & Memory resource usage, lots of tools available JVM Measurement Tools

Slide 211

Slide 211 text

Control JVM Optimizations 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker

Slide 212

Slide 212 text

Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker JVM Optimizations

Slide 213

Slide 213 text

Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker JVM Optimizations

Slide 214

Slide 214 text

Gradle Caches Build Cache Configuration Cache Dependency Cache Script Cache Remote Support Remote Support Remote Support ๐Ÿ’ป Local Support

Slide 215

Slide 215 text

Gradle Caches Build Cache Configuration Cache Dependency Cache Script Cache Remote Support Remote Support Remote Support ๐Ÿ’ป Local Support

Slide 216

Slide 216 text

# Enables the configuration cache: org.gradle.configuration-cache=true org.gradle.configuration-cache.problems=warn org.gradle.configuration-cache.inputs.unsafe.ignore.file-system- checks=(see github.com/androidx/androidx repo gradle.properties) gradle.properties

Slide 217

Slide 217 text

Gradle Cache directories ~/.gradle โ”œโ”€โ”€ caches โ”‚ โ””โ”€โ”€ โ‹ฎ Dependency Cache Script Cache

Slide 218

Slide 218 text

Gradle Cache directories ~/.gradle โ”œโ”€โ”€ caches โ”‚ โ””โ”€โ”€ โ‹ฎ ~/project โ”œโ”€โ”€ .gradle โ”œโ”€โ”€ app โ”œโ”€โ”€ build โ”œโ”€โ”€ gradle โ””โ”€โ”€ โ‹ฎ Dependency Cache Script Cache

Slide 219

Slide 219 text

Gradle Cache directories ~/project โ”œโ”€โ”€ .gradle โ”œโ”€โ”€ app โ”œโ”€โ”€ build โ”œโ”€โ”€ gradle โ””โ”€โ”€ โ‹ฎ

Slide 220

Slide 220 text

Gradle Cache directories ~/project โ”œโ”€โ”€ .gradle โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ app โ”œโ”€โ”€ build โ”œโ”€โ”€ gradle โ””โ”€โ”€ โ‹ฎ

Slide 221

Slide 221 text

Gradle Cache directories ~/project โ”œโ”€โ”€ .gradle โ”‚ โ”œโ”€โ”€ 8.1 โ”‚ โ”œโ”€โ”€ 8.2 โ”‚ โ”œโ”€โ”€ โ‹ฎ โ”‚ โ””โ”€โ”€ configuration-cache โ”œโ”€โ”€ app โ”œโ”€โ”€ build โ”œโ”€โ”€ gradle โ””โ”€โ”€ โ‹ฎ

Slide 222

Slide 222 text

~/project โ”œโ”€โ”€ .gradle โ”‚ โ”œโ”€โ”€ 8.1 โ”‚ โ”œโ”€โ”€ 8.2 โ”‚ โ”œโ”€โ”€ โ‹ฎ โ”‚ โ””โ”€โ”€ configuration-cache โ”œโ”€โ”€ app โ”œโ”€โ”€ build โ”œโ”€โ”€ gradle โ””โ”€โ”€ โ‹ฎ Configuration Cache Gradle Cache directories

Slide 223

Slide 223 text

~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/modules-2 .gradle/<< gradle_version >> .gradle/configuration-cache ci-cache.yml

Slide 224

Slide 224 text

As of Gradle 8.1 encryption was added. Asking Gradle support - configuration cache on CI is unsupported Gradle Configuration Cache on CI

Slide 225

Slide 225 text

As of Gradle 8.1 encryption was added. Gradle's response has been that configuration cache on CI is not supported Gradle Configuration Cache on CI

Slide 226

Slide 226 text

As of Gradle 8.1 encryption was added. Gradle's response has been that configuration cache on CI is not supported. But we persisted anyway... Gradle Configuration Cache on CI

Slide 227

Slide 227 text

Gradle Configuration Cache on CI Environment Variable Environment Variables Prefixed File system Entry File System property Value from custom source 35 1 2007 4 52 5 build/reports/configuration-cache/4tnz..i67j/27o6..b0jl/configuration-cache-report.html We looked at the reports for all inputs

Slide 228

Slide 228 text

But despite everything 100% of inputs being consistent we continued to get an unhelpful error message: Gradle Configuration Cache on CI

Slide 229

Slide 229 text

Gradle Configuration Cache on CI Calculating task graph as no configuration cache is available for tasks

Slide 230

Slide 230 text

Gradle Configuration Cache on CI โ€œYou need to cache the encryption key.โ€ โ€” Inaki Villar

Slide 231

Slide 231 text

Gradle Configuration Cache on CI Caveat to this method - you will be caching any secrets used by this build, so either: โ€ข Only use this for builds that do not rely on secrets โ€ข Knowingly store this with the security risk

Slide 232

Slide 232 text

No content

Slide 233

Slide 233 text

Gradle Configuration Cache on CI

Slide 234

Slide 234 text

Gradle User Home directory ~/.gradle โ”œโ”€โ”€ caches โ”‚ โ”œโ”€โ”€ 8.1 โ”‚ โ”‚ โ€ฆ โ”‚ โ”œโ”€โ”€ 8.4 โ”‚ โ”‚ โ””โ”€โ”€ cc-keystore โ”‚ โ”‚ โ”œโ”€โ”€ cache.properties โ”‚ โ”‚ โ”œโ”€โ”€ cc-keystore.lock โ”‚ โ”‚ โ””โ”€โ”€ gradle.keystore โ”‚ โ”œโ”€โ”€ โ”‚ โ”œโ”€โ”€ buildcacheโ€”1 โ”‚ โ”œโ”€โ”€ jars-9 โ”‚ โ”œโ”€โ”€ journal-1 Configuration Cache Encryption

Slide 235

Slide 235 text

~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/modules-2 .gradle/<< gradle_version >> .gradle/configuration-cache ci-cache.yml

Slide 236

Slide 236 text

~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/<< gradle_version >>/cc-keystore ~/.gradle/caches/modules-2 .gradle/<< gradle_version >> .gradle/configuration-cache ci-cache.yml

Slide 237

Slide 237 text

Gradle Configuration Cache on CI Calculating task graph as no configuration cache is available for tasks

Slide 238

Slide 238 text

unit_tests: docker: - image: circleci/android:api-34-node steps: - run: name: Clean Gradle Cache command: | find . -type f -name '*.lock' -delete find . -type f -name 'gc.properties' -delete find ~/.gradle -type f -name '*.lock' -delete find ~/.gradle -type f -name 'gc.properties' -delete when: always - save_cache: paths: - ~/.gradle/caches/<< parameters.gradle_version >>/ generated-gradle-jars - ~/.gradle/caches/<< parameters.gradle_version >>/ kotlin-dsl - .gradle/configuration-cache - ~/.gradle/caches/<< parameters.gradle_version >>/ cc-keystore - .gradle/<< parameters.gradle_version >> config.yml

Slide 239

Slide 239 text

unit_tests: docker: - image: circleci/android:api-34-node steps: - run: name: Clean Gradle Cache command: | find . -type f -name '*.lock' -delete find . -type f -name 'gc.properties' -delete find ~/.gradle -type f -name '*.lock' -delete find ~/.gradle -type f -name 'gc.properties' -delete when: always - save_cache: paths: - ~/.gradle/caches/<< parameters.gradle_version >>/ generated-gradle-jars - ~/.gradle/caches/<< parameters.gradle_version >>/ kotlin-dsl - .gradle/configuration-cache - ~/.gradle/caches/<< parameters.gradle_version >>/ cc-keystore - .gradle/<< parameters.gradle_version >> config.yml ๐Ÿคฆ

Slide 240

Slide 240 text

Gradle Configuration Cache on CI Calculating task graph as configuration cache cannot be reused because an input to task โ€˜:gradle-plugin:generateExternalPluginSpecBuilders' has changed.

Slide 241

Slide 241 text

Gradle Configuration Cache on CI Calculating task graph as configuration cache cannot be reused because an input to task โ€˜:gradle-plugin:generateExternalPluginSpecBuilders' has changed.

Slide 242

Slide 242 text

โ€œGradle has compilation avoidance for non- ABI changes in included builds, but due to a bug any inline function would cause all build scripts to be recompiled even when there is a non-ABI change.โ€ - Nicklas Ansman Gradle Configuration Cache on CI

Slide 243

Slide 243 text

~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/<< gradle_version >>/cc-keystore ~/.gradle/caches/modules-2 .gradle/<< gradle_version >> .gradle/configuration-cache gradle-plugin/build/pluginDescriptors ci-cache.yml

Slide 244

Slide 244 text

Gradle Configuration Cache on CI Could not load the value of field `patterns` of `org.gradle.configurationcache.serialization.codecs.DirectoryTree Spec` bean found in field `fileSystemInputs` of `org.gradle.configurationcache.fingerprint.ConfigurationCacheFing erprint$WorkInputs` bean found in Gradle runtime. > Failed to instrument class org/jetbrains/kotlin/gradle/tasks/ KotlinCompile$ScriptFilterSpec in ClassLoaderScopeIdentifier.Id{coreAndPlugins:settings[:gradle- plugin]:buildSrc[:gradle-plugin]:root-project[:gradle-plugin] (export)}

Slide 245

Slide 245 text

Gradle Configuration Cache on CI Could not load the value of field `patterns` of `org.gradle.configurationcache.serialization.codecs.DirectoryTree Spec` bean found in field `fileSystemInputs` of `org.gradle.configurationcache.fingerprint.ConfigurationCacheFing erprint$WorkInputs` bean found in Gradle runtime. > Failed to instrument class org/jetbrains/kotlin/gradle/tasks/ KotlinCompile$ScriptFilterSpec in ClassLoaderScopeIdentifier.Id{coreAndPlugins:settings[:gradle- plugin]:buildSrc[:gradle-plugin]:root-project[:gradle-plugin] (export)}

Slide 246

Slide 246 text

~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/<< gradle_version >>/cc-keystore ~/.gradle/caches/modules-2 .gradle/<< gradle_version >> .gradle/configuration-cache gradle-plugin/build/pluginDescriptors ci-cache.yml

Slide 247

Slide 247 text

~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/<< gradle_version >>/cc-keystore ~/.gradle/caches/modules-2 .gradle/<< gradle_version >> .gradle/configuration-cache gradle-plugin/build ci-cache.yml

Slide 248

Slide 248 text

~/.gradle/caches/<< gradle_version >>/generated-gradle-jars ~/.gradle/caches/<< gradle_version >>/kotlin-dsl ~/.gradle/caches/<< gradle_version >>/cc-keystore ~/.gradle/caches/modules-2 ~/.gradle/caches/jars-9 ~/.gradle/caches/transforms-3 .gradle/<< gradle_version >> .gradle/configuration-cache gradle-plugin/build ci-cache.yml

Slide 249

Slide 249 text

Reusing configuration cache. Gradle Configuration Cache on CI

Slide 250

Slide 250 text

Gradle Configuration Cache on CI

Slide 251

Slide 251 text

Gradle Configuration Cache on CI

Slide 252

Slide 252 text

โ€œWow, I was prepared for a faster PR time but that was blazingโ€ - Jacob Duron Gradle Configuration Cache on CI

Slide 253

Slide 253 text

Gradle Caches Build Cache Configuration Cache Dependency Cache Script Cache Remote Support Remote Support Remote Support ๐Ÿ’ป Local Support

Slide 254

Slide 254 text

Gradle Caches Build Cache Configuration Cache Dependency Cache Script Cache Remote Support Remote Support Remote Support Remote Support

Slide 255

Slide 255 text

Control Gradle Configuration Cache Reuse 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker

Slide 256

Slide 256 text

Control Gradle Configuration Cache Reuse 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker

Slide 257

Slide 257 text

40 30 20 10 0 Gradle Configuration Cache Reuse Control Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker

Slide 258

Slide 258 text

Overhead Cost Optimizations ๐Ÿ“ฆ git clone pulls 400 MB big images (2.5 GB) Minutes to save cache ~60 seconds to restore cache

Slide 259

Slide 259 text

image size 2560 2048 1536 1024 512 Circle/ Android 0 JRE Bare megabytes Android

Slide 260

Slide 260 text

download & extract time 75 60 45 30 15 0 seconds Circle/ Android JRE Bare Android

Slide 261

Slide 261 text

download & extract time 75 60 45 30 15 0 seconds Circle/ Android JRE Bare Android

Slide 262

Slide 262 text

500 400 300 200 100 0 default no blobs no trees shallow megabytes clone & checkout size

Slide 263

Slide 263 text

75 60 45 30 15 0 seconds default no blobs no trees shallow clone & checkout time

Slide 264

Slide 264 text

75 60 45 30 15 0 seconds default no blobs no trees shallow clone & checkout time

Slide 265

Slide 265 text

Why did we go with treeless if its not the fastest? Shallow is faster, but it comes with a heavy penalty for any additional git operations clone & checkout time

Slide 266

Slide 266 text

prune 40 30 20 10 0 Control Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes cache save gradle execution gradle configuration cache restore git Docker

Slide 267

Slide 267 text

Control 40 30 20 10 0 Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes prune cache save gradle execution gradle configuration cache restore git Docker

Slide 268

Slide 268 text

40 30 20 10 0 Control Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes clone optimization cache save gradle execution gradle configuration cache restore git Docker

Slide 269

Slide 269 text

40 30 20 10 0 Control Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes clone optimization cache save gradle execution gradle configuration cache restore git Docker

Slide 270

Slide 270 text

40 30 20 10 0 Control Partial Cache Hit Cache Hit Core Change Cache Hit Leaf Change minutes Present Day Unit Test Runs cache save gradle execution gradle configuration cache restore git Docker

Slide 271

Slide 271 text

Present Day CI Runs 12 9 6 3 0 Unit Tests Release R8 UI Tests Unit Tests Release R8 UI tests minutes

Slide 272

Slide 272 text

CI Speed Optimizations Cache Dependencies Cache Build Tasks Cache Configuration JVM Tuning Compiler Flags ๐Ÿ“ฆ treeless or shallow clone prune images ๐Ÿ‹ ๐Ÿ“ฆ Cache Optimizations ๐Ÿšฉ

Slide 273

Slide 273 text

CI Speed Optimizations Cache Dependencies Cache Build Tasks Cache Configuration JVM Tuning Compiler Flags ๐Ÿ“ฆ treeless or shallow clone prune images ๐Ÿ‹ ๐Ÿ“ฆ Cache Optimizations ๐Ÿšฉ ๐Ÿšง โœ… โœ… โœ… โœ… Future Talk

Slide 274

Slide 274 text

Topics not covered ๐Ÿ‘ฉ๐Ÿ’ป Local Dev Environment ๐Ÿญ Parallelization ๐Ÿ“ธ Automatic Screenshot Testing

Slide 275

Slide 275 text

Metaspace Citations Thomas Stรผfe Blog https://stuefe.de/posts/metaspace/what-is-metaspace/ https://stuefe.de/posts/metaspace/metaspace-architecture/ https://stuefe.de/posts/metaspace/what-is-compressed-class-space/ https://stuefe.de/posts/metaspace/sizing-metaspace/ https://stuefe.de/posts/metaspace/analyze-metaspace-with-jcmd/ Oracle Class Metadata, PermGen and changes to native memory allocation

Slide 276

Slide 276 text

Gradle Documentation Gradle Files & Directories https://docs.gradle.org/current/userguide/directory_layout.html Gradle CI Ephemeral Best Practices https://docs.gradle.org/current/userguide/ dependency_resolution.html#sub:ephemeral-ci-cache Gradle Recommends using Daemons for CI https://docs.gradle.org/current/userguide/ gradle_daemon.html#continuous_integration

Slide 277

Slide 277 text

Kotlin JVM Args Documentation New Method https://kotlinlang.org/docs/gradle-compilation-and-caches.html#kotlin- daemon-jvmargs-property Deprecated Method https://kotlinlang.org/docs/gradle-compilation-and-caches.html#kotlin- daemon-jvm-options-system-property

Slide 278

Slide 278 text

JDK Version Documentation JDK Release Notes Summaries for GC Algo Changes 19 | 20 | 21 JVM SoftReferences Tuning CodeCache

Slide 279

Slide 279 text

JVM SoftReferences JVM SoftReferences

Slide 280

Slide 280 text

JVM CodeCache Tuning CodeCache

Slide 281

Slide 281 text

Shadow Jobs by Zac Sweers https://slack.engineering/shadow-jobs/