No content

Agenda ● How Netflix does builds ○ Gradle / AGP / Android Studio / CI/CD ○ Eng Mode ○ Modularisation ○ Gradle profiler ● The Developer Productivity Journey ○ The first Build Scan™ ○ Build caching, who’s enabled it and who hasn’t? ○ Identifying build performance bottlenecks ○ Enabling local and remote build caching ● Build Performance Regressions: One year later ○ Android SDK and JDK version ○ CI vs Local cache-miss ○ Code generation ○ Dependencies management ● Wrap Up

How Netflix Does Builds Build Environment & Tools Kapt with Dagger/Hilt, Epoxy, Auto Value App Bundles Jetpack Compose Android Studio / AGP 7.2.1 / Kotlin 1.7.10 Gradle project with ~300 Modules and 10+ variants / flavors

SCM : Bitbucket/stash, CI : Jenkins, CD : Spinnaker Pull request with: ● Static analysis with ktlint, detekt, lint (custom lint) ● Unit testing and on-device testing ● 4.6K CI builds per week (220K per year) ● 2.3K local builds per week (110K per year) How Netflix Does Builds Workflow / CI / CD

A Day In the Life of an Android Developer at Netflix Commit to Publish ● Make change in a feature branch ● Open a pull request ● Wait for automatic code review ● Wait for peers to review ● Merge to main ● Included in next release train (gated if necessary)

A Day In the Life of an Android Developer at Netflix Commit to Publish ● Make change in a feature branch ● Open a pull request ● Wait for automatic code review ● Wait for peers to review ● Merge to main ● Included in next release train (gated if necessary)

A Day In the Life of an Android Developer at Netflix Commit to Publish ● Make change in a feature branch ● Open a pull request ● Wait for automatic code review ● Wait for peers to review ● Merge to main ● Included in next release train (gated if necessary)

Netflix Android Developer Productivity Wins What is Eng Build and how it helped make devs more productive

./gradlew :NetflixMainApplication:assembleDebug

./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 7m 11s 3512 actionable tasks: 3512 executed, 0 up-to-date

./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 7m 11s 3512 actionable tasks: 3512 executed, 0 up-to-date ./gradlew :NetflixMainApplication:assembleDebug

./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 7m 11s 3512 actionable tasks: 3512 executed, 0 up-to-date ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 11s 3512 actionable tasks: 0 executed, 3512 up-to-date

./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 7m 11s 3512 actionable tasks: 3512 executed, 0 up-to-date ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 2m 1s 3512 actionable tasks: 562 executed, 2950 up-to-date

def buildDate = getBuildDate() android { defaultConfig { buildConfigField "String", "BUILD_DATE", "\"$buildDate\"" } }

public final class BuildConfig { public static final boolean DEBUG = Boolean.parseBoolean("true"); public static final String LIBRARY_PACKAGE_NAME = "com.netflix… public static final String BUILD_TYPE = "debug"; -/ Field from default config. public static final String BUILD_DATE = "02082022-091312"; }

def buildDate = if (engBuild()) "N-A" else getBuildDate() android { defaultConfig { buildConfigField "String", "BUILD_DATE", "\"$buildDate\"" } }

Eng mode ● Optimize incremental compilation ● Remove unnecessary plugins ● Remove variants ● Remove app bundles

First Results of Remote Build Cache for Netflix CI CI build/test time has improved by 46%. Average CI build time is down from 14.6 min to 8 min. The remote cache is saving 6.6 min per CI build on aggregate for all CI builds. EngBuild assembleDebug build time: ~2 min versus assembleDebug build time of ~7 min

Netflix Android Developer Productivity Wins Why Modularize? - More usability - Speed up incremental builds - Improved caching and parallelism

Modularization - Monolithic app

Modularization - First modules

Modularization - First modules ✏ ✏ code change ⚙ need recompilation

Modularization - First modules ⚙ ✏ ⚙ ✏ code change ⚙ need recompilation

Modularization - First modules ✏ ✏ code change ⚙ need recompilation

Modularization - First modules ✏ ⚙ ⚙ ⚙ ⚙ ✏ code change ⚙ need recompilation

Modularization - Api / Impl

Modularization - Api / Impl

Modularization - Api / Impl ✏ ✏ code change ⚙ need recompilation

Modularization - Api / Impl ⚙ ✏ ⚙ ✏ code change ⚙ need recompilation

Modularization - Api / Impl ✏

Modularization - Api / Impl ✏ ⚙ ⚙ ⚙ ⚙ ⚙ ⚙ ⚙ ⚙

Modularization ● Dependency Injection using api and impl modules ● Multiple DI framework available : Dagger, Hilt, Anvil, in house… ● Better architecture, testable … faster incremental build time

./gradlew newModule Modularization - Tooling

./gradlew newModule Module type-: 1: library: 2: ui: Enter selection (default: ui:) [1-.2] Module name? (default: ): droidcon Modularization - Tooling

./gradlew newModule Module type-: 1: library: 2: ui: Enter selection (default: ui:) [1-.2] Module name? (default: ): droidcon BUILD SUCCESSFUL in 1m 35s Modularization - Tooling

How Netflix Does Builds Best Practices Eng Build mode ● Enabled by default ● Optimized for build ● No app bundle ● Less plugins Api & impl modules ● Dependency injection ● Easier testing ● Compilation avoidance ● Tooling

Tools to profile your build Keeping track of our build performances, measuring the impact of changes…

Benchmark the build ● Gradle-profiler ● Define build scenarios ● Run it ● Pro Tip: Config time analysis github.com/gradle/gradle-profiler

Ask the audience: Who has created a build scan?

Grab the lowest hanging DPE fruit: The Build Cache Faster build and test feedback cycles Improved DevProd and developer experience Higher quality software

Build Cache and Binary Repositories ● A build cache is very different and complementary to the concept of a binary repository (sometimes referred to as a dependency cache or artifact repository). ● Whereas a binary repository is for caching binaries that represent dependencies from one source repository to another, a build cache caches build actions, like Gradle tasks or Maven goals. A build cache makes building a single source repository faster. ● A modern and productive development environment should include both binary repositories and build caches because they are complementary. ● Because they share the goal of making the build process more efficient and because certain features overlap, it is not uncommon to assume that both address the same issues.

Ask the audience: do you have local build cache enabled by default?

How it works: Local & Remote Caching Local Cache Remote Cache Build Action (Gradle Task, Maven Goal) PUT PUT READ READ Child Node Child Node Child Node ● When the build action inputs have not changed, the output can be reused from a previous run. ● Build caching can be applied to build actions like compilation, testing, source code generation, Javadoc, checkstyle, and PMD. ● Build caching works out of the box for many standard Gradle tasks and Maven goals. ● For Maven goals or Gradle tasks that are not pre-configured to be cacheable, there is a cache configuration DSL available. That way custom build actions are also cacheable. ● The typical use case is to let CI builds push to a distributed remote build cache, and allow other CI builds as well as developer builds to pull from it. ● A local build cache is also available to reuse outputs produced earlier on the same machine.

Ask the audience: Who is using the remote cache?

How to Setup the remote build cache ● Free (via docker image) ● Gradle Enterprise ● Use build scans for analysis docs.gradle.com/build-cache-node/

Enabling caching of custom tasks docs.gradle.org/current/userguide/build_cache

When we first met “Send me a Build Scan!” scans.gradle.com

No content

Build Caching: Is it enabled? How well is it working?

Build Scan -> Performance Tab

Build Scan -> Build Cache Tab

Build Scan -> Task Execution Tab

Show me all cacheable tasks that were executed Aggregate build scan data from build scans Build caching: who’s using it, who’s not? Debugging cache misses The lowest hanging fruit for speeding CI builds: Remote Build Cache

Build Scan -> Timeline view -> Critical Path Aggregate build scan data from build scans Build caching: who’s using it, who’s not? Debugging cache misses The lowest hanging fruit for speeding CI builds: Remote Build Cache

docs.gradle.org/current/userguide/build_cache_debugging.html Debugging Build Cache Misses with Task Input File Capturing

Build Bottlenecks discovered & Resolved Problem Solution Many cache misses locally due to empty directories remaining in source tree Gradle Doctor Plugin: Removing local directories led to more cache hits. Issues with core AGP tasks were causing cache misses Gradle Cache Fix Plugin: Applying the android-cache-fix-gradle-plugin fixed many of these issues. The Gradle team feeds these improvements back to the AGP team to integrate AGP was generating PNG files for SVG inputs, but the outputs were not reproducible resulting in downstream cache misses that were exposed by GE Since the PNG files were not required for debug, this file generation was removed. assembleDebug execution time went from 3:41 to 0:27. The generateBuildConfig task was configured with a timestamp input that resulted in `BuildConfig.java` being different on each build invocation causing caused downstream changes in compiled outputs, and meant that local developers were not able to use these from the remote cache. Moving this volatility to a leaf module resulted in cache hits for all other modules (but still got a cache miss for that particular leaf module).

First Results of Remote Build Cache for Netflix CI CI build/test time has improved by 46%. Average CI build time is down from 14.6 min to 8 min. The remote cache is saving 6.6 min per CI build on aggregate for all CI builds. ● 24 min of build execution time saved per CI build ● Serial Execution Factor: 3.5 ● Result: 6.6 Min wait time saved per CI build (24.5/3.5)

First Results of Remote Build Cache for Netflix CI The best case for CI is 3 min 18 sec!

Impact of remote caching on PR build times ● Average PR check build time is down from 39 min to 25 min ● Best case ~11 min 25 sec. ● The remote cache is saving on average 14 min per PR build. ● The Netflix Android team ran 251 PR check builds last week.

Impact of remote caching on PR Build Times Best Case is 11 min 25 sec 11 min

One year later…

Cache misses are back ● Different JDK Versions: Identify and fix major differences in JDK between CI/local that were causing cache misses on all compile jobs ● Setting Android Properties: Identify that cache misses were occurring for CompileLibraryResourcesTask which were fixed by enabling certain Android properties ● Older Versions of Dagger: Identify a cacheability issue caused by an older version of Dagger and verify that the upgraded version fixed it ● Non deterministic codegen: Identify code generations with differences between CI/local

Cacheable Task Executed: 835 💥

No content

No content

Java version mismatch caches/transforms-3/3c71c6377bd209bb9c86571321e581b4/transformed/output caches/transforms-3/a4bcee52a5b625e21f22eee7819e81c6/transformed/output

Java version mismatch : 11.0.15 vs 11.0.12 💥 caches/transforms-3/3c71c6377bd209bb9c86571321e581b4/transformed/output caches/transforms-3/a4bcee52a5b625e21f22eee7819e81c6/transformed/output

Use Input Normalization allprojects { normalization { runtimeClasspath { metaInf { ignoreAttribute("Implementation-Version") ignoreAttribute("Created-By") } } } }

Cacheable Task Executed: 24 🎉

After we fixed all the cache misses… PR builds at record speed: best case 5 min Screen shot 5 min

Pop Quiz: what changed in AGP 7.1 that contributed to this speed up?

Lint was made Cacheable in AGP 7.1

Few days after…

Cacheable Task Executed: 515 💥

__Schema.kt - Apollo GraphQL code generation 💥

No content

Cacheable Task Executed: 20 🎉 AssembleDebug time back down to 2 min

No content

Managing dependencies ● Dependencies centralized : Dependencies.kt in buildSrc/

Managing dependencies ● Dependencies centralized : Dependencies.kt in buildSrc/ ● buildSrc/ is part of build script

Managing dependencies ● Dependencies centralized : Dependencies.kt in buildSrc/ ● buildSrc/ is part of build script ● Changes in buildSrc/ break incremental compilation

Gradle Version Catalog ● All dependencies centralized in a file : gradle/libs.versions.toml

Gradle Version Catalog ● All dependencies centralized in a file : gradle/libs.versions.toml ● Available in Gradle 7.4

Gradle Version Catalog ● All dependencies centralized in a file : gradle/libs.versions.toml ● Available in Gradle 7.4 ● Not breaking incremental compilation

Gradle Version Catalog dependencies { implementation libs.my.dep implementation libs.bundles.androidx implementation libs.bundles.compose } my-dep = { module = "group:artifact", version = "version" }

buildSrc/Dependencies.kt gradle/libs.versions.toml

Managing dependencies ● Problems: Transitive dependencies

Managing dependencies ● Problems: Transitive dependencies

Gradle Dependency locking ● From Netflix Nebula, available in Gradle ● Lock all dependencies on a file ● Fail the build when lockfiles are not up-to-date

Gradle dependency locking gradle/libs.versions.toml

Gradle dependency locking .lockfiles/app_gradle.lockfile (generated)

Gradle dependency locking .lockfiles/app_gradle.lockfile (generated)

Gradle dependency locking configurations { compileClasspath { resolutionStrategy.activateDependencyLocking() } } gradle app:dependencies --write-locks

Send me a build scan for analysis… @Rooz via Gradle-Community.Slack.com

No content

Performance Best Practices ⬢ Upgrade to the latest of all tools and plugins: Gradle, AGP, Kotlin ⬢ Tools: Build Scans, Gradle Profiler ⬢ Build Essentials: Daemon, Parallel, Configuration on demand ⬢ Modularize ⬢ L1: Optimize your incremental build cache ⬢ L2: Local Cache ⬢ L3: File Watching, Configuration cache ⬢ Final Boss: Remote Cache ⬢ Enable caching, parallel, file system watching, configuration cache ⬢ Identify and fix performance issues like bottlenecks, cache misses, configuration time slow downs ⬢ Use convention plugins to organize build logic ⬢ Cache Fix plugin ⬢ Gradle Doctor Plugin

No content

We’re hiring https://jobs.netflix.com Android

No content

Thank you!