Netflix + Gradle, A Journey in Developer Productivity

A case study on how Netflix and Gradle have been working together to optimize Netflix Android builds, find bottlenecks, and improve developer productivity. Emmanuel Boudrant from Netflix and Rooz Mohazzabi from Gradle will share wins and findings from their journey.

Transcript

2. Agenda • How Netflix does builds ◦ Gradle / AGP / Android Studio / CI/CD ◦ Eng Mode ◦ Modularisation ◦ Gradle profiler • The Developer Productivity Journey ◦ The first Build Scan™ ◦ Build caching, who’s enabled it and who hasn’t? ◦ Identifying build performance bottlenecks ◦ Enabling local and remote build caching • Build Performance Regressions: One year later ◦ Android SDK and JDK version ◦ CI vs Local cache-miss ◦ Code generation ◦ Dependencies management • Wrap Up
3. How Netflix Does Builds Build Environment & Tools Kapt with Dagger/Hilt, Epoxy, Auto Value App Bundles Jetpack Compose Android Studio / AGP 7.2.1 / Kotlin 1.7.10 Gradle project with ~300 Modules and 10+ variants / flavors
4. SCM : Bitbucket/stash, CI : Jenkins, CD : Spinnaker Pull request with: • Static analysis with ktlint, detekt, lint (custom lint) • Unit testing and on-device testing • 4.6K CI builds per week (220K per year) • 2.3K local builds per week (110K per year) How Netflix Does Builds Workflow / CI / CD
5. A Day In the Life of an Android Developer at Netflix Commit to Publish • Make change in a feature branch • Open a pull request • Wait for automatic code review • Wait for peers to review • Merge to main • Included in next release train (gated if necessary)
6. A Day In the Life of an Android Developer at Netflix Commit to Publish • Make change in a feature branch • Open a pull request • Wait for automatic code review • Wait for peers to review • Merge to main • Included in next release train (gated if necessary)
7. A Day In the Life of an Android Developer at Netflix Commit to Publish • Make change in a feature branch • Open a pull request • Wait for automatic code review • Wait for peers to review • Merge to main • Included in next release train (gated if necessary)
8. Netflix Android Developer Productivity Wins What is Eng Build and how it helped make devs more productive
9. ./gradlew :NetflixMainApplication:assembleDebug
10. ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 7m 11s 3512 actionable tasks: 3512 executed, 0 up-to-date
11. ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 7m 11s 3512 actionable tasks: 3512 executed, 0 up-to-date ./gradlew :NetflixMainApplication:assembleDebug
12. ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 7m 11s 3512 actionable tasks: 3512 executed, 0 up-to-date ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 11s 3512 actionable tasks: 0 executed, 3512 up-to-date
13. ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 7m 11s 3512 actionable tasks: 3512 executed, 0 up-to-date ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 2m 1s 3512 actionable tasks: 562 executed, 2950 up-to-date
14. def buildDate = getBuildDate() android { defaultConfig { buildConfigField "String", "BUILD_DATE", "\"$buildDate\"" } }
15. public final class BuildConfig { public static final boolean DEBUG = Boolean.parseBoolean("true"); public static final String LIBRARY_PACKAGE_NAME = "com.netflix… public static final String BUILD_TYPE = "debug"; -/ Field from default config. public static final String BUILD_DATE = "02082022-091312"; }
16. def buildDate = if (engBuild()) "N-A" else getBuildDate() android { defaultConfig { buildConfigField "String", "BUILD_DATE", "\"$buildDate\"" } }
17. Eng mode • Optimize incremental compilation • Remove unnecessary plugins • Remove variants • Remove app bundles
18. First Results of Remote Build Cache for Netflix CI CI build/test time has improved by 46%. Average CI build time is down from 14.6 min to 8 min. The remote cache is saving 6.6 min per CI build on aggregate for all CI builds. EngBuild assembleDebug build time: ~2 min versus assembleDebug build time of ~7 min
19. Netflix Android Developer Productivity Wins Why Modularize? - More usability - Speed up incremental builds - Improved caching and parallelism
20. Modularization - Monolithic app
21. Modularization - First modules
22. Modularization - First modules ✏ ✏ code change ⚙ need recompilation
23. Modularization - First modules ⚙ ✏ ⚙ ✏ code change ⚙ need recompilation
24. Modularization - First modules ✏ ✏ code change ⚙ need recompilation
25. Modularization - First modules ✏ ⚙ ⚙ ⚙ ⚙ ✏ code change ⚙ need recompilation
26. Modularization - Api / Impl
27. Modularization - Api / Impl
28. Modularization - Api / Impl ✏ ✏ code change ⚙ need recompilation
29. Modularization - Api / Impl ⚙ ✏ ⚙ ✏ code change ⚙ need recompilation
30. Modularization - Api / Impl ✏
31. Modularization - Api / Impl ✏ ⚙ ⚙ ⚙ ⚙ ⚙ ⚙ ⚙ ⚙
32. Modularization • Dependency Injection using api and impl modules • Multiple DI framework available : Dagger, Hilt, Anvil, in house… • Better architecture, testable … faster incremental build time
33. ./gradlew newModule Modularization - Tooling
34. ./gradlew newModule Module type-: 1: library: 2: ui: Enter selection (default: ui:) [1-.2] Module name? (default: ): droidcon Modularization - Tooling
35. ./gradlew newModule Module type-: 1: library: 2: ui: Enter selection (default: ui:) [1-.2] Module name? (default: ): droidcon BUILD SUCCESSFUL in 1m 35s Modularization - Tooling
36. How Netflix Does Builds Best Practices Eng Build mode • Enabled by default • Optimized for build • No app bundle • Less plugins Api & impl modules • Dependency injection • Easier testing • Compilation avoidance • Tooling
37. Tools to profile your build Keeping track of our build performances, measuring the impact of changes…
38. Benchmark the build • Gradle-profiler • Define build scenarios • Run it • Pro Tip: Config time analysis github.com/gradle/gradle-profiler
39. Ask the audience: Who has created a build scan?
40. Grab the lowest hanging DPE fruit: The Build Cache Faster build and test feedback cycles Improved DevProd and developer experience Higher quality software
41. Build Cache and Binary Repositories • A build cache is very different and complementary to the concept of a binary repository (sometimes referred to as a dependency cache or artifact repository). • Whereas a binary repository is for caching binaries that represent dependencies from one source repository to another, a build cache caches build actions, like Gradle tasks or Maven goals. A build cache makes building a single source repository faster. • A modern and productive development environment should include both binary repositories and build caches because they are complementary. • Because they share the goal of making the build process more efficient and because certain features overlap, it is not uncommon to assume that both address the same issues.
42. Ask the audience: do you have local build cache enabled by default?
43. How it works: Local & Remote Caching Local Cache Remote Cache Build Action (Gradle Task, Maven Goal) PUT PUT READ READ Child Node Child Node Child Node • When the build action inputs have not changed, the output can be reused from a previous run. • Build caching can be applied to build actions like compilation, testing, source code generation, Javadoc, checkstyle, and PMD. • Build caching works out of the box for many standard Gradle tasks and Maven goals. • For Maven goals or Gradle tasks that are not pre-configured to be cacheable, there is a cache configuration DSL available. That way custom build actions are also cacheable. • The typical use case is to let CI builds push to a distributed remote build cache, and allow other CI builds as well as developer builds to pull from it. • A local build cache is also available to reuse outputs produced earlier on the same machine.
44. Ask the audience: Who is using the remote cache?
45. How to Setup the remote build cache • Free (via docker image) • Gradle Enterprise • Use build scans for analysis docs.gradle.com/build-cache-node/
46. Enabling caching of custom tasks docs.gradle.org/current/userguide/build_cache
47. When we first met “Send me a Build Scan!” scans.gradle.com
49. Build Caching: Is it enabled? How well is it working?
50. Build Scan -> Performance Tab
51. Build Scan -> Build Cache Tab
52. Build Scan -> Task Execution Tab
53. Show me all cacheable tasks that were executed Aggregate build scan data from build scans Build caching: who’s using it, who’s not? Debugging cache misses The lowest hanging fruit for speeding CI builds: Remote Build Cache
54. Build Scan -> Timeline view -> Critical Path Aggregate build scan data from build scans Build caching: who’s using it, who’s not? Debugging cache misses The lowest hanging fruit for speeding CI builds: Remote Build Cache
55. docs.gradle.org/current/userguide/build_cache_debugging.html Debugging Build Cache Misses with Task Input File Capturing
56. Build Bottlenecks discovered & Resolved Problem Solution Many cache misses locally due to empty directories remaining in source tree Gradle Doctor Plugin: Removing local directories led to more cache hits. Issues with core AGP tasks were causing cache misses Gradle Cache Fix Plugin: Applying the android-cache-fix-gradle-plugin fixed many of these issues. The Gradle team feeds these improvements back to the AGP team to integrate AGP was generating PNG files for SVG inputs, but the outputs were not reproducible resulting in downstream cache misses that were exposed by GE Since the PNG files were not required for debug, this file generation was removed. assembleDebug execution time went from 3:41 to 0:27. The generateBuildConfig task was configured with a timestamp input that resulted in `BuildConfig.java` being different on each build invocation causing caused downstream changes in compiled outputs, and meant that local developers were not able to use these from the remote cache. Moving this volatility to a leaf module resulted in cache hits for all other modules (but still got a cache miss for that particular leaf module).
57. First Results of Remote Build Cache for Netflix CI CI build/test time has improved by 46%. Average CI build time is down from 14.6 min to 8 min. The remote cache is saving 6.6 min per CI build on aggregate for all CI builds. • 24 min of build execution time saved per CI build • Serial Execution Factor: 3.5 • Result: 6.6 Min wait time saved per CI build (24.5/3.5)
58. First Results of Remote Build Cache for Netflix CI The best case for CI is 3 min 18 sec!
59. Impact of remote caching on PR build times • Average PR check build time is down from 39 min to 25 min • Best case ~11 min 25 sec. • The remote cache is saving on average 14 min per PR build. • The Netflix Android team ran 251 PR check builds last week.
60. Impact of remote caching on PR Build Times Best Case is 11 min 25 sec 11 min
61. One year later…
62. Cache misses are back • Different JDK Versions: Identify and fix major differences in JDK between CI/local that were causing cache misses on all compile jobs • Setting Android Properties: Identify that cache misses were occurring for CompileLibraryResourcesTask which were fixed by enabling certain Android properties • Older Versions of Dagger: Identify a cacheability issue caused by an older version of Dagger and verify that the upgraded version fixed it • Non deterministic codegen: Identify code generations with differences between CI/local
63. Cacheable Task Executed: 835 💥
66. Java version mismatch caches/transforms-3/3c71c6377bd209bb9c86571321e581b4/transformed/output caches/transforms-3/a4bcee52a5b625e21f22eee7819e81c6/transformed/output
67. Java version mismatch : 11.0.15 vs 11.0.12 💥 caches/transforms-3/3c71c6377bd209bb9c86571321e581b4/transformed/output caches/transforms-3/a4bcee52a5b625e21f22eee7819e81c6/transformed/output
68. Use Input Normalization allprojects { normalization { runtimeClasspath { metaInf { ignoreAttribute("Implementation-Version") ignoreAttribute("Created-By") } } } }
69. Cacheable Task Executed: 24 🎉
70. After we fixed all the cache misses… PR builds at record speed: best case 5 min Screen shot 5 min
71. Pop Quiz: what changed in AGP 7.1 that contributed to this speed up?
72. Lint was made Cacheable in AGP 7.1
73. Few days after…
74. Cacheable Task Executed: 515 💥
75. __Schema.kt - Apollo GraphQL code generation 💥
77. Cacheable Task Executed: 20 🎉 AssembleDebug time back down to 2 min
79. Managing dependencies • Dependencies centralized : Dependencies.kt in buildSrc/
80. Managing dependencies • Dependencies centralized : Dependencies.kt in buildSrc/ • buildSrc/ is part of build script
81. Managing dependencies • Dependencies centralized : Dependencies.kt in buildSrc/ • buildSrc/ is part of build script • Changes in buildSrc/ break incremental compilation
82. Gradle Version Catalog • All dependencies centralized in a file : gradle/libs.versions.toml
83. Gradle Version Catalog • All dependencies centralized in a file : gradle/libs.versions.toml • Available in Gradle 7.4
84. Gradle Version Catalog • All dependencies centralized in a file : gradle/libs.versions.toml • Available in Gradle 7.4 • Not breaking incremental compilation
85. Gradle Version Catalog dependencies { implementation libs.my.dep implementation libs.bundles.androidx implementation libs.bundles.compose } my-dep = { module = "group:artifact", version = "version" }
86. buildSrc/Dependencies.kt gradle/libs.versions.toml
87. Managing dependencies • Problems: Transitive dependencies
88. Managing dependencies • Problems: Transitive dependencies
89. Gradle Dependency locking • From Netflix Nebula, available in Gradle • Lock all dependencies on a file • Fail the build when lockfiles are not up-to-date
90. Gradle dependency locking gradle/libs.versions.toml
91. Gradle dependency locking .lockfiles/app_gradle.lockfile (generated)
92. Gradle dependency locking .lockfiles/app_gradle.lockfile (generated)
93. Gradle dependency locking configurations { compileClasspath { resolutionStrategy.activateDependencyLocking() } } gradle app:dependencies --write-locks
94. Send me a build scan for analysis… @Rooz via Gradle-Community.Slack.com
96. Performance Best Practices ⬢ Upgrade to the latest of all tools and plugins: Gradle, AGP, Kotlin ⬢ Tools: Build Scans, Gradle Profiler ⬢ Build Essentials: Daemon, Parallel, Configuration on demand ⬢ Modularize ⬢ L1: Optimize your incremental build cache ⬢ L2: Local Cache ⬢ L3: File Watching, Configuration cache ⬢ Final Boss: Remote Cache ⬢ Enable caching, parallel, file system watching, configuration cache ⬢ Identify and fix performance issues like bottlenecks, cache misses, configuration time slow downs ⬢ Use convention plugins to organize build logic ⬢ Cache Fix plugin ⬢ Gradle Doctor Plugin
98. We’re hiring https://jobs.netflix.com Android
100. Thank you!