Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Netflix + Gradle, A Journey in Developer Productivity

Netflix + Gradle, A Journey in Developer Productivity

A case study on how Netflix and Gradle have been working together to optimize Netflix Android builds, find bottlenecks, and improve developer productivity. Emmanuel Boudrant from Netflix and Rooz Mohazzabi from Gradle will share wins and findings from their journey.

Emmanuel Boudrant

September 02, 2022
Tweet

Other Decks in Programming

Transcript

  1. None
  2. Agenda • How Netflix does builds ◦ Gradle / AGP

    / Android Studio / CI/CD ◦ Eng Mode ◦ Modularisation ◦ Gradle profiler • The Developer Productivity Journey ◦ The first Build Scan™ ◦ Build caching, who’s enabled it and who hasn’t? ◦ Identifying build performance bottlenecks ◦ Enabling local and remote build caching • Build Performance Regressions: One year later ◦ Android SDK and JDK version ◦ CI vs Local cache-miss ◦ Code generation ◦ Dependencies management • Wrap Up
  3. How Netflix Does Builds Build Environment & Tools Kapt with

    Dagger/Hilt, Epoxy, Auto Value App Bundles Jetpack Compose Android Studio / AGP 7.2.1 / Kotlin 1.7.10 Gradle project with ~300 Modules and 10+ variants / flavors
  4. SCM : Bitbucket/stash, CI : Jenkins, CD : Spinnaker Pull

    request with: • Static analysis with ktlint, detekt, lint (custom lint) • Unit testing and on-device testing • 4.6K CI builds per week (220K per year) • 2.3K local builds per week (110K per year) How Netflix Does Builds Workflow / CI / CD
  5. A Day In the Life of an Android Developer at

    Netflix Commit to Publish • Make change in a feature branch • Open a pull request • Wait for automatic code review • Wait for peers to review • Merge to main • Included in next release train (gated if necessary)
  6. A Day In the Life of an Android Developer at

    Netflix Commit to Publish • Make change in a feature branch • Open a pull request • Wait for automatic code review • Wait for peers to review • Merge to main • Included in next release train (gated if necessary)
  7. A Day In the Life of an Android Developer at

    Netflix Commit to Publish • Make change in a feature branch • Open a pull request • Wait for automatic code review • Wait for peers to review • Merge to main • Included in next release train (gated if necessary)
  8. Netflix Android Developer Productivity Wins What is Eng Build and

    how it helped make devs more productive
  9. ./gradlew :NetflixMainApplication:assembleDebug

  10. ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 7m 11s 3512 actionable tasks:

    3512 executed, 0 up-to-date
  11. ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 7m 11s 3512 actionable tasks:

    3512 executed, 0 up-to-date ./gradlew :NetflixMainApplication:assembleDebug
  12. ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 7m 11s 3512 actionable tasks:

    3512 executed, 0 up-to-date ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 11s 3512 actionable tasks: 0 executed, 3512 up-to-date
  13. ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 7m 11s 3512 actionable tasks:

    3512 executed, 0 up-to-date ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 2m 1s 3512 actionable tasks: 562 executed, 2950 up-to-date
  14. def buildDate = getBuildDate() android { defaultConfig { buildConfigField "String",

    "BUILD_DATE", "\"$buildDate\"" } }
  15. public final class BuildConfig { public static final boolean DEBUG

    = Boolean.parseBoolean("true"); public static final String LIBRARY_PACKAGE_NAME = "com.netflix… public static final String BUILD_TYPE = "debug"; -/ Field from default config. public static final String BUILD_DATE = "02082022-091312"; }
  16. def buildDate = if (engBuild()) "N-A" else getBuildDate() android {

    defaultConfig { buildConfigField "String", "BUILD_DATE", "\"$buildDate\"" } }
  17. Eng mode • Optimize incremental compilation • Remove unnecessary plugins

    • Remove variants • Remove app bundles
  18. First Results of Remote Build Cache for Netflix CI CI

    build/test time has improved by 46%. Average CI build time is down from 14.6 min to 8 min. The remote cache is saving 6.6 min per CI build on aggregate for all CI builds. EngBuild assembleDebug build time: ~2 min versus assembleDebug build time of ~7 min
  19. Netflix Android Developer Productivity Wins Why Modularize? - More usability

    - Speed up incremental builds - Improved caching and parallelism
  20. Modularization - Monolithic app

  21. Modularization - First modules

  22. Modularization - First modules ✏ ✏ code change ⚙ need

    recompilation
  23. Modularization - First modules ⚙ ✏ ⚙ ✏ code change

    ⚙ need recompilation
  24. Modularization - First modules ✏ ✏ code change ⚙ need

    recompilation
  25. Modularization - First modules ✏ ⚙ ⚙ ⚙ ⚙ ✏

    code change ⚙ need recompilation
  26. Modularization - Api / Impl

  27. Modularization - Api / Impl

  28. Modularization - Api / Impl ✏ ✏ code change ⚙

    need recompilation
  29. Modularization - Api / Impl ⚙ ✏ ⚙ ✏ code

    change ⚙ need recompilation
  30. Modularization - Api / Impl ✏

  31. Modularization - Api / Impl ✏ ⚙ ⚙ ⚙ ⚙

    ⚙ ⚙ ⚙ ⚙
  32. Modularization • Dependency Injection using api and impl modules •

    Multiple DI framework available : Dagger, Hilt, Anvil, in house… • Better architecture, testable … faster incremental build time
  33. ./gradlew newModule Modularization - Tooling

  34. ./gradlew newModule Module type-: 1: library: 2: ui: Enter selection

    (default: ui:) [1-.2] Module name? (default: ): droidcon Modularization - Tooling
  35. ./gradlew newModule Module type-: 1: library: 2: ui: Enter selection

    (default: ui:) [1-.2] Module name? (default: ): droidcon BUILD SUCCESSFUL in 1m 35s Modularization - Tooling
  36. How Netflix Does Builds Best Practices Eng Build mode •

    Enabled by default • Optimized for build • No app bundle • Less plugins Api & impl modules • Dependency injection • Easier testing • Compilation avoidance • Tooling
  37. Tools to profile your build Keeping track of our build

    performances, measuring the impact of changes…
  38. Benchmark the build • Gradle-profiler • Define build scenarios •

    Run it • Pro Tip: Config time analysis github.com/gradle/gradle-profiler
  39. Ask the audience: Who has created a build scan?

  40. Grab the lowest hanging DPE fruit: The Build Cache Faster

    build and test feedback cycles Improved DevProd and developer experience Higher quality software
  41. Build Cache and Binary Repositories • A build cache is

    very different and complementary to the concept of a binary repository (sometimes referred to as a dependency cache or artifact repository). • Whereas a binary repository is for caching binaries that represent dependencies from one source repository to another, a build cache caches build actions, like Gradle tasks or Maven goals. A build cache makes building a single source repository faster. • A modern and productive development environment should include both binary repositories and build caches because they are complementary. • Because they share the goal of making the build process more efficient and because certain features overlap, it is not uncommon to assume that both address the same issues.
  42. Ask the audience: do you have local build cache enabled

    by default?
  43. How it works: Local & Remote Caching Local Cache Remote

    Cache Build Action (Gradle Task, Maven Goal) PUT PUT READ READ Child Node Child Node Child Node • When the build action inputs have not changed, the output can be reused from a previous run. • Build caching can be applied to build actions like compilation, testing, source code generation, Javadoc, checkstyle, and PMD. • Build caching works out of the box for many standard Gradle tasks and Maven goals. • For Maven goals or Gradle tasks that are not pre-configured to be cacheable, there is a cache configuration DSL available. That way custom build actions are also cacheable. • The typical use case is to let CI builds push to a distributed remote build cache, and allow other CI builds as well as developer builds to pull from it. • A local build cache is also available to reuse outputs produced earlier on the same machine.
  44. Ask the audience: Who is using the remote cache?

  45. How to Setup the remote build cache • Free (via

    docker image) • Gradle Enterprise • Use build scans for analysis docs.gradle.com/build-cache-node/
  46. Enabling caching of custom tasks docs.gradle.org/current/userguide/build_cache

  47. When we first met “Send me a Build Scan!” scans.gradle.com

  48. None
  49. Build Caching: Is it enabled? How well is it working?

  50. Build Scan -> Performance Tab

  51. Build Scan -> Build Cache Tab

  52. Build Scan -> Task Execution Tab

  53. Show me all cacheable tasks that were executed Aggregate build

    scan data from build scans Build caching: who’s using it, who’s not? Debugging cache misses The lowest hanging fruit for speeding CI builds: Remote Build Cache
  54. Build Scan -> Timeline view -> Critical Path Aggregate build

    scan data from build scans Build caching: who’s using it, who’s not? Debugging cache misses The lowest hanging fruit for speeding CI builds: Remote Build Cache
  55. docs.gradle.org/current/userguide/build_cache_debugging.html Debugging Build Cache Misses with Task Input File Capturing

  56. Build Bottlenecks discovered & Resolved Problem Solution Many cache misses

    locally due to empty directories remaining in source tree Gradle Doctor Plugin: Removing local directories led to more cache hits. Issues with core AGP tasks were causing cache misses Gradle Cache Fix Plugin: Applying the android-cache-fix-gradle-plugin fixed many of these issues. The Gradle team feeds these improvements back to the AGP team to integrate AGP was generating PNG files for SVG inputs, but the outputs were not reproducible resulting in downstream cache misses that were exposed by GE Since the PNG files were not required for debug, this file generation was removed. assembleDebug execution time went from 3:41 to 0:27. The generateBuildConfig task was configured with a timestamp input that resulted in `BuildConfig.java` being different on each build invocation causing caused downstream changes in compiled outputs, and meant that local developers were not able to use these from the remote cache. Moving this volatility to a leaf module resulted in cache hits for all other modules (but still got a cache miss for that particular leaf module).
  57. First Results of Remote Build Cache for Netflix CI CI

    build/test time has improved by 46%. Average CI build time is down from 14.6 min to 8 min. The remote cache is saving 6.6 min per CI build on aggregate for all CI builds. • 24 min of build execution time saved per CI build • Serial Execution Factor: 3.5 • Result: 6.6 Min wait time saved per CI build (24.5/3.5)
  58. First Results of Remote Build Cache for Netflix CI The

    best case for CI is 3 min 18 sec!
  59. Impact of remote caching on PR build times • Average

    PR check build time is down from 39 min to 25 min • Best case ~11 min 25 sec. • The remote cache is saving on average 14 min per PR build. • The Netflix Android team ran 251 PR check builds last week.
  60. Impact of remote caching on PR Build Times Best Case

    is 11 min 25 sec 11 min
  61. One year later…

  62. Cache misses are back • Different JDK Versions: Identify and

    fix major differences in JDK between CI/local that were causing cache misses on all compile jobs • Setting Android Properties: Identify that cache misses were occurring for CompileLibraryResourcesTask which were fixed by enabling certain Android properties • Older Versions of Dagger: Identify a cacheability issue caused by an older version of Dagger and verify that the upgraded version fixed it • Non deterministic codegen: Identify code generations with differences between CI/local
  63. Cacheable Task Executed: 835 💥

  64. None
  65. None
  66. Java version mismatch caches/transforms-3/3c71c6377bd209bb9c86571321e581b4/transformed/output caches/transforms-3/a4bcee52a5b625e21f22eee7819e81c6/transformed/output

  67. Java version mismatch : 11.0.15 vs 11.0.12 💥 caches/transforms-3/3c71c6377bd209bb9c86571321e581b4/transformed/output caches/transforms-3/a4bcee52a5b625e21f22eee7819e81c6/transformed/output

  68. Use Input Normalization allprojects { normalization { runtimeClasspath { metaInf

    { ignoreAttribute("Implementation-Version") ignoreAttribute("Created-By") } } } }
  69. Cacheable Task Executed: 24 🎉

  70. After we fixed all the cache misses… PR builds at

    record speed: best case 5 min Screen shot 5 min
  71. Pop Quiz: what changed in AGP 7.1 that contributed to

    this speed up?
  72. Lint was made Cacheable in AGP 7.1

  73. Few days after…

  74. Cacheable Task Executed: 515 💥

  75. __Schema.kt - Apollo GraphQL code generation 💥

  76. None
  77. Cacheable Task Executed: 20 🎉 AssembleDebug time back down to

    2 min
  78. None
  79. Managing dependencies • Dependencies centralized : Dependencies.kt in buildSrc/

  80. Managing dependencies • Dependencies centralized : Dependencies.kt in buildSrc/ •

    buildSrc/ is part of build script
  81. Managing dependencies • Dependencies centralized : Dependencies.kt in buildSrc/ •

    buildSrc/ is part of build script • Changes in buildSrc/ break incremental compilation
  82. Gradle Version Catalog • All dependencies centralized in a file

    : gradle/libs.versions.toml
  83. Gradle Version Catalog • All dependencies centralized in a file

    : gradle/libs.versions.toml • Available in Gradle 7.4
  84. Gradle Version Catalog • All dependencies centralized in a file

    : gradle/libs.versions.toml • Available in Gradle 7.4 • Not breaking incremental compilation
  85. Gradle Version Catalog dependencies { implementation libs.my.dep implementation libs.bundles.androidx implementation

    libs.bundles.compose } my-dep = { module = "group:artifact", version = "version" }
  86. buildSrc/Dependencies.kt gradle/libs.versions.toml

  87. Managing dependencies • Problems: Transitive dependencies

  88. Managing dependencies • Problems: Transitive dependencies

  89. Gradle Dependency locking • From Netflix Nebula, available in Gradle

    • Lock all dependencies on a file • Fail the build when lockfiles are not up-to-date
  90. Gradle dependency locking gradle/libs.versions.toml

  91. Gradle dependency locking .lockfiles/app_gradle.lockfile (generated)

  92. Gradle dependency locking .lockfiles/app_gradle.lockfile (generated)

  93. Gradle dependency locking configurations { compileClasspath { resolutionStrategy.activateDependencyLocking() } }

    gradle app:dependencies --write-locks
  94. Send me a build scan for analysis… @Rooz via Gradle-Community.Slack.com

  95. None
  96. Performance Best Practices ⬢ Upgrade to the latest of all

    tools and plugins: Gradle, AGP, Kotlin ⬢ Tools: Build Scans, Gradle Profiler ⬢ Build Essentials: Daemon, Parallel, Configuration on demand ⬢ Modularize ⬢ L1: Optimize your incremental build cache ⬢ L2: Local Cache ⬢ L3: File Watching, Configuration cache ⬢ Final Boss: Remote Cache ⬢ Enable caching, parallel, file system watching, configuration cache ⬢ Identify and fix performance issues like bottlenecks, cache misses, configuration time slow downs ⬢ Use convention plugins to organize build logic ⬢ Cache Fix plugin ⬢ Gradle Doctor Plugin
  97. None
  98. We’re hiring https://jobs.netflix.com Android

  99. None
  100. Thank you!