Netflix + Gradle, A Journey in Developer Productivity

A case study on how Netflix and Gradle have been working together to optimize Netflix Android builds, find bottlenecks, and improve developer productivity. Emmanuel Boudrant from Netflix and Rooz Mohazzabi from Gradle will share wins and findings from their journey.

Transcript

1. View Slide

2. Agenda
   ● How Netflix does builds
   ○ Gradle / AGP / Android Studio / CI/CD
   ○ Eng Mode
   ○ Modularisation
   ○ Gradle profiler
   ● The Developer Productivity Journey
   ○ The first Build Scan™
   ○ Build caching, who’s enabled it and who hasn’t?
   ○ Identifying build performance bottlenecks
   ○ Enabling local and remote build caching
   ● Build Performance Regressions: One year later
   ○ Android SDK and JDK version
   ○ CI vs Local cache-miss
   ○ Code generation
   ○ Dependencies management
   ● Wrap Up
   

   View Slide

3. How Netflix Does Builds
   Build Environment & Tools
   Kapt with Dagger/Hilt,
   Epoxy, Auto Value
   App Bundles
   Jetpack Compose
   Android Studio / AGP
   7.2.1 / Kotlin 1.7.10
   Gradle project with
   ~300 Modules and
   10+ variants /
   flavors
   

   View Slide

4. SCM : Bitbucket/stash, CI : Jenkins, CD : Spinnaker
   Pull request with:
   ● Static analysis with ktlint, detekt, lint (custom lint)
   ● Unit testing and on-device testing
   ● 4.6K CI builds per week (220K per year)
   ● 2.3K local builds per week (110K per year)
   How Netflix Does Builds
   Workflow / CI / CD
   

   View Slide

5. A Day In the Life of an Android Developer at Netflix
   Commit to Publish
   ● Make change in a feature branch
   ● Open a pull request
   ● Wait for automatic code review
   ● Wait for peers to review
   ● Merge to main
   ● Included in next release train (gated if necessary)
   

   View Slide

6. A Day In the Life of an Android Developer at Netflix
   Commit to Publish
   ● Make change in a feature branch
   ● Open a pull request
   ● Wait for automatic code review
   ● Wait for peers to review
   ● Merge to main
   ● Included in next release train (gated if necessary)
   

   View Slide

7. A Day In the Life of an Android Developer at Netflix
   Commit to Publish
   ● Make change in a feature branch
   ● Open a pull request
   ● Wait for automatic code review
   ● Wait for peers to review
   ● Merge to main
   ● Included in next release train (gated if necessary)
   

   View Slide

8. Netflix Android Developer Productivity Wins
   What is Eng Build and how it helped make devs
   more productive
   

   View Slide

9. ./gradlew :NetflixMainApplication:assembleDebug
   

   View Slide

10. ./gradlew :NetflixMainApplication:assembleDebug
    BUILD SUCCESSFUL in 7m 11s
    3512 actionable tasks: 3512 executed, 0 up-to-date
    

    View Slide

11. ./gradlew :NetflixMainApplication:assembleDebug
    BUILD SUCCESSFUL in 7m 11s
    3512 actionable tasks: 3512 executed, 0 up-to-date
    ./gradlew :NetflixMainApplication:assembleDebug
    

    View Slide

12. ./gradlew :NetflixMainApplication:assembleDebug
    BUILD SUCCESSFUL in 7m 11s
    3512 actionable tasks: 3512 executed, 0 up-to-date
    ./gradlew :NetflixMainApplication:assembleDebug
    BUILD SUCCESSFUL in 11s
    3512 actionable tasks: 0 executed, 3512 up-to-date
    

    View Slide

13. ./gradlew :NetflixMainApplication:assembleDebug
    BUILD SUCCESSFUL in 7m 11s
    3512 actionable tasks: 3512 executed, 0 up-to-date
    ./gradlew :NetflixMainApplication:assembleDebug
    BUILD SUCCESSFUL in 2m 1s
    3512 actionable tasks: 562 executed, 2950 up-to-date
    

    View Slide

14. def buildDate = getBuildDate()
    android {
    defaultConfig {
    buildConfigField "String", "BUILD_DATE", "\"$buildDate\""
    }
    }
    

    View Slide

15. public final class BuildConfig {
    public static final boolean DEBUG = Boolean.parseBoolean("true");
    public static final String LIBRARY_PACKAGE_NAME = "com.netflix…
    public static final String BUILD_TYPE = "debug";
    -/ Field from default config.
    public static final String BUILD_DATE = "02082022-091312";
    }
    

    View Slide

16. def buildDate = if (engBuild()) "N-A" else getBuildDate()
    android {
    defaultConfig {
    buildConfigField "String", "BUILD_DATE", "\"$buildDate\""
    }
    }
    

    View Slide

17. Eng mode
    ● Optimize incremental compilation
    ● Remove unnecessary plugins
    ● Remove variants
    ● Remove app bundles
    

    View Slide

18. First Results of Remote Build Cache for Netflix CI
    CI build/test time has improved by 46%. Average CI build time is down from 14.6 min to 8 min. The remote
    cache is saving 6.6 min per CI build on aggregate for all CI builds.
    EngBuild assembleDebug build time: ~2 min
    versus assembleDebug build time of ~7 min
    

    View Slide

19. Netflix Android Developer Productivity Wins
    Why Modularize?
    - More usability
    - Speed up incremental builds
    - Improved caching and parallelism
    

    View Slide

20. Modularization - Monolithic app
    

    View Slide

21. Modularization - First modules
    

    View Slide

22. Modularization - First modules
    ✏
    ✏ code change
    ⚙ need recompilation
    

    View Slide

23. Modularization - First modules
    ⚙
    ✏
    ⚙
    ✏ code change
    ⚙ need recompilation
    

    View Slide

24. Modularization - First modules
    ✏
    ✏ code change
    ⚙ need recompilation
    

    View Slide

25. Modularization - First modules
    ✏
    ⚙
    ⚙
    ⚙ ⚙
    ✏ code change
    ⚙ need recompilation
    

    View Slide

26. Modularization - Api / Impl
    

    View Slide

27. Modularization - Api / Impl
    

    View Slide

28. Modularization - Api / Impl
    ✏
    ✏ code change
    ⚙ need recompilation
    

    View Slide

29. Modularization - Api / Impl
    ⚙
    ✏
    ⚙
    ✏ code change
    ⚙ need recompilation
    

    View Slide

30. Modularization - Api / Impl
    ✏
    

    View Slide

31. Modularization - Api / Impl
    ✏
    ⚙
    ⚙
    ⚙
    ⚙ ⚙
    ⚙
    ⚙
    ⚙
    

    View Slide

32. Modularization
    ● Dependency Injection using api and impl modules
    ● Multiple DI framework available : Dagger, Hilt, Anvil, in house…
    ● Better architecture, testable … faster incremental build time
    

    View Slide

33. ./gradlew newModule
    Modularization - Tooling
    

    View Slide

34. ./gradlew newModule
    Module type-:
    1: library:
    2: ui:
    Enter selection (default: ui:) [1-.2]
    Module name? (default: ): droidcon
    Modularization - Tooling
    

    View Slide

35. ./gradlew newModule
    Module type-:
    1: library:
    2: ui:
    Enter selection (default: ui:) [1-.2]
    Module name? (default: ): droidcon
    BUILD SUCCESSFUL in 1m 35s
    Modularization - Tooling
    

    View Slide

36. How Netflix Does Builds
    Best Practices
    Eng Build mode
    ● Enabled by default
    ● Optimized for build
    ● No app bundle
    ● Less plugins
    Api & impl modules
    ● Dependency injection
    ● Easier testing
    ● Compilation avoidance
    ● Tooling
    

    View Slide

37. Tools to profile your build
    Keeping track of our build performances,
    measuring the impact of changes…
    

    View Slide

38. Benchmark the build
    ● Gradle-profiler
    ● Define build scenarios
    ● Run it
    ● Pro Tip: Config time analysis
    github.com/gradle/gradle-profiler
    

    View Slide

39. Ask the audience: Who has created a build scan?
    

    View Slide

40. Grab the lowest hanging DPE fruit: The Build Cache
    Faster build and test
    feedback cycles
    Improved DevProd and
    developer experience
    Higher quality
    software
    

    View Slide

41. Build Cache and Binary Repositories
    ● A build cache is very different and complementary to the concept of
    a binary repository (sometimes referred to as a dependency cache
    or artifact repository).
    ● Whereas a binary repository is for caching binaries that represent
    dependencies from one source repository to another, a build cache
    caches build actions, like Gradle tasks or Maven goals. A build cache
    makes building a single source repository faster.
    ● A modern and productive development environment should include
    both binary repositories and build caches because they are
    complementary.
    ● Because they share the goal of making the build process more
    efficient and because certain features overlap, it is not uncommon
    to assume that both address the same issues.
    

    View Slide

42. Ask the audience:
    do you have local build cache enabled by default?
    

    View Slide

43. How it works: Local & Remote Caching
    Local Cache
    Remote
    Cache
    Build Action
    (Gradle Task, Maven Goal)
    PUT PUT
    READ READ
    Child Node
    Child Node
    Child Node
    ● When the build action inputs have not changed,
    the output can be reused from a previous run.
    ● Build caching can be applied to build actions
    like compilation, testing, source code
    generation, Javadoc, checkstyle, and PMD.
    ● Build caching works out of the box for many
    standard Gradle tasks and Maven goals.
    ● For Maven goals or Gradle tasks that are not
    pre-configured to be cacheable, there is a cache
    configuration DSL available. That way custom
    build actions are also cacheable.
    ● The typical use case is to let CI builds push to a
    distributed remote build cache, and allow other
    CI builds as well as developer builds to pull from
    it.
    ● A local build cache is also available to reuse
    outputs produced earlier on the same machine.
    

    View Slide

44. Ask the audience: Who is using the remote cache?
    

    View Slide

45. How to Setup the remote build cache
    ● Free (via docker image)
    ● Gradle Enterprise
    ● Use build scans for analysis
    docs.gradle.com/build-cache-node/
    

    View Slide

46. Enabling caching of custom tasks
    docs.gradle.org/current/userguide/build_cache
    

    View Slide

47. When we first met
    “Send me a Build Scan!”
    scans.gradle.com
    

    View Slide

48. View Slide

49. Build Caching: Is it enabled? How well is it working?
    

    View Slide

50. Build Scan -> Performance Tab
    

    View Slide

51. Build Scan -> Build Cache Tab
    

    View Slide

52. Build Scan -> Task Execution Tab
    

    View Slide

53. Show me all cacheable tasks that were executed
    Aggregate build scan data from build scans
    Build caching: who’s using it, who’s not?
    Debugging cache misses
    The lowest hanging fruit for speeding CI builds: Remote Build Cache
    

    View Slide

54. Build Scan -> Timeline view -> Critical Path
    Aggregate build scan data from build scans
    Build caching: who’s using it, who’s not?
    Debugging cache misses
    The lowest hanging fruit for speeding CI builds: Remote Build Cache
    

    View Slide

55. docs.gradle.org/current/userguide/build_cache_debugging.html
    Debugging Build Cache Misses with Task Input File Capturing
    

    View Slide

56. Build Bottlenecks discovered & Resolved
    Problem Solution
    Many cache misses locally due to empty directories remaining in
    source tree
    Gradle Doctor Plugin: Removing local directories led to
    more cache hits.
    Issues with core AGP tasks were causing cache misses Gradle Cache Fix Plugin: Applying the
    android-cache-fix-gradle-plugin fixed many of these issues.
    The Gradle team feeds these improvements back to the
    AGP team to integrate
    AGP was generating PNG files for SVG inputs, but the outputs
    were not reproducible resulting in downstream cache misses that
    were exposed by GE
    Since the PNG files were not required for debug, this file
    generation was removed. assembleDebug execution time
    went from 3:41 to 0:27.
    The generateBuildConfig task was configured with a timestamp
    input that resulted in `BuildConfig.java` being different on each
    build invocation causing caused downstream changes in compiled
    outputs, and meant that local developers were not able to use
    these from the remote cache.
    Moving this volatility to a leaf module resulted in cache hits
    for all other modules (but still got a cache miss for that
    particular leaf module).
    

    View Slide

57. First Results of Remote Build Cache for Netflix CI
    CI build/test time has improved by 46%. Average CI build time is down from 14.6 min to 8 min. The remote
    cache is saving 6.6 min per CI build on aggregate for all CI builds.
    ● 24 min of build
    execution time saved
    per CI build
    ● Serial Execution
    Factor: 3.5
    ● Result: 6.6 Min wait
    time saved per CI
    build (24.5/3.5)
    

    View Slide

58. First Results of Remote Build Cache for Netflix CI
    The best case for CI is 3 min 18 sec!
    

    View Slide

59. Impact of remote caching on PR build times
    ● Average PR check build time is down from 39 min to 25 min
    ● Best case ~11 min 25 sec.
    ● The remote cache is saving on average 14 min per PR build.
    ● The Netflix Android team ran 251 PR check builds last week.
    

    View Slide

60. Impact of remote caching on PR Build Times
    Best Case is 11 min 25 sec
    11 min
    

    View Slide

61. One year later…
    

    View Slide

62. Cache misses are back
    ● Different JDK Versions: Identify and fix major differences in JDK between CI/local that
    were causing cache misses on all compile jobs
    ● Setting Android Properties: Identify that cache misses were occurring for
    CompileLibraryResourcesTask which were fixed by enabling certain Android properties
    ● Older Versions of Dagger: Identify a cacheability issue caused by an older version of
    Dagger and verify that the upgraded version fixed it
    ● Non deterministic codegen: Identify code generations with differences between CI/local
    

    View Slide

63. Cacheable Task Executed: 835 💥
    

    View Slide

64. View Slide

65. View Slide

66. Java version mismatch
    caches/transforms-3/3c71c6377bd209bb9c86571321e581b4/transformed/output
    caches/transforms-3/a4bcee52a5b625e21f22eee7819e81c6/transformed/output
    

    View Slide

67. Java version mismatch : 11.0.15 vs 11.0.12 💥
    caches/transforms-3/3c71c6377bd209bb9c86571321e581b4/transformed/output
    caches/transforms-3/a4bcee52a5b625e21f22eee7819e81c6/transformed/output
    

    View Slide

68. Use Input Normalization
    allprojects {
    normalization {
    runtimeClasspath {
    metaInf {
    ignoreAttribute("Implementation-Version")
    ignoreAttribute("Created-By")
    }
    }
    }
    }
    

    View Slide

69. Cacheable Task Executed: 24 🎉
    

    View Slide

70. After we fixed all the cache misses…
    PR builds at record speed: best case 5 min
    Screen shot
    5 min
    

    View Slide

71. Pop Quiz: what changed in AGP 7.1 that
    contributed to this speed up?
    

    View Slide

72. Lint was made Cacheable in AGP 7.1
    

    View Slide

73. Few days after…
    

    View Slide

74. Cacheable Task Executed: 515 💥
    

    View Slide

75. __Schema.kt - Apollo GraphQL code generation 💥
    

    View Slide

76. View Slide

77. Cacheable Task Executed: 20 🎉
    AssembleDebug time back down to 2 min
    

    View Slide

78. View Slide

79. Managing dependencies
    ● Dependencies centralized : Dependencies.kt in buildSrc/
    

    View Slide

80. Managing dependencies
    ● Dependencies centralized : Dependencies.kt in buildSrc/
    ● buildSrc/ is part of build script
    

    View Slide

81. Managing dependencies
    ● Dependencies centralized : Dependencies.kt in buildSrc/
    ● buildSrc/ is part of build script
    ● Changes in buildSrc/ break incremental compilation
    

    View Slide

82. Gradle Version Catalog
    ● All dependencies centralized in a file : gradle/libs.versions.toml
    

    View Slide

83. Gradle Version Catalog
    ● All dependencies centralized in a file : gradle/libs.versions.toml
    ● Available in Gradle 7.4
    

    View Slide

84. Gradle Version Catalog
    ● All dependencies centralized in a file : gradle/libs.versions.toml
    ● Available in Gradle 7.4
    ● Not breaking incremental compilation
    

    View Slide

85. Gradle Version Catalog
    dependencies {
    implementation libs.my.dep
    implementation libs.bundles.androidx
    implementation libs.bundles.compose
    }
    my-dep = { module = "group:artifact", version = "version" }
    

    View Slide

86. buildSrc/Dependencies.kt
    gradle/libs.versions.toml
    

    View Slide

87. Managing dependencies
    ● Problems: Transitive dependencies
    

    View Slide

88. Managing dependencies
    ● Problems: Transitive dependencies
    

    View Slide

89. Gradle Dependency locking
    ● From Netflix Nebula, available in Gradle
    ● Lock all dependencies on a file
    ● Fail the build when lockfiles are not up-to-date
    

    View Slide

90. Gradle dependency locking
    gradle/libs.versions.toml
    

    View Slide

91. Gradle dependency locking
    .lockfiles/app_gradle.lockfile (generated)
    

    View Slide

92. Gradle dependency locking
    .lockfiles/app_gradle.lockfile (generated)
    

    View Slide

93. Gradle dependency locking
    configurations {
    compileClasspath {
    resolutionStrategy.activateDependencyLocking()
    }
    }
    gradle app:dependencies --write-locks
    

    View Slide

94. Send me a build scan for analysis…
    @Rooz via Gradle-Community.Slack.com
    

    View Slide

95. View Slide

96. Performance Best Practices
    ⬢ Upgrade to the latest of all tools and plugins: Gradle, AGP, Kotlin
    ⬢ Tools: Build Scans, Gradle Profiler
    ⬢ Build Essentials: Daemon, Parallel, Configuration on demand
    ⬢ Modularize
    ⬢ L1: Optimize your incremental build cache
    ⬢ L2: Local Cache
    ⬢ L3: File Watching, Configuration cache
    ⬢ Final Boss: Remote Cache
    ⬢ Enable caching, parallel, file system watching, configuration cache
    ⬢ Identify and fix performance issues like bottlenecks, cache misses, configuration time slow downs
    ⬢ Use convention plugins to organize build logic
    ⬢ Cache Fix plugin
    ⬢ Gradle Doctor Plugin
    

    View Slide

97. View Slide

98. We’re hiring
    https://jobs.netflix.com
    Android
    

    View Slide

99. View Slide

100. Thank you!
     

     View Slide