How to make Kotlin compile 2x faster on a super large project?

In this talk, we'll introduce 5 advanced optimizations, from easy to hard to existing Kotlin compilers which are related to cache rate improvement and cache reuse. One of the heavyweight optimizations we call the "baseline kotlin cache". As we all know, normally, any changes to a low-level module will make the parent module cache fingerprint change invalid. We solved this problem by turning the fuzzy cache of the compileKotlin task into an incremental record bin, and reducing the full build time of the TikTok app in any scenario to 50%! Those solutions from TikTok could help Kotlin module build 2x+ faster than it used to be.

Transcript

1. Droidcon San Francisco - June, 2024 Make Kotlin compile 2x faster Pengwen Jiao Tech Lead - Mobile Arch & Infrastructure
2. Droidcon San Francisco - June, 2024 Agenda • Introduction
3. Droidcon San Francisco - June, 2024 Agenda • Introduction • Why build speed is important & challenge?
4. Droidcon San Francisco - June, 2024 Agenda • Introduction • Why build speed is important & challenge? • How to make Kotlin compile 2x faster? 🚀
5. Droidcon San Francisco - June, 2024 Who am I • Tech lead of TikTok Android Engineering Infra
6. Droidcon San Francisco - June, 2024 Who am I • Tech lead of TikTok Android Engineering Infra • 5 years on Android compile toolchain developing (Gradle/Kotlin/AGP etc.)
7. Droidcon San Francisco - June, 2024 Who am I • Tech lead of TikTok Android Engineering Infra • 5 years on Android compile toolchain developing (Gradle/Kotlin/AGP etc.) • 3+ years on whole DevOps lifecycle efficiency improvement
8. Droidcon San Francisco - June, 2024 Droidcon San Francisco - June, 2024 TikTok Project Lines of Code 5M+ • Super MonoRepo, multi products shared
9. Droidcon San Francisco - June, 2024 Droidcon San Francisco - June, 2024 TikTok Project Lines of Code 5M+ • Super MonoRepo, multi products shared • Fully modularization, 900+ modules
10. Droidcon San Francisco - June, 2024 Droidcon San Francisco - June, 2024 TikTok Project Lines of Code 5M+ • Super MonoRepo, multi products shared • Fully modularization, 900+ modules • Fully source code compilation
11. Droidcon San Francisco - June, 2024 Droidcon San Francisco - June, 2024 TikTok Project Lines of Code 5M+ Android developers 600+ Build count 6k/d
12. Droidcon San Francisco - June, 2024 Why build speed important? • 1% build faster, around 1M$ saving / year
13. Droidcon San Francisco - June, 2024 Why build speed important? • 1% build faster, around 1M$ saving / year • Less wait time on previewing code changes
14. Droidcon San Francisco - June, 2024 Why build speed important? • 1% build faster, around 1M$ saving / year • Less wait time on previewing code changes • Better experience
15. Droidcon San Francisco - June, 2024 How to optimize build speed?
16. Droidcon San Francisco - June, 2024 How to optimize build speed? Make single file build faster. Question Answers
17. Droidcon San Francisco - June, 2024 Droidcon San Francisco - June, 2024 Upgrade machine! Faster CPU , Faster build
18. Droidcon San Francisco - June, 2024 How to optimize build speed? Make single file build faster. Question Answers Reduce the amount of compiled code
19. Droidcon San Francisco - June, 2024 Publish part of project as aar √ Reduce amount of compiled code
20. Droidcon San Francisco - June, 2024 Publish part of project as aar √ Reduce amount of compiled code X Doesn’t work good with cache system X Have trouble when IDE highlight X Not compatible with compiler features like desugar and inline
21. Droidcon San Francisco - June, 2024 How to optimize build speed? Make single file build faster. Question Answers Reduce the amount of compiled code Make a good usage of cache and incremental build
22. Droidcon San Francisco - June, 2024 Optimizations Remove unused dependencies Decouple modules Classpath isolation 1 2 3 4 ways to make a better usage of cache and incremental build, which let kotlin compile 2x faster Cache to incremental 4
23. Droidcon San Francisco - June, 2024 How Gradle execute tasks task exec input output • EXECUTED • UP-TO-DATE • FROM-CACHE • SKIPPED • NO-SOURCE
24. Droidcon San Francisco - June, 2024 How Gradle execute tasks task exec input output • EXECUTED: default status, execute task logic • UP-TO-DATE: input no changes & output existed • FROM-CACHE: input fingerprint matches cache key
25. Droidcon San Francisco - June, 2024 Optimizations Remove unused dependencies Decouple modules Classpath isolation 1 2 4 ways to make a better usage of cache and incremental build, which let kotlin compile 2x faster Cache to incremental 3 4
26. Droidcon San Francisco - June, 2024 Dependency Tree module A (waiting) module B (waiting) module D (compiling) module C (compiling) module E (compiled) abi classpath build .gradle
27. Droidcon San Francisco - June, 2024 Dependency Tree module A (waiting) module B (waiting) module D (compiling) module C (compiling) module E (compiled) decoupled decoupled
28. Droidcon San Francisco - June, 2024 Dependency Tree module A (waiting) module B (compiling) module D (compiling) module C (compiling) module E (compiled) What does complex deps affect? • Longest build chain time cost • Incremental build rate • Build cache hit rate
29. Droidcon San Francisco - June, 2024 Find unused dependencies • ./gradlew dependencies manually sort out module dependencies and clean • com.autonomousapps.dependency-analysis detect module abi input to check whether in use (use longer time)
30. Droidcon San Francisco - June, 2024 Optimizations Remove unused dependencies Decouple modules Classpath isolation √ 4 ways to make a better usage of cache and incremental build, which let kotlin compile 2x faster Cache to incremental 3 4 2
31. Droidcon San Francisco - June, 2024 Project design example app network Clz A Clz B Clz C Clz D common base im image usually change login
32. Droidcon San Francisco - June, 2024 Miscellaneous module network Clz A Clz B Clz C Clz D Big base module image Clz E Clz F Clz G Clz H
33. Droidcon San Francisco - June, 2024 Measure single responsibility network Clz A Clz B Clz C Clz D Big base module image Single Responsibility Score = avg[each dep edge](used refs count/export refs count)
34. Droidcon San Francisco - June, 2024 Measure single responsibility network Clz A Clz B Clz C Clz D Big base module image Single Responsibility Score = avg[each dep edge](used refs count/export refs count)
35. Droidcon San Francisco - June, 2024 Measure single responsibility network Clz A Clz B Clz C Clz D Big base module image Single Responsibility Score = avg[each dep edge](used refs count/export refs count)
36. Droidcon San Francisco - June, 2024 Increase single responsibility score network Clz A Clz B Clz C Clz D base model image math utils camera utils
37. Droidcon San Francisco - June, 2024 Measure modify affect app biz network biz im biz image usually change biz login Hotspot Score = longest dependency depth * modify count per time period longest dependency depth = 4
38. Droidcon San Francisco - June, 2024 Decrease modify affect biz image app biz network biz im biz login image api network api im api login api DI / SPI binding
39. Droidcon San Francisco - June, 2024 Principle on designing project structure • Single responsibility • More frequently it change, higher layer it place • Keep least export Use SPI (Service Provider Interface) or DI (Dependency Injection) to abstract modules if necessary
40. Droidcon San Francisco - June, 2024 Optimizations Remove unused dependencies Decouple modules Classpath isolation √ 4 ways to make a better usage of cache and incremental build, which let kotlin compile 2x faster Cache to incremental 4 3 √
41. Droidcon San Francisco - June, 2024 Include plugin classpathes ${project_root}/build.gradle
42. Droidcon San Francisco - June, 2024 How Gradle execute tasks task exec input output • EXECUTED: default status, execute task logic • UP-TO-DATE: input no changes & output existed • FROM-CACHE: input fingerprint matches cache key • SKIPPED: task not meet onlyIf predicate or excluded • NO-SOURCE: no input
43. Droidcon San Francisco - June, 2024 Task cache key (fingerprint) Value Properties File Properties Input Task Implementation Output Files Output Cacheable? Hash fingerprint + Cache Service Replace
44. Droidcon San Francisco - June, 2024 Task cache key (fingerprint) Run tasks with -Dorg.gradle.caching.debug=true obtain cache key calculation details Change plugin list
45. Droidcon San Francisco - June, 2024 Gradle system classloader settings loader root build script module build script project loader app Project shared loaders Module isolated loaders module build script project loader Library 1 module build script project loader Library 2
46. Droidcon San Francisco - June, 2024 Gradle system classloader settings loader root build script module build script project loader app Project shared loaders Module isolated loaders module build script project loader Library 1 module build script project loader Library 2 Global cache breaking 💥
47. Droidcon San Francisco - June, 2024 Gradle system classloader settings loader root build script module build script project loader app Project shared loaders Module isolated loaders module build script project loader Library 1 module build script project loader Library 2
48. Droidcon San Francisco - June, 2024 Optimizations Remove unused dependencies Decouple modules Classpath isolation √ 4 ways to make a better usage of cache and incremental build, which let kotlin compile 2x faster Cache to incremental 4 √ √
49. Droidcon San Francisco - June, 2024 hit Cached build hit hit hit hit hit build time 40s
50. Droidcon San Francisco - June, 2024 hit Cached build Build after change code (without opt) hit hit hit hit hit miss miss hit miss miss hit make change here build time 40s build time 200s
51. Droidcon San Francisco - June, 2024 Why cache pack cannot be used for incremental build?
52. Droidcon San Francisco - June, 2024 How Kotlin compiler do incremental build • caches-jvm/ jvm classes blocks cache files • last-build.bin stored last build time stamp • build-history.bin A List<BuildDifference> binary stream records history build diffs
53. Droidcon San Francisco - June, 2024 Restore Kotlin incremental environment Kotlin compile task Regular gradle cache Extract Remote
54. Droidcon San Francisco - June, 2024 Restore Kotlin incremental environment Kotlin compile task Regular gradle cache Custom Kotlin incremental cache Extract Extract Remote
55. Droidcon San Francisco - June, 2024 Build after change code (Incremental environment applied) incr. incr. hit incr. incr. hit make change here build time 200s build time 50s Build after change code (without opt) miss miss hit miss miss hit make change here
56. Droidcon San Francisco - June, 2024 How about adding one more step?
57. Droidcon San Francisco - June, 2024 Problem of Gradle Cache system Low match possibility: need always keep several unstable parameters 100% match to find the cache
58. Droidcon San Francisco - June, 2024 Baseline cache Fuzzy cache matching Regular gradle cache Kotlin incremental cache Remote cache repository Cache pack (94.5% match) Cache pack (25.5% match) Cache pack (99.8% match) Cache pack (97.4% match) Cache pack (90.5% match) Required (compiler ver., plugin list, etc ) Fuzzy cache fingerprint Optional (dependency list, code hash, etc.) query
59. Droidcon San Francisco - June, 2024 Baseline cache Fuzzy cache matching Regular gradle cache Kotlin incremental cache Remote cache repository Cache pack (94.5% match) Cache pack (25.5% match) Cache pack (99.8% match) Cache pack (97.4% match) Cache pack (90.5% match) Required (compiler ver., plugin list, etc ) Fuzzy cache fingerprint Optional (dependency list, code hash, etc.) query
60. Droidcon San Francisco - June, 2024 Baseline cache Fuzzy cache matching Diff cache and local source local source Kotlin compiler submit dirty file & trigger incremental Cache pack (99.8% match)
61. Droidcon San Francisco - June, 2024 Baseline cache Fuzzy cache matching Regular gradle cache Kotlin incremental cache Extract Extract Diff cache and local source Remote cache repository local source Kotlin compiler inject diff & trigger incremental Cache pack (94.5% match) Cache pack (25.5% match) Cache pack (99.8% match) Cache pack (97.4% match) Cache pack (90.5% match) Hit rate: 30% -> 80%+
62. Droidcon San Francisco - June, 2024 What’s more • Baseline cache can get rid of restrictions from Gradle cache mechanism; • Baseline cache can be also applied in any tasks which supports incremental build like transform, proguard, mergeResources to make everything cacheable;
63. Droidcon San Francisco - June, 2024 Recap
64. Droidcon San Francisco - June, 2024 Flat project structure to make build parallelize Remove unused dependencies Never leave the frequent changed module in lower layer Decouple modules Problem in cache system design and one possible solution Cache to incremental Choose a proper scope for classpath and plugin Classpath isolation 2+x Kotlin efficiency improvement
65. Droidcon San Francisco - June, 2024 Questions?
66. Droidcon San Francisco - June, 2024 Keep in touch! mailto://sinoreimu@gmail.com