Lightning fast Android builds with Gradle + Buck

Gradle is the typical build system of choice for android development, but it's performance tends to degrade as code bases become more complex. There exist alternate build systems like Buck (built by Facebook) that can build significantly faster than Gradle for Android, but they can be hard to migrate to and maintain. What if it is possible to have both the simplicity of expressing build logic in Gradle and also the fast build speeds of Buck? In this talk, Gautam will go over the open source OkBuck Gradle plugin and how it can help improve your android build speeds without having to learn and maintain a new build system.

Transcript

1. Lightning fast Android builds with Gradle + Buck Gautam Korlam Android Developer Experience, Uber Slides: https://goo.gl/JXrjk3
2. What do you use to build Android Apps?
3. Gradle for Android Development • Most widely used build system for Android • Officially recommended by Google • Tools ecosystem - Android Gradle Plugin, Android Studio, Third party plugins • Easy to understand, maintain and extend
4. Gradle Project Structure
5. Single Gradle Module Pros • Simple to start a project • Fast gradle configuration time Cons • Builds start becoming slower as code size grows • Instant run/incremental compilation - do not work with all annotation processors
6. Multiple Remote Dependencies Pros • Reusable and focused dependencies • No need to build and test precompiled artifacts Cons • Making changes in them is no longer atomic • Dependency Hell
7. Multiple Gradle Modules Pros • No dependency hell • Build and test as needed Cons • Configuration and Gradle sync times start to grow • Memory usage goes up a lot
8. How does Gradle Scale?
9. 100+ modules • We found that gradle does not scale very well with large android projects • A single line change can take several minutes to build and test • Daemon, parallel builds and configuration on demand help, but not by much • Gradle build caching/better incremental builds are still not quite there yet • As code is split into smaller modules, Android Studio becomes slow
10. Waiting for Gradle builds Good, we’ll wait for our gradle builds here!
11. Alternative Build Systems • Buck (Facebook), Bazel (Google) and Pants (Twitter) • Focused around what to produce rather than what to run • Efficient Parallelization of projects with many small modules • Cache outputs and avoid rebuilding if public api does not change • Advanced capabilities like remote build cache
12. Why Buck? • Most mature among all and actively developed • Heaviest focus on mobile and scaling complex builds • Closest to the Android Gradle Plugin in terms of supported features • Provides fast and correct incremental builds and tests for Android Development • Used at Uber for building iOS as well https://buckbuild.com
13. How Fast is Buck? Build all Apps Gradle Buck Speed Up Clean 8 mins 40 secs 3 mins 20 secs 2.6x Incremental - ABI Change 3 mins 47 secs 43 secs 5.3x Incremental - Same ABI 2 mins 47 secs 30 secs 5.6x No-op 3 mins 38 secs 0.2 secs 1090x • Test project on github with many gradle modules • Can be built with both gradle and buck https://github.com/kageiit/android-studio-gradle-test Gradle - 3.4-rc-1 Android Gradle Plugin - 2.3.0-beta3
14. How Fast is Buck? Build all Apps Gradle Buck Speed Up Clean 8 mins 40 secs 3 mins 20 secs 2.6x Incremental - ABI Change 3 mins 47 secs 43 secs 5.3x Incremental - Same ABI 2 mins 47 secs 30 secs 5.6x No-op 3 mins 38 secs 0.2 secs 1090x • Test project on github with many gradle modules • Can be built with both gradle and buck https://github.com/kageiit/android-studio-gradle-test Gradle - 3.4-rc-1 Android Gradle Plugin - 2.3.0-beta3
15. How Fast is Buck? Build all Apps Gradle Buck Speed Up Clean 8 mins 40 secs 3 mins 20 secs 2.6x Incremental - ABI Change 3 mins 47 secs 43 secs 5.3x Incremental - Same ABI 2 mins 47 secs 30 secs 5.6x No-op 3 mins 38 secs 0.2 secs 1090x • Test project on github with many gradle modules • Can be built with both gradle and buck https://github.com/kageiit/android-studio-gradle-test Gradle - 3.4-rc-1 Android Gradle Plugin - 2.3.0-beta3
16. How Fast is Buck? Build all Apps Gradle Buck Speed Up Clean 8 mins 40 secs 3 mins 20 secs 2.6x Incremental - ABI Change 3 mins 47 secs 43 secs 5.3x Incremental - Same ABI 2 mins 47 secs 30 secs 5.6x No-op 3 mins 38 secs 0.2 secs 1090x • Test project on github with many gradle modules • Can be built with both gradle and buck https://github.com/kageiit/android-studio-gradle-test Gradle - 3.4-rc-1 Android Gradle Plugin - 2.3.0-beta3
17. Hidden Cost of Build Time 3 mins per iteration 25 Hours spent per day 30 secs per iteration 4 Hours spent per day 20 Developers Build and Deploy 25 times a day VS 2 Extra Developers!
18. Android Builds at Uber • 300+ gradle modules in project • Clean buck builds are 180x faster than gradle (network cache!) • Incremental buck builds are ~8x faster than gradle • Most day to day tasks benefit greatly from the network cache • Clean builds are rare. CI builds are fully incremental with buck https://github.com/uber/buck-http-cache
19. Sounds awesome but why is buck not used widely?
20. Verbose Build Configuration
21. No Dependency Resolution
22. Other Considerations • Migrating completely can be painful and time consuming • Need to rewrite a lot of existing tools • Engineers need to learn a new build system • Isolation from Android tools, knowledge and community focused around gradle • Cannot take coffee breaks during builds
23. Simplicity of Gradle + Fast Build Speeds of Buck
24. OkBuck • Open source gradle plugin that lets you build gradle projects with buck! • Generates buck build files for complex android gradle projects seamlessly • Provides many configuration options for customizing generated BUCK files • Bridges the gap between simplicity of gradle and complexity of buck • Used at Uber for building android projects with buck https://github.com/uber/okbuck
25. Under the Hood • Creates a model of the gradle dependency graph in memory • Downloads external dependencies to a local cache • Translates the project’s gradle task model into buck’s rule based model
28. Seamless Transition • Extracts android gradle plugin options to use during buck builds ◦ Sources, resources and manifests from flavors/build types ◦ Annotation processors ◦ Consumer proguard rules ◦ Custom lint jars • Provides a wrapper around buck similar to the gradle wrapper • Wrapper downloads/installs/updates buck • Watches project changes and automatically regenerates BUCK files if needed
29. • Build Flavors/Variants • Multidex • Exopackage • Annotation Processors • Proguard • Manifest Placeholders • Retrolambda • Java Unit Tests • Lint Supported Features What is Supported? • Robolectric Unit Tests • Espresso UI Tests • Aidl • Code Coverage • Intellij Project Generation • Custom Gradle Plugin Hooks • Vector Drawables • Custom BuildConfig Fields • Transform API • Flavor Dimensions • Kotlin • Databinding • NDK Integration • Jack Not Supported Yet (Planned)
30. Migration Guide
31. Apply the Plugin
32. Quick Start
33. Quick Start
34. Quick Start
35. Quick Start
36. Quick Start
37. Quick Start
38. Quick Start https://github.com/uber/okbuck/blob/master/Usage.md
39. Configure Buck to Build Fast • Move all dependencies to a top level configuration file • Split up your project into a number of smaller modules • Disable unnecessary gradle plugins when generating BUCK files • Many configuration options can be tweaked in the.buckconfig file • Invest in adding exopackage support to your apps https://buckbuild.com/concept/buckconfig.html
40. Exopackage • An exopackage is a small shell of an Android app that contains the minimal code and resources needed to run • Loading the application code at runtime avoids a full reinstall when testing a Java change • < 15s average incremental deploy https://buckbuild.com/article/exopackage.html
41. Making code modular • Breakup the gigantic app module into smaller focused modules • Less frequently touched code is a good candidate for separation • Keeping top level app/apk module small allows for fastest exopackage builds • Smaller modules build/test faster and cache better • buck suggest gives possible ways to untangle class level dependencies https://buckbuild.com/command/suggest.html
42. Setup your IDE • Install the Buck Intellij Plugin • Lets you perform frequent tasks like deploying apks and debugging • Configure the Buck Plugin to invoke all buck commands via the wrapper • Intellij Community Edition plays nicer with buck projects than Android Studio • This means no more Gradle Sync! https://plugins.jetbrains.com/plugin/7826?pr=idea
43. What’s Next? • Full compatibility with android gradle plugin • Better Kotlin Support • More docs, samples and best practices • Fully open source most of our android tooling stack
44. Who is using Okbuck + Buck?
45. Questions? Thank You @kageiit