Save 37% off PRO during our Black Friday Sale! »

Working With Alternative Build Systems

3f9ebfb254c56d281d91bbee960219a8?s=47 Gautam Korlam
November 26, 2019

Working With Alternative Build Systems

As a mobile codebase grows, there are new types of challenges developers must tackle. One of the common challenges is the scalability of build systems like Gradle. Uber, Google & Facebook use Buck and Bazel to work with large codebases. In this talk you will learn how these alternative build systems impact both the developer experience and the performance of builds. You will also learn if, when and how you should switch to them for your own codebase.

3f9ebfb254c56d281d91bbee960219a8?s=128

Gautam Korlam

November 26, 2019
Tweet

Transcript

  1. Working With Alterna/ve Build Systems Gautam Korlam & Kurt Nelson

    ! Developer Pla4orm at Uber
  2. Gradle at scale

  3. The configura-on phase becomes slow

  4. Groovy and Kotlin are full programming languages, not configura8on languages

  5. Gradle holds the whole project model in the IDE

  6. Performance tuning is hard Tuning 3rdparty plugins is harder

  7. Working with mul/ple languages is not easy

  8. Reproducibility is key

  9. Dependency management

  10. Extensibility with plugins

  11. Enterprise feature set

  12. How do we fix these issues?

  13. A language built for builds - Starlark Determinis)c, Herme)c, immutable,

    performant, simple and built with tooling in mind
  14. What are the alterna*ve build systems?

  15. Bazel Buck

  16. Bazel and Buck are language and toolchain agnos3c build systems

    that use starlark and are tailored towards large modular codebases
  17. Bazel Built by Google Used by Dropbox, Google, Ly2, Spo4fy,

    Square, Uber
  18. Buck Built by Facebook Used by Airbnb, Facebook, Ly3, Square,

    Uber
  19. Okbuck Built by Uber Hybrid inter-op between Gradle and Buck

  20. Example configura0ons using Starlark and Bazel

  21. Example android kotlin app1 load("@io_bazel_rules_kotlin//kotlin:kotlin.bzl", "kt_android_library") kt_android_library( name = "lib",

    srcs = glob(["java/com/example/bazel/*.kt"]), custom_package = "com.example.bazel", manifest = "AndroidManifest.xml", resource_files = glob(["res/**"]), visibility = ["//:__pkg__"], deps = [ "@maven//:androidx_appcompat_appcompat", "@maven//:androidx_core_core", "@maven//:androidx_core_core_ktx", "@maven//:androidx_drawerlayout_drawerlayout", "@maven//:androidx_fragment_fragment", "@maven//:androidx_lifecycle_lifecycle_common", "@maven//:androidx_lifecycle_lifecycle_viewmodel", ], ) 1 For more examples, checkout github
  22. Example android kotlin app1 android_binary( name = "app", custom_package =

    "com.example.bazel", manifest = "AndroidManifest.xml", deps = [ ":lib", ], ) 1 For more examples, checkout github
  23. Why use an alterna-ve build system?

  24. They are fast!

  25. Scale be(er as your codebase grows

  26. Design differences Gradle Bazel Task Graph Ac,on Graph Lazy if

    configured by task All targets lazy by default Combining different languages in one build is difficult Fully na,ve graph of different languages possible
  27. Combining Languages cc_library( name = "main-jni-lib", srcs = [ "@local_jdk//:jni_header",

    "@local_jdk//:jni_md_header-linux", "Main.cc" ], hdrs = [ "Main.h" ], includes = [ "external/local_jdk/include", "external/local_jdk/include/linux" ], ) cc_binary( name = "libmain-jni.so", deps = [ ":main-jni-lib" ], linkshared = 1, )
  28. Combining Languages java_binary( name = "Main", srcs = [ "Main.java"

    ], main_class = "Main", data = [ ":libmain-jni.so" ], jvm_flags = [ "-Djava.library.path=." ], )
  29. Fool proof extensibility Gradle Bazel Caching is opt-in Caching is

    built in Easy to have side effects Impossible to have side effects Need to understand plugin api and internals Standard Starlark Modifying exis@ng behavior is not straight forward All rules can be wrapped in a macro
  30. Extending using macros2 def kt_android_library(name, exports = [], visibility =

    None, **kwargs): """Creates an Android sandwich library. `srcs`, `deps`, `plugins` are routed to `kt_jvm_library` the other android related attributes are handled by the native `android_library` rule. """ native.android_library( name = name, exports = exports + _kt_android_artifact(name, **kwargs), visibility = visibility, testonly = kwargs.get("testonly", default=0), ) 2 For more details on bazel kotlin rules, checkout github
  31. Extending using macros2 def _kt_android_artifact(name, srcs = [], deps =

    [], plugins = [], **kwargs): """Delegates Android related build attributes to the native rules but uses the Kotlin builder to compile Java and Kotlin srcs. Returns a sequence of labels that wrapping macro should export. """ base_name = name + "_base" kt_name = name + "_kt" base_deps = deps + ["@io_bazel_rules_kotlin//kotlin/internal/jvm:android_sdk"] native.android_library( name = base_name, visibility = ["//visibility:private"], exports = base_deps, **kwargs ) _kt_jvm_library( name = kt_name, srcs = srcs, deps = base_deps + [base_name], plugins = plugins, testonly = kwargs.get("testonly", default=0), visibility = ["//visibility:private"], ) return [base_name, kt_name] 2 For more details on bazel kotlin rules, checkout github
  32. Organiza(onal Gradle Bazel Popular standard for Java/Kotlin Any Language or

    pla8orm, including iOS Android developers know it already Android developers can build iOS or backend Follow Android plugin updates Local control over when updates happen Supported by Google and usable out of the box Requires resourcing
  33. How do alterna,ve build systems solve problems at scale?

  34. Herme%c and reproducible builds out of the box

  35. Tooling exists for large refactorings of the build graph

  36. Shared infrastructure for all All languages are on equal foo.ng

    All targets can be remotely built and cached All pla&orms can share common code like models
  37. Changes the developer experience

  38. Flakiness is a thing of the past!

  39. Take updates on your own cadence

  40. Adding new targets is simple Developers naturally build smaller targets

  41. All developers are empowered to extend the build system

  42. Downsides

  43. Immediate access to latest Android build features o3en requires work

  44. IDE integra,on is not the best

  45. Learning curve for engineers

  46. Community and resources3 3 More info about BazelCon

  47. Should I switch to an alterna2ve build system?

  48. When should I consider it?

  49. How do I go about switching to an alterna4ve?

  50. Thanks Ques%ons?