Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Bazel

 Bazel

Talk was presented at Droidcon NYC 2019.

Why Bazel worth taking a look at and how it can help you scale development of your Android and iOS apps while (mostly) outperforming standard tooling like Gradle and xcodebuild.

In the talk we'll discuss following major topics:

- Multi-module setup vs build system
- Why Gradle is slow and how they're fixing it
- Why Gradle is fast and how they're fixing it
- ABI vs API vs implementation
- Compile avoidance
- Remote caching
- Remote execution
- Build configuration DSL (Groovy/Kotlin vs Starlark)
- IDE support
- CI/CD support
- Testing

Artem Zinnatullin

August 27, 2019
Tweet

More Decks by Artem Zinnatullin

Other Decks in Programming

Transcript

  1. Bazel

    View Slide

  2. Bazel

    View Slide

  3. View Slide

  4. Tacking a
    look at
    Bazel

    View Slide

  5. Build System Use Cases

    View Slide

  6. Build System Use Cases: User

    View Slide

  7. Build System Use Cases: User

    View Slide

  8. Build System
    Use Cases: User

    View Slide

  9. Build System Use Cases: Automation

    View Slide

  10. Build System Use Cases: Automation

    View Slide

  11. Build System Use Cases: Plugins/Extensions

    View Slide

  12. Build System Use Cases
    -Resolve dependencies

    View Slide

  13. Build System Use Cases
    -Resolve dependencies
    -Build (compile, process resources, etc)

    View Slide

  14. Build System Use Cases
    -Resolve dependencies
    -Build (compile, process resources, etc)
    -Run apps/tests

    View Slide

  15. Build System Use Cases
    -Resolve dependencies
    -Build (compile, process resources, etc)
    -Run apps/tests
    -Integrate with IDE (modules, actions, etc)

    View Slide

  16. Build System Use Cases
    -Resolve dependencies
    -Build (compile, process resources, etc)
    -Run apps/tests
    -Integrate with IDE (modules, actions, etc)
    -Integrate with scripts & CI

    View Slide

  17. Build System Use Cases
    -Resolve dependencies
    -Build (compile, process resources, etc)
    -Run apps/tests
    -Integrate with IDE (modules, actions, etc)
    -Integrate with scripts & CI
    -Plugins/Extensions API

    View Slide

  18. Build System Use Cases
    -Resolve dependencies
    -Build (compile, process resources, etc)
    -Run apps/tests
    -Integrate with IDE (modules, actions, etc)
    -Integrate with scripts & CI
    -Plugins/Extensions API
    -etc

    View Slide

  19. Design of Bazel:
    Configuration

    View Slide

  20. Design of Bazel: Configuration
    java_library(
    name = "a",
    srcs = [ "A.java" ],
    deps = [
    "//:b"
    ],
    )
    java_library(
    name = "b",
    srcs = [ "B.java" ],
    )
    Bazel BUILD file

    View Slide

  21. Design of Bazel: Configuration
    ‣Readable

    View Slide

  22. Design of Bazel: Configuration
    ‣Readable
    ‣Writeable

    View Slide

  23. Design of Bazel: Configuration
    ‣Readable
    ‣Writeable
    ‣Maintainable

    View Slide

  24. Design of Bazel: Configuration
    ‣Static Markdown Language

    View Slide

  25. Design of Bazel: Configuration
    ‣Static Markdown Language
    -XML
    -JSON
    -YAML (no God please no)
    -TOML
    -etc

    View Slide

  26. Design of Bazel: Configuration
    ‣Static Markdown Language
    -XML
    -JSON
    -YAML (no God please no)
    -TOML
    -etc
    ‣Programming Language

    View Slide

  27. Design of Bazel: Configuration
    ‣Static Markdown Language
    -XML
    -JSON
    -YAML (no God please no)
    -TOML
    -etc
    ‣Programming Language
    -Dynamically Typed
    -Statically Typed
    -etc

    View Slide

  28. Design of Bazel: Configuration

    4.0.0
    com.lyft.dagger1
    dagger-parent
    pom
    1.3.0
    Dagger (Parent)
    A fast dependency injector for Android and Java.
    https://github.com/square/dagger

    compiler
    core

    Maven pom.xml

    View Slide

  29. Design of Bazel: Configuration
    apply plugin: 'com.android.library'
    apply plugin: 'kotlin-android'
    android {
    defaultConfig {
    minSdkVersion versions.minSdk
    compileSdkVersion versions.compileSdk
    buildToolsVersion versions.buildTools
    }
    }
    dependencies {
    api project(':redom:api')
    api androidXLibraries.recyclerView
    }
    Gradle build.gradle

    View Slide

  30. Design of Bazel: Configuration
    java_library(
    name = "lib",
    srcs = glob(["src/main/java/**/*.java"]),
    deps = [
    "//:b",
    ]
    )
    java_test(
    name = "test",
    srcs = glob(["src/test/java/**/*.java"]),
    deps = [
    ":lib",
    ]
    )
    Bazel BUILD file

    View Slide

  31. Design of Bazel: Configuration
    ‣Reproducibility

    View Slide

  32. Design of Bazel: Configuration
    ‣Reproducibility
    -Limited stdlib

    View Slide

  33. Design of Bazel: Configuration
    ‣Reproducibility
    -Limited stdlib
    -No untracked IO

    View Slide

  34. Design of Bazel: Configuration
    ‣Reproducibility
    -Limited stdlib
    -No untracked IO
    -External deps need sha256

    View Slide

  35. Design of Bazel: Configuration
    apply plugin: 'java-library'
    apply plugin: 'kotlin'
    // yoooooo don't do this
    version = Calendar.instance.get(Calendar.DAY_OF_WEEK) == 2
    ? System.currentTimeMillis()
    : new Random().nextInt()

    View Slide

  36. Design of Bazel: Configuration
    ‣Gradle: Groovy

    View Slide

  37. Design of Bazel: Configuration
    ‣Gradle: Groovy
    ‣Groovy syntax and dynamic type system ()

    View Slide

  38. Design of Bazel: Configuration
    ‣Gradle: Groovy
    ‣Groovy syntax and dynamic type system ()
    ‣Unlimited mutability

    View Slide

  39. Design of Bazel: Configuration
    ‣Gradle: Groovy
    ‣Groovy syntax and dynamic type system ()
    ‣Unlimited mutability
    ‣Unlimited IO

    View Slide

  40. Design of Bazel: Configuration
    ‣Gradle: Groovy
    ‣Groovy syntax and dynamic type system ()
    ‣Unlimited mutability
    ‣Unlimited IO
    ‣Full JDK stdlib

    View Slide

  41. Design of Bazel: Configuration
    ‣Gradle: Groovy
    ‣Groovy syntax and dynamic type system ()
    ‣Unlimited mutability
    ‣Unlimited IO
    ‣Full JDK stdlib
    ‣Can write Java in Groovy

    View Slide

  42. View Slide

  43. Design of Bazel: Configuration
    ‣Gradle: Kotlin

    View Slide

  44. Design of Bazel: Configuration
    ‣Gradle: Kotlin
    ‣Kotlin syntax and static type system ()

    View Slide

  45. Design of Bazel: Configuration
    ‣Gradle: Kotlin
    ‣Kotlin syntax and static type system ()
    ‣Unlimited mutability

    View Slide

  46. Design of Bazel: Configuration
    ‣Gradle: Kotlin
    ‣Kotlin syntax and static type system ()
    ‣Unlimited mutability
    ‣Unlimited IO

    View Slide

  47. Design of Bazel: Configuration
    ‣Gradle: Kotlin
    ‣Kotlin syntax and static type system ()
    ‣Unlimited mutability
    ‣Unlimited IO
    ‣Full JDK stdlib

    View Slide

  48. Design of Bazel: Configuration
    See “Fundamental Design Issues of
    Gradle Build System”
    https:!//artemzin.com/blog/fundamental-design-issues-of-gradle-build-system/
    *includes proposals to solve the issues

    View Slide

  49. Design of Bazel: Configuration
    Enter:
    Skylark
    Starlark

    View Slide

  50. Design of Bazel: Configuration
    ‣Bazel Starlark Evaluation Model
    https:"//docs.bazel.build/versions/master/skylark/concepts.html#evaluation-model

    View Slide

  51. Design of Bazel: Configuration
    ‣Bazel Starlark Evaluation Model
    -Loading Phase
    -Read BUILD and .bzl files required for given build
    -Bazel targets are directly mapped to file paths in the
    project
    https:"//docs.bazel.build/versions/master/skylark/concepts.html#evaluation-model

    View Slide

  52. Design of Bazel: Configuration
    ‣Bazel Starlark Evaluation Model
    -Loading Phase
    -Read BUILD and .bzl files required for given build
    -Bazel targets are directly mapped to file paths in the
    project
    -Analysis Phase
    -Rules are executed
    -Rules instantiate actual Actions
    -Action graph is composed and validated
    https:"//docs.bazel.build/versions/master/skylark/concepts.html#evaluation-model

    View Slide

  53. Design of Bazel: Configuration
    ‣Bazel Starlark Evaluation Model
    -Loading Phase
    -Read BUILD and .bzl files required for given build
    -Bazel targets are directly mapped to file paths in the
    project
    -Analysis Phase
    -Rules are executed
    -Rules instantiate actual Actions
    -Action graph is composed and validated
    -Execution Phase
    -Action graph is executed
    https:"//docs.bazel.build/versions/master/skylark/concepts.html#evaluation-model

    View Slide

  54. Design of Bazel: Configuration
    ‣Starlark
    https:"//docs.bazel.build/versions/master/skylark/language.html

    View Slide

  55. Design of Bazel: Configuration
    ‣Starlark
    ‣Python 3 syntax and dynamic type system ()
    https:"//docs.bazel.build/versions/master/skylark/language.html

    View Slide

  56. Design of Bazel: Configuration
    ‣Starlark
    ‣Python 3 syntax and dynamic type system ()
    ‣Custom runtime
    https:"//docs.bazel.build/versions/master/skylark/language.html

    View Slide

  57. Design of Bazel: Configuration
    ‣Starlark
    ‣Python 3 syntax and dynamic type system ()
    ‣Custom runtime
    ‣Custom stdlib
    https:"//docs.bazel.build/versions/master/skylark/language.html

    View Slide

  58. Design of Bazel: Configuration
    ‣Starlark
    ‣Python 3 syntax and dynamic type system ()
    ‣Custom runtime
    ‣Custom stdlib
    ‣Extremely limited mutability
    https:"//docs.bazel.build/versions/master/skylark/language.html

    View Slide

  59. Design of Bazel: Configuration
    ‣Starlark
    ‣Python 3 syntax and dynamic type system ()
    ‣Custom runtime
    ‣Custom stdlib
    ‣Extremely limited mutability
    ‣No IO
    https:"//docs.bazel.build/versions/master/skylark/language.html

    View Slide

  60. Design of Bazel: Configuration
    ‣Starlark
    ‣Python 3 syntax and dynamic type system ()
    ‣Custom runtime
    ‣Custom stdlib
    ‣Extremely limited mutability
    ‣No IO
    ‣Bazel provides build & IO specific APIs
    https:"//docs.bazel.build/versions/master/skylark/language.html

    View Slide

  61. Design of Bazel: Configuration
    ‣Starlark
    ‣Python 3 syntax and dynamic type system ()
    ‣Custom runtime
    ‣Custom stdlib
    ‣Extremely limited mutability
    ‣No IO
    ‣Bazel provides build & IO specific APIs
    ‣Can be used outside of Bazel!
    https:"//docs.bazel.build/versions/master/skylark/language.html

    View Slide

  62. Design of Bazel: Configuration
    java_library(
    name = "lib",
    srcs = glob(["src/main/java/**/*.java"]),
    deps = [
    "//:b",
    ]
    )
    java_test(
    name = "test",
    srcs = glob(["src/test/java/**/*.java"]),
    deps = [
    ":lib",
    ]
    )
    Bazel BUILD file

    View Slide

  63. Design of Bazel: Configuration
    java_library(
    name = "lib",
    srcs = glob(["src/main/java/**/*.java"]),
    deps = [
    "//:b",
    ]
    )
    java_test(
    name = "test",
    srcs = glob(["src/test/java/**/*.java"]),
    deps = [
    ":lib",
    ]
    )
    Bazel BUILD file Target with name “lib”

    View Slide

  64. Design of Bazel: Configuration
    ‣Starlark
    https:"//docs.bazel.build/versions/master/skylark/language.html

    View Slide

  65. Design of Bazel: Configuration
    ‣Starlark
    -Functions
    https:"//docs.bazel.build/versions/master/skylark/language.html

    View Slide

  66. Design of Bazel: Configuration
    ‣Starlark
    -Functions
    - Rules
    https:"//docs.bazel.build/versions/master/skylark/language.html

    View Slide

  67. Design of Bazel: Configuration
    ‣Starlark
    -Functions
    - Rules
    - Macros
    https:"//docs.bazel.build/versions/master/skylark/language.html

    View Slide

  68. Design of Bazel: Configuration
    ‣Starlark
    -Functions
    - Rules
    - Macros
    -Variables
    - Mutable within a “context”
    - Frozen once context has been evaluated
    https:"//docs.bazel.build/versions/master/skylark/language.html

    View Slide

  69. Design of Bazel: Configuration
    ‣Rule
    https:"//docs.bazel.build/versions/master/skylark/rules.html

    View Slide

  70. Design of Bazel: Configuration
    ‣Rule
    -Function that performs reproducible IO for given Inputs
    https:"//docs.bazel.build/versions/master/skylark/rules.html

    View Slide

  71. Design of Bazel: Configuration
    ‣Rule
    -Function that performs reproducible IO for given Inputs
    -Declares set of public and private attributes
    (parameters)
    https:"//docs.bazel.build/versions/master/skylark/rules.html

    View Slide

  72. Design of Bazel: Configuration
    ‣Rule
    -Function that performs reproducible IO for given Inputs
    -Declares set of public and private attributes
    (parameters)
    -Examples: kotlin_library, java_test, etc
    https:"//docs.bazel.build/versions/master/skylark/rules.html

    View Slide

  73. Design of Bazel: Configuration
    ‣Rule
    -Function that performs reproducible IO for given Inputs
    -Declares set of public and private attributes
    (parameters)
    -Examples: kotlin_library, java_test, etc
    -Cannot perform IO during configuration/analysis
    https:"//docs.bazel.build/versions/master/skylark/rules.html

    View Slide

  74. Design of Bazel: Configuration
    ‣Rule
    -Function that performs reproducible IO for given Inputs
    -Declares set of public and private attributes
    (parameters)
    -Examples: kotlin_library, java_test, etc
    -Cannot perform IO during configuration/analysis
    -Declares list of actions to perform IO during execution
    https:"//docs.bazel.build/versions/master/skylark/rules.html

    View Slide

  75. Design of Bazel: Configuration
    ‣Rule
    -Function that performs reproducible IO for given Inputs
    -Declares set of public and private attributes
    (parameters)
    -Examples: kotlin_library, java_test, etc
    -Cannot perform IO during configuration/analysis
    -Declares list of actions to perform IO during execution
    -Exposes Provider(s) — rule’s public API for other rules
    https:"//docs.bazel.build/versions/master/skylark/rules.html

    View Slide

  76. Design of Bazel: Configuration
    android_instrumentation_test = rule(
    attrs = attrs.add(
    ANDROID_INSTRUMENTATION_TEST_ATTRS,
    ADDON_ATTRS
    ),
    fragments = ["android"],
    implementation = _impl,
    outputs = _outputs,
    test = True,
    )
    https:"//github.com/bazelbuild/rules_android

    View Slide

  77. Design of Bazel: Configuration
    https:"//github.com/bazelbuild/rules_android
    def _impl(ctx):
    # Main entry point into the Android test runner.
    test_runner = get_test_runner(ctx)
    ………
    # TODO(str): remove after android_test migration
    if ctx.attr._android_test_migration:
    # Test APK is a predeclared output on android_test
    ctx.actions.run_shell(
    inputs = [ctx.attr.test_app[ApkInfo].signed_apk],
    outputs = [ctx.outputs.test_app],
    command = "cp %s %s" % (ctx.attr.test_app[ApkInfo].signed_apk.path,
    ctx.outputs.test_app.path),
    )
    runfiles = runfiles.merge(ctx.runfiles(files = [ctx.outputs.test_app]))
    apks_to_install = []

    View Slide

  78. Design of Bazel: Configuration
    ‣Macro
    https:"//docs.bazel.build/versions/master/skylark/macros.html

    View Slide

  79. Design of Bazel: Configuration
    ‣Macro
    -Can instantiate rule(s)
    https:"//docs.bazel.build/versions/master/skylark/macros.html

    View Slide

  80. Design of Bazel: Configuration
    ‣Macro
    -Can instantiate rule(s)
    -Useful for code reuse, composing logic, default
    values
    https:"//docs.bazel.build/versions/master/skylark/macros.html

    View Slide

  81. Design of Bazel: Configuration
    ‣Macro
    -Can instantiate rule(s)
    -Useful for code reuse, composing logic, default
    values
    -Present only during Loading phase
    https:"//docs.bazel.build/versions/master/skylark/macros.html

    View Slide

  82. Design of Bazel: Configuration
    ‣Macro
    -Can instantiate rule(s)
    -Useful for code reuse, composing logic, default
    values
    -Present only during Loading phase
    -Examples: lyft_kotlin_library,
    lyft_android_binary, etc
    https:"//docs.bazel.build/versions/master/skylark/macros.html

    View Slide

  83. Design of Bazel: Configuration
    def lyft_android_library(**kwargs):

    View Slide

  84. Design of Bazel: Configuration
    def lyft_android_library(**kwargs):
    if kwargs.get("language") == "kotlin":
    _apply_detekt(**kwargs)

    View Slide

  85. Design of Bazel: Configuration
    def lyft_android_library(**kwargs):
    if kwargs.get("language") == "kotlin":
    _apply_detekt(**kwargs)
    _apply_pmd(**kwargs)

    View Slide

  86. Design of Bazel: Configuration
    def lyft_android_library(**kwargs):
    if kwargs.get("language") == "kotlin":
    _apply_detekt(**kwargs)
    _apply_pmd(**kwargs)
    kwargs = _apply_error_prone(**kwargs)

    View Slide

  87. Design of Bazel: Configuration
    def lyft_android_library(**kwargs):
    if kwargs.get("language") == "kotlin":
    _apply_detekt(**kwargs)
    _apply_pmd(**kwargs)
    kwargs = _apply_error_prone(**kwargs)
    kwargs = _apply_java_null_pkg_info_gen(
    **kwargs
    )

    View Slide

  88. Design of Bazel: Configuration
    def lyft_android_library(**kwargs):
    if kwargs.get("language") == "kotlin":
    _apply_detekt(**kwargs)
    _apply_pmd(**kwargs)
    kwargs = _apply_error_prone(**kwargs)
    kwargs = _apply_java_nullability_pkg_info_generator(**kwargs)
    android_library(**kwargs)

    View Slide

  89. Design of Bazel: Configuration
    Starlark vs multi-module
    projects

    View Slide

  90. Design of Bazel: Configuration
    ‣Parallelism

    View Slide

  91. Design of Bazel: Configuration
    ‣Parallelism
    -Limited mutability lets configure packages
    independently

    View Slide

  92. Design of Bazel: Configuration
    ‣Parallelism
    -Limited mutability lets configure packages
    independently
    -Lack of IO makes configuration CPU bound

    View Slide

  93. Design of Bazel: Configuration
    ‣Cache-ability
    -Lack of mutability and IO makes
    configuration cacheable

    View Slide

  94. Design of Bazel: Configuration
    ‣Configuration Avoidance
    -Bazel only evaluates BUILD and .bzl files
    required for the build command

    View Slide

  95. Design of Bazel: Configuration
    https:"//artemzin.com/blog/fundamental-design-issues-of-gradle-build-system/
    Gradle: config time / number of modules
    Seconds
    0
    6.5
    13
    19.5
    26
    Number of modules
    400 600 800 1000
    10
    15
    20
    25

    View Slide

  96. Design of Bazel: Configuration
    Gradle is working on Configuration Caching!

    View Slide

  97. Design of Bazel: Configuration
    ‣Visibility
    -Defines what packages or targets are
    allowed to directly depend on this target
    https:"//docs.bazel.build/versions/master/be/common-definitions.html#common-attributes

    View Slide

  98. Design of Bazel: Configuration
    java_library(
    name = "a",
    srcs = [ "A.java" ],
    deps = [
    "//:b"
    ],
    visibility = [ "//visibility:public" ]
    )
    java_library(
    name = "b",
    srcs = [ "B.java" ],
    visibility = [ "//myfriendlypkg:lib" ]
    )

    View Slide

  99. Design of Bazel: Configuration
    ‣Maintainability

    View Slide

  100. Design of Bazel: Configuration
    ‣Maintainability
    ‣Buildifier — tool for formatting & linting
    BUILD and .bzl files
    https:"//github.com/bazelbuild/buildtools

    View Slide

  101. Design of Bazel: Configuration
    ‣Maintainability
    ‣Buildifier — tool for formatting & linting
    BUILD and .bzl files
    ‣Buildozer — tool for mass refactoring of
    BUILD files
    https:"//github.com/bazelbuild/buildtools

    View Slide

  102. Design of Bazel: Configuration
    ‣Maintainability
    ‣Buildifier — tool for formatting & linting
    BUILD and .bzl files
    ‣Buildozer — tool for mass refactoring of
    BUILD files
    ‣Both written in Go, run pretty fast
    https:"//github.com/bazelbuild/buildtools

    View Slide

  103. Design of Bazel: Configuration
    java_library(
    name = "a",
    srcs = ["A.java"],
    deps = ["//:b"],
    )
    Before:

    View Slide

  104. Design of Bazel: Configuration
    java_library(
    name = "a",
    srcs = ["A.java"],
    deps = ["//:b"],
    )
    Before:

    View Slide

  105. Design of Bazel: Configuration
    java_library(
    name = "a",
    srcs = ["A.java"],
    deps = ["//:b"],
    )
    java_library(
    name = "a",
    srcs = ["A.java"],
    deps = ["//:b"],
    )
    Before:
    After:

    View Slide

  106. Design of Bazel: Configuration
    java_library(
    name = "a",
    srcs = ["A.java"],
    deps = ["//:b"],
    )
    https:"//github.com/bazelbuild/buildtools/blob/master/buildozer/README.md

    View Slide

  107. Design of Bazel: Configuration
    java_library(
    name = "a",
    srcs = ["A.java"],
    deps = ["//:b"],
    )
    https:"//github.com/bazelbuild/buildtools/blob/master/buildozer/README.md

    View Slide

  108. Design of Bazel: Configuration
    java_library(
    name = "a",
    srcs = ["A.java"],
    deps = ["//:b"],
    )
    lyft_java_library(
    name = "a",
    srcs = ["A.java"],
    deps = ["//:b"],
    )
    https:"//github.com/bazelbuild/buildtools/blob/master/buildozer/README.md

    View Slide

  109. Design of Bazel:
    Execution

    View Slide

  110. Design of Bazel: Execution

    View Slide

  111. Design of Bazel: Execution
    ‣Commands
    Command Purpose
    bazel query What Targets are there
    https:"//docs.bazel.build/versions/master/command-line-reference.html

    View Slide

  112. Design of Bazel: Execution
    ‣Commands
    Command Purpose
    bazel query What Targets are there
    bazel build Build one or more Targets
    https:"//docs.bazel.build/versions/master/command-line-reference.html

    View Slide

  113. Design of Bazel: Execution
    ‣Commands
    Command Purpose
    bazel query What Targets are there
    bazel build Build one or more Targets
    bazel test Run one or more Test Targets
    https:"//docs.bazel.build/versions/master/command-line-reference.html

    View Slide

  114. Design of Bazel: Execution
    ‣Commands
    Command Purpose
    bazel query What Targets are there
    bazel build Build one or more Targets
    bazel test Run one or more Test Targets
    bazel run Run one or more Run Targets
    https:"//docs.bazel.build/versions/master/command-line-reference.html

    View Slide

  115. Design of Bazel: Execution
    ‣Incrementability

    View Slide

  116. Design of Bazel: Execution
    ‣Incrementability
    ‣Bazel won’t do what has already been done
    (duh)

    View Slide

  117. Design of Bazel: Execution
    ‣Incrementability
    ‣Bazel won’t do what has already been done
    (duh)
    ‣Compilation is not incremental

    View Slide

  118. Design of Bazel: Execution
    ‣Incrementability
    ‣Bazel won’t do what has already been done
    (duh)
    ‣Compilation is not incremental
    ‣Tests results are also cached (you can opt
    out)

    View Slide

  119. Design of Bazel: Execution

    View Slide

  120. Design of Bazel: Execution
    ‣Parallelism
    ‣Bazel by default runs what it can in
    parallel

    View Slide

  121. Design of Bazel: Execution
    ‣ABI/header Extraction

    View Slide

  122. Design of Bazel: Execution
    ‣ABI/header Extraction
    -Bazel computes ABI for supported languages
    and uses it

    View Slide

  123. Design of Bazel: Execution

    View Slide

  124. Design of Bazel: Execution

    View Slide

  125. Design of Bazel: Execution

    View Slide

  126. Design of Bazel: Execution
    public class MyClass {
    private String x;
    public void a() {
    this.x = b();
    }
    private String b() {
    Random random = new Random();
    StringBuilder str = new StringBuilder();
    for (int i = 0; i < random.nextInt(1024); i++) {
    str.append(i);
    }
    return str.toString();
    }
    public static Runnable c(String s) {
    return () -> {
    new MyClass().b();
    };
    }
    }
    MyClass.java

    View Slide

  127. Design of Bazel: Execution
    public class MyClass {
    private String x;
    public void a() {
    this.x = b();
    }
    private String b() {
    Random random = new Random();
    StringBuilder str = new StringBuilder();
    for (int i = 0; i < random.nextInt(1024); i++) {
    str.append(i);
    }
    return str.toString();
    }
    public static Runnable c(String s) {
    return () -> {
    new MyClass().b();
    };
    }
    MyClass.java

    View Slide

  128. Design of Bazel: Execution
    public class MyClass {
    private String x;
    public void a() {
    this.x = b();
    }
    private String b() {
    Random random = new Random();
    StringBuilder str = new StringBuilder();
    for (int i = 0; i < random.nextInt(1024); i++) {
    str.append(i);
    }
    return str.toString();
    }
    public static Runnable c(String s) {
    return () -> {
    new MyClass().b();
    };
    }
    MyClass.java

    View Slide

  129. Design of Bazel: Execution
    public class MyClass {
    private String x;
    public void a() {
    this.x = b();
    }
    private String b() {
    Random random = new Random();
    StringBuilder str = new StringBuilder();
    for (int i = 0; i < random.nextInt(1024); i++) {
    str.append(i);
    }
    return str.toString();
    }
    public static Runnable c(String s) {
    return () -> {
    new MyClass().b();
    };
    }
    MyClass.java

    View Slide

  130. Design of Bazel: Execution
    public class MyClass {
    private String x;
    public void a() {
    this.x = b();
    }
    private String b() {
    Random random = new Random();
    StringBuilder str = new StringBuilder();
    for (int i = 0; i < random.nextInt(1024); i++) {
    str.append(i);
    }
    return str.toString();
    }
    public static Runnable c(String s) {
    return () -> {
    new MyClass().b();
    };
    }
    MyClass.java

    View Slide

  131. Design of Bazel: Execution
    public class MyClass {
    public void a() {
    this.x = b();
    }
    private String b() {
    Random random = new Random();
    StringBuilder str = new StringBuilder();
    for (int i = 0; i < random.nextInt(1024); i++) {
    str.append(i);
    }
    return str.toString();
    }
    public static Runnable c(String s) {
    return () -> {
    new MyClass().b();
    };
    }
    }
    MyClass.java

    View Slide

  132. Design of Bazel: Execution
    public class MyClass {
    public void a() {
    this.x = b();
    }
    private String b() {
    Random random = new Random();
    StringBuilder str = new StringBuilder();
    for (int i = 0; i < random.nextInt(1024); i++) {
    str.append(i);
    }
    return str.toString();
    }
    public static Runnable c(String s) {
    return () -> {
    new MyClass().b();
    };
    }
    MyClass.java

    View Slide

  133. Design of Bazel: Execution
    public class MyClass {
    public void a() {
    this.x = b();
    }
    private String b() {
    Random random = new Random();
    StringBuilder str = new StringBuilder();
    for (int i = 0; i < random.nextInt(1024); i++) {
    str.append(i);
    }
    return str.toString();
    }
    public static Runnable c(String s) {
    return () -> {
    new MyClass().b();
    };
    }
    MyClass.java

    View Slide

  134. Design of Bazel: Execution
    public class MyClass {
    public void a();
    private String b() {
    Random random = new Random();
    StringBuilder str = new StringBuilder();
    for (int i = 0; i < random.nextInt(1024); i++) {
    str.append(i);
    }
    return str.toString();
    }
    public static Runnable c(String s) {
    return () -> {
    new MyClass().b();
    };
    }
    }
    MyClass.java

    View Slide

  135. Design of Bazel: Execution
    public class MyClass {
    public void a();
    private String b() {
    Random random = new Random();
    StringBuilder str = new StringBuilder();
    for (int i = 0; i < random.nextInt(1024); i++) {
    str.append(i);
    }
    return str.toString();
    }
    public static Runnable c(String s) {
    return () -> {
    new MyClass().b();
    };
    }
    }
    MyClass.java

    View Slide

  136. Design of Bazel: Execution
    public class MyClass {
    public void a();
    private String b() {
    Random random = new Random();
    StringBuilder str = new StringBuilder();
    for (int i = 0; i < random.nextInt(1024); i++) {
    str.append(i);
    }
    return str.toString();
    }
    public static Runnable c(String s) {
    return () -> {
    new MyClass().b();
    };
    }
    }
    MyClass.java

    View Slide

  137. Design of Bazel: Execution
    public class MyClass {
    public void a();
    private String b() {
    Random random = new Random();
    StringBuilder str = new StringBuilder();
    for (int i = 0; i < random.nextInt(1024); i++) {
    str.append(i);
    }
    return str.toString();
    }
    public static Runnable c(String s) {
    return () -> {
    new MyClass().b();
    };
    }
    }
    MyClass.java

    View Slide

  138. Design of Bazel: Execution
    public class MyClass {
    public void a();
    public static Runnable c(String s) {
    return () -> {
    new MyClass().b();
    };
    }
    }
    MyClass.java

    View Slide

  139. Design of Bazel: Execution
    public class MyClass {
    public void a();
    public static Runnable c(String s) {
    return () -> {
    new MyClass().b();
    };
    }
    }
    MyClass.java

    View Slide

  140. Design of Bazel: Execution
    public class MyClass {
    public void a();
    public static Runnable c(String s) {
    return () -> {
    new MyClass().b();
    };
    }
    }
    MyClass.java

    View Slide

  141. Design of Bazel: Execution
    public class MyClass {
    public void a();
    public static Runnable c(String s);
    }

    View Slide

  142. Design of Bazel: Execution
    public class MyClass {
    public void a();
    public static Runnable c(String s);
    }
    MyClass_Header.java
    (actually, it’s .class right away)

    View Slide

  143. Design of Bazel: Execution

    View Slide

  144. Design of Bazel: Execution
    ABI/Header extraction
    Works for entire
    action graph

    View Slide

  145. Design of Bazel: Execution

    View Slide

  146. Design of Bazel: Execution
    ‣Persistent Workers
    https:"//blog.bazel.build/2015/12/10/java-workers.html

    View Slide

  147. Design of Bazel: Execution
    ‣Persistent Workers
    ‣Simple Request/Response Protocol between Bazel
    and a Worker

    View Slide

  148. Design of Bazel: Execution
    ‣Persistent Workers
    ‣Simple Request/Response Protocol between Bazel
    and a Worker
    ‣Bazel sends requests as Protobuf messages to
    Worker’s stdin

    View Slide

  149. Design of Bazel: Execution
    ‣Persistent Workers
    ‣Simple Request/Response Protocol between Bazel
    and a Worker
    ‣Bazel sends requests as Protobuf messages to
    Worker’s stdin
    ‣Bazel reads responses as Protobuf messages from
    Worker’s stdout

    View Slide

  150. Design of Bazel: Execution
    ‣Persistent Workers
    ‣Simple Request/Response Protocol between Bazel
    and a Worker
    ‣Bazel sends requests as Protobuf messages to
    Worker’s stdin
    ‣Bazel reads responses as Protobuf messages from
    Worker’s stdout
    ‣PW is useful for JVM-based or other tools that
    run faster when warmed up

    View Slide

  151. Design of Bazel: Execution
    ‣Persistent Workers
    ‣Simple Request/Response Protocol between Bazel
    and a Worker
    ‣Bazel sends requests as Protobuf messages to
    Worker’s stdin
    ‣Bazel reads responses as Protobuf messages from
    Worker’s stdout
    ‣PW is useful for JVM-based or other tools that
    run faster when warmed up
    ‣Expect changes in parallelism and maybe +JSON

    View Slide

  152. Design of Bazel: Execution
    ‣Persistent Workers
    ‣Simple Request/Response Protocol between Bazel
    and a Worker
    ‣Bazel sends requests as Protobuf messages to
    Worker’s stdin
    ‣Bazel reads responses as Protobuf messages from
    Worker’s stdout
    ‣PW is useful for JVM-based or other tools that
    run faster when warmed up
    ‣Expect changes in parallelism and maybe +JSON
    ‣In theory, PW can be used for actual incremental
    compilation since it keeps state

    View Slide

  153. Design of Bazel: Execution
    ‣Persistent Workers
    https:"//blog.bazel.build/2015/12/10/java-workers.html

    View Slide

  154. Design of Bazel: Execution
    https:"//github.com/artem-zinnatullin/error-prone-performance
    ‣Persistent Workers
    Gradle with Javac accidentally class loaded each time

    View Slide

  155. Design of Bazel: Execution
    ‣Sandboxing
    https:"//blog.bazel.build/2015/09/11/sandboxing.html

    View Slide

  156. Design of Bazel: Execution
    ‣Sandboxing
    ‣Another important piece of reproducible
    builds

    View Slide

  157. Design of Bazel: Execution
    ‣Sandboxing
    ‣Another important piece of reproducible
    builds
    ‣Isolates filesystem and process environment
    for build actions similarly to containers

    View Slide

  158. Design of Bazel: Execution
    ‣Sandboxing
    ‣Another important piece of reproducible
    builds
    ‣Isolates filesystem and process environment
    for build actions similarly to containers
    ‣Supports Linux and macOS, Windows WIP

    View Slide

  159. Design of Bazel: Execution
    ‣Sandboxing
    ‣Another important piece of reproducible
    builds
    ‣Isolates filesystem and process environment
    for build actions similarly to containers
    ‣Supports Linux and macOS, Windows WIP
    ‣macOS performance is not great yet (we saw
    30% degradation)

    View Slide

  160. Design of Bazel: Execution
    Remote Cache
    https:"//docs.bazel.build/versions/master/remote-caching.html

    View Slide

  161. Design of Bazel: Execution

    View Slide

  162. Design of Bazel: Execution
    ‣Remote Cache
    ‣Action Cache
    ‣Output Cache
    ‣Has amazing optimization “Remote Builds
    without the Bytes”

    View Slide

  163. Design of Bazel: Execution
    Remote Execution
    https:"//docs.bazel.build/versions/master/remote-execution.html

    View Slide

  164. Design of Bazel: Execution
    Remote Execution
    https:"//docs.bazel.build/versions/master/remote-execution.html

    View Slide

  165. Design of Bazel: Execution
    Remote
    Execution
    https:"//docs.bazel.build/versions/master/remote-execution.html

    View Slide

  166. Design of Bazel: Execution
    Remote
    Execut

    View Slide

  167. Design of Bazel: Execution

    View Slide

  168. Design of Bazel: Execution
    ‣Remote Bazel Execution

    View Slide

  169. Design of Bazel: Execution
    ‣Remote Bazel Execution
    -Acts as Remote Cache too

    View Slide

  170. Design of Bazel: Execution
    ‣Remote Bazel Execution
    -Acts as Remote Cache too
    -Automatic selection of execution strategy
    (local vs remote)

    View Slide

  171. Design of Bazel: Execution
    ‣Remote Bazel Execution
    -Acts as Remote Cache too
    -Automatic selection of execution strategy
    (local vs remote)
    -Remote environment for actions with Docker
    images

    View Slide

  172. Design of Bazel: Execution
    ‣Remote Bazel Execution
    -Acts as Remote Cache too
    -Automatic selection of execution strategy
    (local vs remote)
    -Remote environment for actions with Docker
    images
    -Virtually indefinitely parallelism

    View Slide

  173. Design of Bazel: Execution
    ‣Remote Bazel Execution
    -Acts as Remote Cache too
    -Automatic selection of execution strategy
    (local vs remote)
    -Remote environment for actions with Docker
    images
    -Virtually indefinitely parallelism
    -Google Cloud provides it as service yo
    (): RBE

    View Slide

  174. Design of Bazel: Execution
    ‣Mobile Install
    https:"//docs.bazel.build/versions/master/mobile-install.html

    View Slide

  175. Design of Bazel: Execution
    ‣Mobile Install
    -Like “Apply Changes” but made years ago
    https:"//docs.bazel.build/versions/master/mobile-install.html

    View Slide

  176. Design of Bazel: Execution
    ‣Mobile Install
    -Like “Apply Changes” but made years ago
    -Works by installing 2 apps:
    • Stub app with custom application class to
    load code & resources from “Contents app”
    • "Contents app" with actual code and resources
    https:"//docs.bazel.build/versions/master/mobile-install.html

    View Slide

  177. Design of Bazel: Execution
    ‣Mobile Install
    -Like “Apply Changes” but made years ago
    -Works by installing 2 apps:
    • Stub app with custom application class to
    load code & resources from “Contents app”
    • "Contents app" with actual code and resources
    -Incrementally updates dex files and resources
    in “Contents app”
    https:"//docs.bazel.build/versions/master/mobile-install.html

    View Slide

  178. Design of Bazel: Execution
    ‣Mobile Install
    -Like “Apply Changes” but made years ago
    -Works by installing 2 apps:
    • Stub app with custom application class to
    load code & resources from “Contents app”
    • "Contents app" with actual code and resources
    -Incrementally updates dex files and resources
    in “Contents app”
    -Has few corner cases and limitations
    https:"//docs.bazel.build/versions/master/mobile-install.html

    View Slide

  179. Design of Bazel: Execution
    https:"//docs.bazel.build/versions/master/mobile-install.html

    View Slide

  180. Design of Bazel:
    IDE Integration

    View Slide

  181. Design of Bazel:
    IDE Integration

    View Slide

  182. Design of Bazel:
    IDE Integration

    View Slide

  183. Design of Bazel: IDE Integration

    View Slide

  184. Design of Bazel: IDE Integration
    -IDE integration uses Bazel CLI commands

    View Slide

  185. Design of Bazel: IDE Integration
    -IDE integration uses Bazel CLI commands
    -IDE queries Bazel about the project
    (bazel query …)

    View Slide

  186. Design of Bazel: IDE Integration
    -IDE integration uses Bazel CLI commands
    -IDE queries Bazel about the project
    (bazel query …)
    -Supports Partial Sync for specific files,
    dirs

    View Slide

  187. Design of Bazel: IDE Integration
    TODO: screenshot of Android Studio Bazel Sync

    View Slide

  188. Design of Bazel: IDE Integration
    ‣IntelliJ Family
    -IntelliJ
    -Android Studio
    -CLion
    https:"//docs.bazel.build/versions/master/ide.html

    View Slide

  189. Design of Bazel: IDE Integration
    ‣Xcode
    -Tulsi and XCHammer to generate Xcode
    project from Bazel project
    -Lyft is working on Focus tool to limit
    number of loaded modules
    -lyft/index-import to import swift and clang
    index files produced as side effect of
    compilation
    https:"//docs.bazel.build/versions/master/ide.html

    View Slide

  190. Design of Bazel:
    Integration w/
    Other Tools

    View Slide

  191. Design of Bazel: Integration w/ Other Tools
    ‣Genrule
    -Run arbitrary CLI tool, Bash, Powershell,
    etc
    -As any other rule, inputs and outputs are
    tracked
    -Sandboxing makes it easier to achieve
    reproducibility
    https:"//docs.bazel.build/versions/master/be/general.html#genrule

    View Slide

  192. Design of Bazel: Integration w/ Other Tools
    genrule(
    name = "mygenrule",
    cmd = "mkdir $(RULEDIR)/mydir; javac -d $(OUTS) $(SRCS)",
    outs = [ "mydir" ],
    srcs = [ "B.java" ],
    )
    https:"//docs.bazel.build/versions/master/be/general.html#genrule

    View Slide

  193. Design of Bazel: Integration w/ Other Tools
    ‣Toolchains and Platforms
    -Bazel has notion of Target and Execution
    Platforms
    -Toolchain defines key/value pairs per Platform
    -Toolchain typically represents a set of
    Tooling for a language
    -Compiler
    -Linker
    -etc
    https:"//docs.bazel.build/versions/master/toolchains.html

    View Slide

  194. Design of Bazel: Integration w/ Other Tools
    ‣Extensions
    -.bzl files written in Starlark with macros,
    rules, constants
    -load(“#//mybazelextensions:file.bzl”,
    “myrule_or_macro”)
    -One of main Bazel 1.0 goals is to extract
    impls from core to external rules repos
    https:"//docs.bazel.build/versions/master/skylark/concepts.html

    View Slide

  195. Design of Bazel: Integration w/ Other Tools
    ‣Extensions

    View Slide

  196. Design of Bazel: Integration w/ Other Tools
    ‣Extensions
    github.com/bazelbuild/

    View Slide

  197. Design of Bazel: Integration w/ Other Tools
    ‣Extensions
    github.com/bazelbuild/
    -rules_java

    View Slide

  198. Design of Bazel: Integration w/ Other Tools
    ‣Extensions
    github.com/bazelbuild/
    -rules_java
    -rules_android

    View Slide

  199. Design of Bazel: Integration w/ Other Tools
    ‣Extensions
    github.com/bazelbuild/
    -rules_java
    -rules_android
    -rules_kotlin

    View Slide

  200. Design of Bazel: Integration w/ Other Tools
    ‣Extensions
    github.com/bazelbuild/
    -rules_java
    -rules_android
    -rules_kotlin
    -rules_jvm_external

    View Slide

  201. Design of Bazel: Integration w/ Other Tools
    ‣Extensions
    github.com/bazelbuild/
    -rules_java
    -rules_android
    -rules_kotlin
    -rules_jvm_external
    -rules_apple

    View Slide

  202. Design of Bazel: Integration w/ Other Tools
    ‣Extensions
    github.com/bazelbuild/
    -rules_java
    -rules_android
    -rules_kotlin
    -rules_jvm_external
    -rules_apple
    -etc

    View Slide

  203. Design of Bazel: Integration w/ Other Tools
    ‣Extensions
    github.com/bazelbuild/
    -rules_java
    -rules_android
    -rules_kotlin
    -rules_jvm_external
    -rules_apple
    -etc
    ‣github.com/you/rules_x !

    View Slide

  204. Bazel
    Q&A: Open https:!//sli.do and
    enter code “bazel”

    View Slide

  205. Artem Zinnatullin
    ‣ @artem_zin
    ‣ artemzin.com
    Come work with me!
    Also: Lyft Mobile Podcast
    tools, libraries, CI, builds @

    View Slide