Bazel

Talk was presented at Droidcon NYC 2019.

Why Bazel worth taking a look at and how it can help you scale development of your Android and iOS apps while (mostly) outperforming standard tooling like Gradle and xcodebuild.

In the talk we'll discuss following major topics:

- Multi-module setup vs build system
- Why Gradle is slow and how they're fixing it
- Why Gradle is fast and how they're fixing it
- ABI vs API vs implementation
- Compile avoidance
- Remote caching
- Remote execution
- Build configuration DSL (Groovy/Kotlin vs Starlark)
- IDE support
- CI/CD support
- Testing

Transcript

1. Bazel
   

   View Slide

2. Bazel
   

   View Slide

3. View Slide

4. Tacking a
   look at
   Bazel
   

   View Slide

5. Build System Use Cases
   

   View Slide

6. Build System Use Cases: User
   

   View Slide

7. Build System Use Cases: User
   

   View Slide

8. Build System
   Use Cases: User
   

   View Slide

9. Build System Use Cases: Automation
   

   View Slide

10. Build System Use Cases: Automation
    

    View Slide

11. Build System Use Cases: Plugins/Extensions
    

    View Slide

12. Build System Use Cases
    -Resolve dependencies
    

    View Slide

13. Build System Use Cases
    -Resolve dependencies
    -Build (compile, process resources, etc)
    

    View Slide

14. Build System Use Cases
    -Resolve dependencies
    -Build (compile, process resources, etc)
    -Run apps/tests
    

    View Slide

15. Build System Use Cases
    -Resolve dependencies
    -Build (compile, process resources, etc)
    -Run apps/tests
    -Integrate with IDE (modules, actions, etc)
    

    View Slide

16. Build System Use Cases
    -Resolve dependencies
    -Build (compile, process resources, etc)
    -Run apps/tests
    -Integrate with IDE (modules, actions, etc)
    -Integrate with scripts & CI
    

    View Slide

17. Build System Use Cases
    -Resolve dependencies
    -Build (compile, process resources, etc)
    -Run apps/tests
    -Integrate with IDE (modules, actions, etc)
    -Integrate with scripts & CI
    -Plugins/Extensions API
    

    View Slide

18. Build System Use Cases
    -Resolve dependencies
    -Build (compile, process resources, etc)
    -Run apps/tests
    -Integrate with IDE (modules, actions, etc)
    -Integrate with scripts & CI
    -Plugins/Extensions API
    -etc
    

    View Slide

19. Design of Bazel:
    Configuration
    

    View Slide

20. Design of Bazel: Configuration
    java_library(
    name = "a",
    srcs = [ "A.java" ],
    deps = [
    "//:b"
    ],
    )
    java_library(
    name = "b",
    srcs = [ "B.java" ],
    )
    Bazel BUILD file
    

    View Slide

21. Design of Bazel: Configuration
    ‣Readable
    

    View Slide

22. Design of Bazel: Configuration
    ‣Readable
    ‣Writeable
    

    View Slide

23. Design of Bazel: Configuration
    ‣Readable
    ‣Writeable
    ‣Maintainable
    

    View Slide

24. Design of Bazel: Configuration
    ‣Static Markdown Language
    

    View Slide

25. Design of Bazel: Configuration
    ‣Static Markdown Language
    -XML
    -JSON
    -YAML (no God please no)
    -TOML
    -etc
    

    View Slide

26. Design of Bazel: Configuration
    ‣Static Markdown Language
    -XML
    -JSON
    -YAML (no God please no)
    -TOML
    -etc
    ‣Programming Language
    

    View Slide

27. Design of Bazel: Configuration
    ‣Static Markdown Language
    -XML
    -JSON
    -YAML (no God please no)
    -TOML
    -etc
    ‣Programming Language
    -Dynamically Typed
    -Statically Typed
    -etc
    

    View Slide

28. Design of Bazel: Configuration
    
    4.0.0
    com.lyft.dagger1
    dagger-parent
    pom
    1.3.0
    Dagger (Parent)
    A fast dependency injector for Android and Java.
    https://github.com/square/dagger
    
    compiler
    core
    
    Maven pom.xml
    

    View Slide

29. Design of Bazel: Configuration
    apply plugin: 'com.android.library'
    apply plugin: 'kotlin-android'
    android {
    defaultConfig {
    minSdkVersion versions.minSdk
    compileSdkVersion versions.compileSdk
    buildToolsVersion versions.buildTools
    }
    }
    dependencies {
    api project(':redom:api')
    api androidXLibraries.recyclerView
    }
    Gradle build.gradle
    

    View Slide

30. Design of Bazel: Configuration
    java_library(
    name = "lib",
    srcs = glob(["src/main/java/**/*.java"]),
    deps = [
    "//:b",
    ]
    )
    java_test(
    name = "test",
    srcs = glob(["src/test/java/**/*.java"]),
    deps = [
    ":lib",
    ]
    )
    Bazel BUILD file
    

    View Slide

31. Design of Bazel: Configuration
    ‣Reproducibility
    

    View Slide

32. Design of Bazel: Configuration
    ‣Reproducibility
    -Limited stdlib
    

    View Slide

33. Design of Bazel: Configuration
    ‣Reproducibility
    -Limited stdlib
    -No untracked IO
    

    View Slide

34. Design of Bazel: Configuration
    ‣Reproducibility
    -Limited stdlib
    -No untracked IO
    -External deps need sha256
    

    View Slide

35. Design of Bazel: Configuration
    apply plugin: 'java-library'
    apply plugin: 'kotlin'
    // yoooooo don't do this
    version = Calendar.instance.get(Calendar.DAY_OF_WEEK) == 2
    ? System.currentTimeMillis()
    : new Random().nextInt()
    

    View Slide

36. Design of Bazel: Configuration
    ‣Gradle: Groovy
    

    View Slide

37. Design of Bazel: Configuration
    ‣Gradle: Groovy
    ‣Groovy syntax and dynamic type system ()
    

    View Slide

38. Design of Bazel: Configuration
    ‣Gradle: Groovy
    ‣Groovy syntax and dynamic type system ()
    ‣Unlimited mutability
    

    View Slide

39. Design of Bazel: Configuration
    ‣Gradle: Groovy
    ‣Groovy syntax and dynamic type system ()
    ‣Unlimited mutability
    ‣Unlimited IO
    

    View Slide

40. Design of Bazel: Configuration
    ‣Gradle: Groovy
    ‣Groovy syntax and dynamic type system ()
    ‣Unlimited mutability
    ‣Unlimited IO
    ‣Full JDK stdlib
    

    View Slide

41. Design of Bazel: Configuration
    ‣Gradle: Groovy
    ‣Groovy syntax and dynamic type system ()
    ‣Unlimited mutability
    ‣Unlimited IO
    ‣Full JDK stdlib
    ‣Can write Java in Groovy
    

    View Slide

42. View Slide

43. Design of Bazel: Configuration
    ‣Gradle: Kotlin
    

    View Slide

44. Design of Bazel: Configuration
    ‣Gradle: Kotlin
    ‣Kotlin syntax and static type system ()
    

    View Slide

45. Design of Bazel: Configuration
    ‣Gradle: Kotlin
    ‣Kotlin syntax and static type system ()
    ‣Unlimited mutability
    

    View Slide

46. Design of Bazel: Configuration
    ‣Gradle: Kotlin
    ‣Kotlin syntax and static type system ()
    ‣Unlimited mutability
    ‣Unlimited IO
    

    View Slide

47. Design of Bazel: Configuration
    ‣Gradle: Kotlin
    ‣Kotlin syntax and static type system ()
    ‣Unlimited mutability
    ‣Unlimited IO
    ‣Full JDK stdlib
    

    View Slide

48. Design of Bazel: Configuration
    See “Fundamental Design Issues of
    Gradle Build System”
    https:!//artemzin.com/blog/fundamental-design-issues-of-gradle-build-system/
    *includes proposals to solve the issues
    

    View Slide

49. Design of Bazel: Configuration
    Enter:
    Skylark
    Starlark
    

    View Slide

50. Design of Bazel: Configuration
    ‣Bazel Starlark Evaluation Model
    https:"//docs.bazel.build/versions/master/skylark/concepts.html#evaluation-model
    

    View Slide

51. Design of Bazel: Configuration
    ‣Bazel Starlark Evaluation Model
    -Loading Phase
    -Read BUILD and .bzl files required for given build
    -Bazel targets are directly mapped to file paths in the
    project
    https:"//docs.bazel.build/versions/master/skylark/concepts.html#evaluation-model
    

    View Slide

52. Design of Bazel: Configuration
    ‣Bazel Starlark Evaluation Model
    -Loading Phase
    -Read BUILD and .bzl files required for given build
    -Bazel targets are directly mapped to file paths in the
    project
    -Analysis Phase
    -Rules are executed
    -Rules instantiate actual Actions
    -Action graph is composed and validated
    https:"//docs.bazel.build/versions/master/skylark/concepts.html#evaluation-model
    

    View Slide

53. Design of Bazel: Configuration
    ‣Bazel Starlark Evaluation Model
    -Loading Phase
    -Read BUILD and .bzl files required for given build
    -Bazel targets are directly mapped to file paths in the
    project
    -Analysis Phase
    -Rules are executed
    -Rules instantiate actual Actions
    -Action graph is composed and validated
    -Execution Phase
    -Action graph is executed
    https:"//docs.bazel.build/versions/master/skylark/concepts.html#evaluation-model
    

    View Slide

54. Design of Bazel: Configuration
    ‣Starlark
    https:"//docs.bazel.build/versions/master/skylark/language.html
    

    View Slide

55. Design of Bazel: Configuration
    ‣Starlark
    ‣Python 3 syntax and dynamic type system ()
    https:"//docs.bazel.build/versions/master/skylark/language.html
    

    View Slide

56. Design of Bazel: Configuration
    ‣Starlark
    ‣Python 3 syntax and dynamic type system ()
    ‣Custom runtime
    https:"//docs.bazel.build/versions/master/skylark/language.html
    

    View Slide

57. Design of Bazel: Configuration
    ‣Starlark
    ‣Python 3 syntax and dynamic type system ()
    ‣Custom runtime
    ‣Custom stdlib
    https:"//docs.bazel.build/versions/master/skylark/language.html
    

    View Slide

58. Design of Bazel: Configuration
    ‣Starlark
    ‣Python 3 syntax and dynamic type system ()
    ‣Custom runtime
    ‣Custom stdlib
    ‣Extremely limited mutability
    https:"//docs.bazel.build/versions/master/skylark/language.html
    

    View Slide

59. Design of Bazel: Configuration
    ‣Starlark
    ‣Python 3 syntax and dynamic type system ()
    ‣Custom runtime
    ‣Custom stdlib
    ‣Extremely limited mutability
    ‣No IO
    https:"//docs.bazel.build/versions/master/skylark/language.html
    

    View Slide

60. Design of Bazel: Configuration
    ‣Starlark
    ‣Python 3 syntax and dynamic type system ()
    ‣Custom runtime
    ‣Custom stdlib
    ‣Extremely limited mutability
    ‣No IO
    ‣Bazel provides build & IO specific APIs
    https:"//docs.bazel.build/versions/master/skylark/language.html
    

    View Slide

61. Design of Bazel: Configuration
    ‣Starlark
    ‣Python 3 syntax and dynamic type system ()
    ‣Custom runtime
    ‣Custom stdlib
    ‣Extremely limited mutability
    ‣No IO
    ‣Bazel provides build & IO specific APIs
    ‣Can be used outside of Bazel!
    https:"//docs.bazel.build/versions/master/skylark/language.html
    

    View Slide

62. Design of Bazel: Configuration
    java_library(
    name = "lib",
    srcs = glob(["src/main/java/**/*.java"]),
    deps = [
    "//:b",
    ]
    )
    java_test(
    name = "test",
    srcs = glob(["src/test/java/**/*.java"]),
    deps = [
    ":lib",
    ]
    )
    Bazel BUILD file
    

    View Slide

63. Design of Bazel: Configuration
    java_library(
    name = "lib",
    srcs = glob(["src/main/java/**/*.java"]),
    deps = [
    "//:b",
    ]
    )
    java_test(
    name = "test",
    srcs = glob(["src/test/java/**/*.java"]),
    deps = [
    ":lib",
    ]
    )
    Bazel BUILD file Target with name “lib”
    

    View Slide

64. Design of Bazel: Configuration
    ‣Starlark
    https:"//docs.bazel.build/versions/master/skylark/language.html
    

    View Slide

65. Design of Bazel: Configuration
    ‣Starlark
    -Functions
    https:"//docs.bazel.build/versions/master/skylark/language.html
    

    View Slide

66. Design of Bazel: Configuration
    ‣Starlark
    -Functions
    - Rules
    https:"//docs.bazel.build/versions/master/skylark/language.html
    

    View Slide

67. Design of Bazel: Configuration
    ‣Starlark
    -Functions
    - Rules
    - Macros
    https:"//docs.bazel.build/versions/master/skylark/language.html
    

    View Slide

68. Design of Bazel: Configuration
    ‣Starlark
    -Functions
    - Rules
    - Macros
    -Variables
    - Mutable within a “context”
    - Frozen once context has been evaluated
    https:"//docs.bazel.build/versions/master/skylark/language.html
    

    View Slide

69. Design of Bazel: Configuration
    ‣Rule
    https:"//docs.bazel.build/versions/master/skylark/rules.html
    

    View Slide

70. Design of Bazel: Configuration
    ‣Rule
    -Function that performs reproducible IO for given Inputs
    https:"//docs.bazel.build/versions/master/skylark/rules.html
    

    View Slide

71. Design of Bazel: Configuration
    ‣Rule
    -Function that performs reproducible IO for given Inputs
    -Declares set of public and private attributes
    (parameters)
    https:"//docs.bazel.build/versions/master/skylark/rules.html
    

    View Slide

72. Design of Bazel: Configuration
    ‣Rule
    -Function that performs reproducible IO for given Inputs
    -Declares set of public and private attributes
    (parameters)
    -Examples: kotlin_library, java_test, etc
    https:"//docs.bazel.build/versions/master/skylark/rules.html
    

    View Slide

73. Design of Bazel: Configuration
    ‣Rule
    -Function that performs reproducible IO for given Inputs
    -Declares set of public and private attributes
    (parameters)
    -Examples: kotlin_library, java_test, etc
    -Cannot perform IO during configuration/analysis
    https:"//docs.bazel.build/versions/master/skylark/rules.html
    

    View Slide

74. Design of Bazel: Configuration
    ‣Rule
    -Function that performs reproducible IO for given Inputs
    -Declares set of public and private attributes
    (parameters)
    -Examples: kotlin_library, java_test, etc
    -Cannot perform IO during configuration/analysis
    -Declares list of actions to perform IO during execution
    https:"//docs.bazel.build/versions/master/skylark/rules.html
    

    View Slide

75. Design of Bazel: Configuration
    ‣Rule
    -Function that performs reproducible IO for given Inputs
    -Declares set of public and private attributes
    (parameters)
    -Examples: kotlin_library, java_test, etc
    -Cannot perform IO during configuration/analysis
    -Declares list of actions to perform IO during execution
    -Exposes Provider(s) — rule’s public API for other rules
    https:"//docs.bazel.build/versions/master/skylark/rules.html
    

    View Slide

76. Design of Bazel: Configuration
    android_instrumentation_test = rule(
    attrs = attrs.add(
    ANDROID_INSTRUMENTATION_TEST_ATTRS,
    ADDON_ATTRS
    ),
    fragments = ["android"],
    implementation = _impl,
    outputs = _outputs,
    test = True,
    )
    https:"//github.com/bazelbuild/rules_android
    

    View Slide

77. Design of Bazel: Configuration
    https:"//github.com/bazelbuild/rules_android
    def _impl(ctx):
    # Main entry point into the Android test runner.
    test_runner = get_test_runner(ctx)
    ………
    # TODO(str): remove after android_test migration
    if ctx.attr._android_test_migration:
    # Test APK is a predeclared output on android_test
    ctx.actions.run_shell(
    inputs = [ctx.attr.test_app[ApkInfo].signed_apk],
    outputs = [ctx.outputs.test_app],
    command = "cp %s %s" % (ctx.attr.test_app[ApkInfo].signed_apk.path,
    ctx.outputs.test_app.path),
    )
    runfiles = runfiles.merge(ctx.runfiles(files = [ctx.outputs.test_app]))
    apks_to_install = []
    

    View Slide

78. Design of Bazel: Configuration
    ‣Macro
    https:"//docs.bazel.build/versions/master/skylark/macros.html
    

    View Slide

79. Design of Bazel: Configuration
    ‣Macro
    -Can instantiate rule(s)
    https:"//docs.bazel.build/versions/master/skylark/macros.html
    

    View Slide

80. Design of Bazel: Configuration
    ‣Macro
    -Can instantiate rule(s)
    -Useful for code reuse, composing logic, default
    values
    https:"//docs.bazel.build/versions/master/skylark/macros.html
    

    View Slide

81. Design of Bazel: Configuration
    ‣Macro
    -Can instantiate rule(s)
    -Useful for code reuse, composing logic, default
    values
    -Present only during Loading phase
    https:"//docs.bazel.build/versions/master/skylark/macros.html
    

    View Slide

82. Design of Bazel: Configuration
    ‣Macro
    -Can instantiate rule(s)
    -Useful for code reuse, composing logic, default
    values
    -Present only during Loading phase
    -Examples: lyft_kotlin_library,
    lyft_android_binary, etc
    https:"//docs.bazel.build/versions/master/skylark/macros.html
    

    View Slide

83. Design of Bazel: Configuration
    def lyft_android_library(**kwargs):
    

    View Slide

84. Design of Bazel: Configuration
    def lyft_android_library(**kwargs):
    if kwargs.get("language") == "kotlin":
    _apply_detekt(**kwargs)
    

    View Slide

85. Design of Bazel: Configuration
    def lyft_android_library(**kwargs):
    if kwargs.get("language") == "kotlin":
    _apply_detekt(**kwargs)
    _apply_pmd(**kwargs)
    

    View Slide

86. Design of Bazel: Configuration
    def lyft_android_library(**kwargs):
    if kwargs.get("language") == "kotlin":
    _apply_detekt(**kwargs)
    _apply_pmd(**kwargs)
    kwargs = _apply_error_prone(**kwargs)
    

    View Slide

87. Design of Bazel: Configuration
    def lyft_android_library(**kwargs):
    if kwargs.get("language") == "kotlin":
    _apply_detekt(**kwargs)
    _apply_pmd(**kwargs)
    kwargs = _apply_error_prone(**kwargs)
    kwargs = _apply_java_null_pkg_info_gen(
    **kwargs
    )
    

    View Slide

88. Design of Bazel: Configuration
    def lyft_android_library(**kwargs):
    if kwargs.get("language") == "kotlin":
    _apply_detekt(**kwargs)
    _apply_pmd(**kwargs)
    kwargs = _apply_error_prone(**kwargs)
    kwargs = _apply_java_nullability_pkg_info_generator(**kwargs)
    android_library(**kwargs)
    

    View Slide

89. Design of Bazel: Configuration
    Starlark vs multi-module
    projects
    

    View Slide

90. Design of Bazel: Configuration
    ‣Parallelism
    

    View Slide

91. Design of Bazel: Configuration
    ‣Parallelism
    -Limited mutability lets configure packages
    independently
    

    View Slide

92. Design of Bazel: Configuration
    ‣Parallelism
    -Limited mutability lets configure packages
    independently
    -Lack of IO makes configuration CPU bound
    

    View Slide

93. Design of Bazel: Configuration
    ‣Cache-ability
    -Lack of mutability and IO makes
    configuration cacheable
    

    View Slide

94. Design of Bazel: Configuration
    ‣Configuration Avoidance
    -Bazel only evaluates BUILD and .bzl files
    required for the build command
    

    View Slide

95. Design of Bazel: Configuration
    https:"//artemzin.com/blog/fundamental-design-issues-of-gradle-build-system/
    Gradle: config time / number of modules
    Seconds
    0
    6.5
    13
    19.5
    26
    Number of modules
    400 600 800 1000
    10
    15
    20
    25
    

    View Slide

96. Design of Bazel: Configuration
    Gradle is working on Configuration Caching!
    

    View Slide

97. Design of Bazel: Configuration
    ‣Visibility
    -Defines what packages or targets are
    allowed to directly depend on this target
    https:"//docs.bazel.build/versions/master/be/common-definitions.html#common-attributes
    

    View Slide

98. Design of Bazel: Configuration
    java_library(
    name = "a",
    srcs = [ "A.java" ],
    deps = [
    "//:b"
    ],
    visibility = [ "//visibility:public" ]
    )
    java_library(
    name = "b",
    srcs = [ "B.java" ],
    visibility = [ "//myfriendlypkg:lib" ]
    )
    

    View Slide

99. Design of Bazel: Configuration
    ‣Maintainability
    

    View Slide

100. Design of Bazel: Configuration
     ‣Maintainability
     ‣Buildifier — tool for formatting & linting
     BUILD and .bzl files
     https:"//github.com/bazelbuild/buildtools
     

     View Slide

101. Design of Bazel: Configuration
     ‣Maintainability
     ‣Buildifier — tool for formatting & linting
     BUILD and .bzl files
     ‣Buildozer — tool for mass refactoring of
     BUILD files
     https:"//github.com/bazelbuild/buildtools
     

     View Slide

102. Design of Bazel: Configuration
     ‣Maintainability
     ‣Buildifier — tool for formatting & linting
     BUILD and .bzl files
     ‣Buildozer — tool for mass refactoring of
     BUILD files
     ‣Both written in Go, run pretty fast
     https:"//github.com/bazelbuild/buildtools
     

     View Slide

103. Design of Bazel: Configuration
     java_library(
     name = "a",
     srcs = ["A.java"],
     deps = ["//:b"],
     )
     Before:
     

     View Slide

104. Design of Bazel: Configuration
     java_library(
     name = "a",
     srcs = ["A.java"],
     deps = ["//:b"],
     )
     Before:
     

     View Slide

105. Design of Bazel: Configuration
     java_library(
     name = "a",
     srcs = ["A.java"],
     deps = ["//:b"],
     )
     java_library(
     name = "a",
     srcs = ["A.java"],
     deps = ["//:b"],
     )
     Before:
     After:
     

     View Slide

106. Design of Bazel: Configuration
     java_library(
     name = "a",
     srcs = ["A.java"],
     deps = ["//:b"],
     )
     https:"//github.com/bazelbuild/buildtools/blob/master/buildozer/README.md
     

     View Slide

107. Design of Bazel: Configuration
     java_library(
     name = "a",
     srcs = ["A.java"],
     deps = ["//:b"],
     )
     https:"//github.com/bazelbuild/buildtools/blob/master/buildozer/README.md
     

     View Slide

108. Design of Bazel: Configuration
     java_library(
     name = "a",
     srcs = ["A.java"],
     deps = ["//:b"],
     )
     lyft_java_library(
     name = "a",
     srcs = ["A.java"],
     deps = ["//:b"],
     )
     https:"//github.com/bazelbuild/buildtools/blob/master/buildozer/README.md
     

     View Slide

109. Design of Bazel:
     Execution
     

     View Slide

110. Design of Bazel: Execution
     

     View Slide

111. Design of Bazel: Execution
     ‣Commands
     Command Purpose
     bazel query What Targets are there
     https:"//docs.bazel.build/versions/master/command-line-reference.html
     

     View Slide

112. Design of Bazel: Execution
     ‣Commands
     Command Purpose
     bazel query What Targets are there
     bazel build Build one or more Targets
     https:"//docs.bazel.build/versions/master/command-line-reference.html
     

     View Slide

113. Design of Bazel: Execution
     ‣Commands
     Command Purpose
     bazel query What Targets are there
     bazel build Build one or more Targets
     bazel test Run one or more Test Targets
     https:"//docs.bazel.build/versions/master/command-line-reference.html
     

     View Slide

114. Design of Bazel: Execution
     ‣Commands
     Command Purpose
     bazel query What Targets are there
     bazel build Build one or more Targets
     bazel test Run one or more Test Targets
     bazel run Run one or more Run Targets
     https:"//docs.bazel.build/versions/master/command-line-reference.html
     

     View Slide

115. Design of Bazel: Execution
     ‣Incrementability
     

     View Slide

116. Design of Bazel: Execution
     ‣Incrementability
     ‣Bazel won’t do what has already been done
     (duh)
     

     View Slide

117. Design of Bazel: Execution
     ‣Incrementability
     ‣Bazel won’t do what has already been done
     (duh)
     ‣Compilation is not incremental
     

     View Slide

118. Design of Bazel: Execution
     ‣Incrementability
     ‣Bazel won’t do what has already been done
     (duh)
     ‣Compilation is not incremental
     ‣Tests results are also cached (you can opt
     out)
     

     View Slide

119. Design of Bazel: Execution
     

     View Slide

120. Design of Bazel: Execution
     ‣Parallelism
     ‣Bazel by default runs what it can in
     parallel
     

     View Slide

121. Design of Bazel: Execution
     ‣ABI/header Extraction
     

     View Slide

122. Design of Bazel: Execution
     ‣ABI/header Extraction
     -Bazel computes ABI for supported languages
     and uses it
     

     View Slide

123. Design of Bazel: Execution
     

     View Slide

124. Design of Bazel: Execution
     

     View Slide

125. Design of Bazel: Execution
     

     View Slide

126. Design of Bazel: Execution
     public class MyClass {
     private String x;
     public void a() {
     this.x = b();
     }
     private String b() {
     Random random = new Random();
     StringBuilder str = new StringBuilder();
     for (int i = 0; i < random.nextInt(1024); i++) {
     str.append(i);
     }
     return str.toString();
     }
     public static Runnable c(String s) {
     return () -> {
     new MyClass().b();
     };
     }
     }
     MyClass.java
     

     View Slide

127. Design of Bazel: Execution
     public class MyClass {
     private String x;
     public void a() {
     this.x = b();
     }
     private String b() {
     Random random = new Random();
     StringBuilder str = new StringBuilder();
     for (int i = 0; i < random.nextInt(1024); i++) {
     str.append(i);
     }
     return str.toString();
     }
     public static Runnable c(String s) {
     return () -> {
     new MyClass().b();
     };
     }
     MyClass.java
     

     View Slide

128. Design of Bazel: Execution
     public class MyClass {
     private String x;
     public void a() {
     this.x = b();
     }
     private String b() {
     Random random = new Random();
     StringBuilder str = new StringBuilder();
     for (int i = 0; i < random.nextInt(1024); i++) {
     str.append(i);
     }
     return str.toString();
     }
     public static Runnable c(String s) {
     return () -> {
     new MyClass().b();
     };
     }
     MyClass.java
     

     View Slide

129. Design of Bazel: Execution
     public class MyClass {
     private String x;
     public void a() {
     this.x = b();
     }
     private String b() {
     Random random = new Random();
     StringBuilder str = new StringBuilder();
     for (int i = 0; i < random.nextInt(1024); i++) {
     str.append(i);
     }
     return str.toString();
     }
     public static Runnable c(String s) {
     return () -> {
     new MyClass().b();
     };
     }
     MyClass.java
     

     View Slide

130. Design of Bazel: Execution
     public class MyClass {
     private String x;
     public void a() {
     this.x = b();
     }
     private String b() {
     Random random = new Random();
     StringBuilder str = new StringBuilder();
     for (int i = 0; i < random.nextInt(1024); i++) {
     str.append(i);
     }
     return str.toString();
     }
     public static Runnable c(String s) {
     return () -> {
     new MyClass().b();
     };
     }
     MyClass.java
     

     View Slide

131. Design of Bazel: Execution
     public class MyClass {
     public void a() {
     this.x = b();
     }
     private String b() {
     Random random = new Random();
     StringBuilder str = new StringBuilder();
     for (int i = 0; i < random.nextInt(1024); i++) {
     str.append(i);
     }
     return str.toString();
     }
     public static Runnable c(String s) {
     return () -> {
     new MyClass().b();
     };
     }
     }
     MyClass.java
     

     View Slide

132. Design of Bazel: Execution
     public class MyClass {
     public void a() {
     this.x = b();
     }
     private String b() {
     Random random = new Random();
     StringBuilder str = new StringBuilder();
     for (int i = 0; i < random.nextInt(1024); i++) {
     str.append(i);
     }
     return str.toString();
     }
     public static Runnable c(String s) {
     return () -> {
     new MyClass().b();
     };
     }
     MyClass.java
     

     View Slide

133. Design of Bazel: Execution
     public class MyClass {
     public void a() {
     this.x = b();
     }
     private String b() {
     Random random = new Random();
     StringBuilder str = new StringBuilder();
     for (int i = 0; i < random.nextInt(1024); i++) {
     str.append(i);
     }
     return str.toString();
     }
     public static Runnable c(String s) {
     return () -> {
     new MyClass().b();
     };
     }
     MyClass.java
     

     View Slide

134. Design of Bazel: Execution
     public class MyClass {
     public void a();
     private String b() {
     Random random = new Random();
     StringBuilder str = new StringBuilder();
     for (int i = 0; i < random.nextInt(1024); i++) {
     str.append(i);
     }
     return str.toString();
     }
     public static Runnable c(String s) {
     return () -> {
     new MyClass().b();
     };
     }
     }
     MyClass.java
     

     View Slide

135. Design of Bazel: Execution
     public class MyClass {
     public void a();
     private String b() {
     Random random = new Random();
     StringBuilder str = new StringBuilder();
     for (int i = 0; i < random.nextInt(1024); i++) {
     str.append(i);
     }
     return str.toString();
     }
     public static Runnable c(String s) {
     return () -> {
     new MyClass().b();
     };
     }
     }
     MyClass.java
     

     View Slide

136. Design of Bazel: Execution
     public class MyClass {
     public void a();
     private String b() {
     Random random = new Random();
     StringBuilder str = new StringBuilder();
     for (int i = 0; i < random.nextInt(1024); i++) {
     str.append(i);
     }
     return str.toString();
     }
     public static Runnable c(String s) {
     return () -> {
     new MyClass().b();
     };
     }
     }
     MyClass.java
     

     View Slide

137. Design of Bazel: Execution
     public class MyClass {
     public void a();
     private String b() {
     Random random = new Random();
     StringBuilder str = new StringBuilder();
     for (int i = 0; i < random.nextInt(1024); i++) {
     str.append(i);
     }
     return str.toString();
     }
     public static Runnable c(String s) {
     return () -> {
     new MyClass().b();
     };
     }
     }
     MyClass.java
     

     View Slide

138. Design of Bazel: Execution
     public class MyClass {
     public void a();
     public static Runnable c(String s) {
     return () -> {
     new MyClass().b();
     };
     }
     }
     MyClass.java
     

     View Slide

139. Design of Bazel: Execution
     public class MyClass {
     public void a();
     public static Runnable c(String s) {
     return () -> {
     new MyClass().b();
     };
     }
     }
     MyClass.java
     

     View Slide

140. Design of Bazel: Execution
     public class MyClass {
     public void a();
     public static Runnable c(String s) {
     return () -> {
     new MyClass().b();
     };
     }
     }
     MyClass.java
     

     View Slide

141. Design of Bazel: Execution
     public class MyClass {
     public void a();
     public static Runnable c(String s);
     }
     

     View Slide

142. Design of Bazel: Execution
     public class MyClass {
     public void a();
     public static Runnable c(String s);
     }
     MyClass_Header.java
     (actually, it’s .class right away)
     

     View Slide

143. Design of Bazel: Execution
     

     View Slide

144. Design of Bazel: Execution
     ABI/Header extraction
     Works for entire
     action graph
     

     View Slide

145. Design of Bazel: Execution
     

     View Slide

146. Design of Bazel: Execution
     ‣Persistent Workers
     https:"//blog.bazel.build/2015/12/10/java-workers.html
     

     View Slide

147. Design of Bazel: Execution
     ‣Persistent Workers
     ‣Simple Request/Response Protocol between Bazel
     and a Worker
     

     View Slide

148. Design of Bazel: Execution
     ‣Persistent Workers
     ‣Simple Request/Response Protocol between Bazel
     and a Worker
     ‣Bazel sends requests as Protobuf messages to
     Worker’s stdin
     

     View Slide

149. Design of Bazel: Execution
     ‣Persistent Workers
     ‣Simple Request/Response Protocol between Bazel
     and a Worker
     ‣Bazel sends requests as Protobuf messages to
     Worker’s stdin
     ‣Bazel reads responses as Protobuf messages from
     Worker’s stdout
     

     View Slide

150. Design of Bazel: Execution
     ‣Persistent Workers
     ‣Simple Request/Response Protocol between Bazel
     and a Worker
     ‣Bazel sends requests as Protobuf messages to
     Worker’s stdin
     ‣Bazel reads responses as Protobuf messages from
     Worker’s stdout
     ‣PW is useful for JVM-based or other tools that
     run faster when warmed up
     

     View Slide

151. Design of Bazel: Execution
     ‣Persistent Workers
     ‣Simple Request/Response Protocol between Bazel
     and a Worker
     ‣Bazel sends requests as Protobuf messages to
     Worker’s stdin
     ‣Bazel reads responses as Protobuf messages from
     Worker’s stdout
     ‣PW is useful for JVM-based or other tools that
     run faster when warmed up
     ‣Expect changes in parallelism and maybe +JSON
     

     View Slide

152. Design of Bazel: Execution
     ‣Persistent Workers
     ‣Simple Request/Response Protocol between Bazel
     and a Worker
     ‣Bazel sends requests as Protobuf messages to
     Worker’s stdin
     ‣Bazel reads responses as Protobuf messages from
     Worker’s stdout
     ‣PW is useful for JVM-based or other tools that
     run faster when warmed up
     ‣Expect changes in parallelism and maybe +JSON
     ‣In theory, PW can be used for actual incremental
     compilation since it keeps state
     

     View Slide

153. Design of Bazel: Execution
     ‣Persistent Workers
     https:"//blog.bazel.build/2015/12/10/java-workers.html
     

     View Slide

154. Design of Bazel: Execution
     https:"//github.com/artem-zinnatullin/error-prone-performance
     ‣Persistent Workers
     Gradle with Javac accidentally class loaded each time
     

     View Slide

155. Design of Bazel: Execution
     ‣Sandboxing
     https:"//blog.bazel.build/2015/09/11/sandboxing.html
     

     View Slide

156. Design of Bazel: Execution
     ‣Sandboxing
     ‣Another important piece of reproducible
     builds
     

     View Slide

157. Design of Bazel: Execution
     ‣Sandboxing
     ‣Another important piece of reproducible
     builds
     ‣Isolates filesystem and process environment
     for build actions similarly to containers
     

     View Slide

158. Design of Bazel: Execution
     ‣Sandboxing
     ‣Another important piece of reproducible
     builds
     ‣Isolates filesystem and process environment
     for build actions similarly to containers
     ‣Supports Linux and macOS, Windows WIP
     

     View Slide

159. Design of Bazel: Execution
     ‣Sandboxing
     ‣Another important piece of reproducible
     builds
     ‣Isolates filesystem and process environment
     for build actions similarly to containers
     ‣Supports Linux and macOS, Windows WIP
     ‣macOS performance is not great yet (we saw
     30% degradation)
     

     View Slide

160. Design of Bazel: Execution
     Remote Cache
     https:"//docs.bazel.build/versions/master/remote-caching.html
     

     View Slide

161. Design of Bazel: Execution
     

     View Slide

162. Design of Bazel: Execution
     ‣Remote Cache
     ‣Action Cache
     ‣Output Cache
     ‣Has amazing optimization “Remote Builds
     without the Bytes”
     

     View Slide

163. Design of Bazel: Execution
     Remote Execution
     https:"//docs.bazel.build/versions/master/remote-execution.html
     

     View Slide

164. Design of Bazel: Execution
     Remote Execution
     https:"//docs.bazel.build/versions/master/remote-execution.html
     

     View Slide

165. Design of Bazel: Execution
     Remote
     Execution
     https:"//docs.bazel.build/versions/master/remote-execution.html
     

     View Slide

166. Design of Bazel: Execution
     Remote
     Execut
     

     View Slide

167. Design of Bazel: Execution
     

     View Slide

168. Design of Bazel: Execution
     ‣Remote Bazel Execution
     

     View Slide

169. Design of Bazel: Execution
     ‣Remote Bazel Execution
     -Acts as Remote Cache too
     

     View Slide

170. Design of Bazel: Execution
     ‣Remote Bazel Execution
     -Acts as Remote Cache too
     -Automatic selection of execution strategy
     (local vs remote)
     

     View Slide

171. Design of Bazel: Execution
     ‣Remote Bazel Execution
     -Acts as Remote Cache too
     -Automatic selection of execution strategy
     (local vs remote)
     -Remote environment for actions with Docker
     images
     

     View Slide

172. Design of Bazel: Execution
     ‣Remote Bazel Execution
     -Acts as Remote Cache too
     -Automatic selection of execution strategy
     (local vs remote)
     -Remote environment for actions with Docker
     images
     -Virtually indefinitely parallelism
     

     View Slide

173. Design of Bazel: Execution
     ‣Remote Bazel Execution
     -Acts as Remote Cache too
     -Automatic selection of execution strategy
     (local vs remote)
     -Remote environment for actions with Docker
     images
     -Virtually indefinitely parallelism
     -Google Cloud provides it as service yo
     (): RBE
     

     View Slide

174. Design of Bazel: Execution
     ‣Mobile Install
     https:"//docs.bazel.build/versions/master/mobile-install.html
     

     View Slide

175. Design of Bazel: Execution
     ‣Mobile Install
     -Like “Apply Changes” but made years ago
     https:"//docs.bazel.build/versions/master/mobile-install.html
     

     View Slide

176. Design of Bazel: Execution
     ‣Mobile Install
     -Like “Apply Changes” but made years ago
     -Works by installing 2 apps:
     • Stub app with custom application class to
     load code & resources from “Contents app”
     • "Contents app" with actual code and resources
     https:"//docs.bazel.build/versions/master/mobile-install.html
     

     View Slide

177. Design of Bazel: Execution
     ‣Mobile Install
     -Like “Apply Changes” but made years ago
     -Works by installing 2 apps:
     • Stub app with custom application class to
     load code & resources from “Contents app”
     • "Contents app" with actual code and resources
     -Incrementally updates dex files and resources
     in “Contents app”
     https:"//docs.bazel.build/versions/master/mobile-install.html
     

     View Slide

178. Design of Bazel: Execution
     ‣Mobile Install
     -Like “Apply Changes” but made years ago
     -Works by installing 2 apps:
     • Stub app with custom application class to
     load code & resources from “Contents app”
     • "Contents app" with actual code and resources
     -Incrementally updates dex files and resources
     in “Contents app”
     -Has few corner cases and limitations
     https:"//docs.bazel.build/versions/master/mobile-install.html
     

     View Slide

179. Design of Bazel: Execution
     https:"//docs.bazel.build/versions/master/mobile-install.html
     

     View Slide

180. Design of Bazel:
     IDE Integration
     

     View Slide

181. Design of Bazel:
     IDE Integration
     

     View Slide

182. Design of Bazel:
     IDE Integration
     

     View Slide

183. Design of Bazel: IDE Integration
     

     View Slide

184. Design of Bazel: IDE Integration
     -IDE integration uses Bazel CLI commands
     

     View Slide

185. Design of Bazel: IDE Integration
     -IDE integration uses Bazel CLI commands
     -IDE queries Bazel about the project
     (bazel query …)
     

     View Slide

186. Design of Bazel: IDE Integration
     -IDE integration uses Bazel CLI commands
     -IDE queries Bazel about the project
     (bazel query …)
     -Supports Partial Sync for specific files,
     dirs
     

     View Slide

187. Design of Bazel: IDE Integration
     TODO: screenshot of Android Studio Bazel Sync
     

     View Slide

188. Design of Bazel: IDE Integration
     ‣IntelliJ Family
     -IntelliJ
     -Android Studio
     -CLion
     https:"//docs.bazel.build/versions/master/ide.html
     

     View Slide

189. Design of Bazel: IDE Integration
     ‣Xcode
     -Tulsi and XCHammer to generate Xcode
     project from Bazel project
     -Lyft is working on Focus tool to limit
     number of loaded modules
     -lyft/index-import to import swift and clang
     index files produced as side effect of
     compilation
     https:"//docs.bazel.build/versions/master/ide.html
     

     View Slide

190. Design of Bazel:
     Integration w/
     Other Tools
     

     View Slide

191. Design of Bazel: Integration w/ Other Tools
     ‣Genrule
     -Run arbitrary CLI tool, Bash, Powershell,
     etc
     -As any other rule, inputs and outputs are
     tracked
     -Sandboxing makes it easier to achieve
     reproducibility
     https:"//docs.bazel.build/versions/master/be/general.html#genrule
     

     View Slide

192. Design of Bazel: Integration w/ Other Tools
     genrule(
     name = "mygenrule",
     cmd = "mkdir $(RULEDIR)/mydir; javac -d $(OUTS) $(SRCS)",
     outs = [ "mydir" ],
     srcs = [ "B.java" ],
     )
     https:"//docs.bazel.build/versions/master/be/general.html#genrule
     

     View Slide

193. Design of Bazel: Integration w/ Other Tools
     ‣Toolchains and Platforms
     -Bazel has notion of Target and Execution
     Platforms
     -Toolchain defines key/value pairs per Platform
     -Toolchain typically represents a set of
     Tooling for a language
     -Compiler
     -Linker
     -etc
     https:"//docs.bazel.build/versions/master/toolchains.html
     

     View Slide

194. Design of Bazel: Integration w/ Other Tools
     ‣Extensions
     -.bzl files written in Starlark with macros,
     rules, constants
     -load(“#//mybazelextensions:file.bzl”,
     “myrule_or_macro”)
     -One of main Bazel 1.0 goals is to extract
     impls from core to external rules repos
     https:"//docs.bazel.build/versions/master/skylark/concepts.html
     

     View Slide

195. Design of Bazel: Integration w/ Other Tools
     ‣Extensions
     

     View Slide

196. Design of Bazel: Integration w/ Other Tools
     ‣Extensions
     github.com/bazelbuild/
     

     View Slide

197. Design of Bazel: Integration w/ Other Tools
     ‣Extensions
     github.com/bazelbuild/
     -rules_java
     

     View Slide

198. Design of Bazel: Integration w/ Other Tools
     ‣Extensions
     github.com/bazelbuild/
     -rules_java
     -rules_android
     

     View Slide

199. Design of Bazel: Integration w/ Other Tools
     ‣Extensions
     github.com/bazelbuild/
     -rules_java
     -rules_android
     -rules_kotlin
     

     View Slide

200. Design of Bazel: Integration w/ Other Tools
     ‣Extensions
     github.com/bazelbuild/
     -rules_java
     -rules_android
     -rules_kotlin
     -rules_jvm_external
     

     View Slide

201. Design of Bazel: Integration w/ Other Tools
     ‣Extensions
     github.com/bazelbuild/
     -rules_java
     -rules_android
     -rules_kotlin
     -rules_jvm_external
     -rules_apple
     

     View Slide

202. Design of Bazel: Integration w/ Other Tools
     ‣Extensions
     github.com/bazelbuild/
     -rules_java
     -rules_android
     -rules_kotlin
     -rules_jvm_external
     -rules_apple
     -etc
     

     View Slide

203. Design of Bazel: Integration w/ Other Tools
     ‣Extensions
     github.com/bazelbuild/
     -rules_java
     -rules_android
     -rules_kotlin
     -rules_jvm_external
     -rules_apple
     -etc
     ‣github.com/you/rules_x !
     

     View Slide

204. Bazel
     Q&A: Open https:!//sli.do and
     enter code “bazel”
     

     View Slide

205. Artem Zinnatullin
     ‣ @artem_zin
     ‣ artemzin.com
     Come work with me!
     Also: Lyft Mobile Podcast
     tools, libraries, CI, builds @
     

     View Slide