Building Applications for Container with Bazel

Transcript

1. Go Conference'20 in Autumn SENDAI Fumihiro Ito θοτϥϘגࣜձࣾ Building Applications

   for Container with Bazel

2. ࣗݾ঺հ • Fumihiro Ito ʢҏ౻ ࢙ߒʣ • Software Engineer •

   ࠓ೥1݄ʹೖࣾ͠·ͨ͠ • θοτϥϘגࣜձࣾ • Ϡϑʔגࣜձࣾ100%ࢠձࣾ • ϠϑʔͷΠϯϑϥΛࢧ͑Δج൫ͷR&D • Kubernetes as a Service ͸ϠϑʔͰӡ༻͞Ε600ΫϥελҎ্Λ؅ཧ͍ͯ͠Δ 2

3. We are hiring! https://zlab.co.jp

4. Agenda • Bazelͱ͸ • GoͱBazel • BazelͰίϯςφΛ࡞Δ 4 ࢀߟίʔυͳͲ͸ϦϙδτϦʹ΋Ξοϓϩʔυͯ͋͠Γ·͢ https://github.com/f110/bazel-example/tree/gocon20

5. Bazel ͱ͸

6. Bazelͱ͸ • ΦʔϓϯιʔεͷϏϧυπʔϧ • GoogleͷΤϯδχΞ͕த৺ʹͳͬͯ։ൃΛ͍ͯ͠Δ • ίϛολʔͷଟ͕͘Googler or Xoogler •

   ಺෦Ͱ͸ Blaze ͱݺ͹ΕΔπʔϧ͕͋ΔΒ͍͠ (2006ʙ) • େن໛ͳίʔυɾϢʔβʔϕʔεͰ΋εέʔϧͰ͖Δ • ෳ਺ͷݴޠΛෳ਺ͷϓϥοτϑΥʔϜʹ޲͚ͯϏϧυ͢Δ͜ͱ͕Ͱ͖Δ • Java / C / C++ / Python / Go / Objective-C / JavaScript / Rust / protobuf 6

7. Bazel • Bazel ຊମ͸ Java • αʔόʔʗΫϥΠΞϯτ Ϟσϧ • ඇৗʹ֦ுੑ͕ߴ͍

   • Starlark ͱ͍͏ಠࣗݴޠͰઃఆϑΝΠϧ͕هड़͞ΕΔ • ϏϧυϧʔϧʹΫΤϦΛ࣮ߦ͢Δ͜ͱ͕Ͱ͖Δ • Ϗϧυ͕ಠࣗͷαϯυϘοΫεͷதͰߦΘΕΔ • ڧྗͳΩϟογϡػߏΛ͍࣋ͬͯΔ 7

8. ߴ͍֦ுੑ • Go ΛίϯύΠϧ͢Δ͜ͱ΋Ͱ͖Ε͹ Python ΋Ͱ͖Δ • ίϯςφ΍ rpm/deb ύοέʔδΛ࡞੒͢Δ͜ͱ΋Ͱ͖Δ

   • Java ΍ C++ ͸ຊମʹؚ·ΕΔ͕ͦΕҎ֎͸෼཭͞Ε͍ͯΔ • Starlark ʢޙड़ʣͰϏϧυͷखॱΛίʔυʹམͱ͠ࠐΉ • ϦϙδτϦ಺Ͱ֦ுΛॻ͘͜ͱ΋Ͱ͖Δ • ࠷ॳ͸֦ுΛॻ͘͜ͱ͸͓͢͢Ί͠·ͤΜ 8

9. Starlark • Python ͷΑ͏ͳݴޠʢPythonͰ͸ͳ͍ʣ • Python ͷεΫϦϓτͰ Makefile Λੜ੒͍ͯͨ͠ྺ࢙తܦҢ •

   Ϗϧυϧʔϧ͸͢΂ͯ Starlark Ͱهड़͢Δ • ͨͩ͠ϏϧυϧʔϧͰ Starlark ͷจ๏͕͢΂ͯ࢖͑ΔΘ͚Ͱ͸ͳ͍ • ϏϧυΛهड़͢Δ͜ͱΛ໨తʹσβΠϯ͞Ε͍ͯΔ 9

10. Starlark • จ๏͸ Python3 ʹΠϯεύΠΞ͞Ε͍ͯΔ • Ϗϧυͷ֦ு΋ Starlark Ͱهड़͢Δ 10

    def fizz_buzz(n): for i in range(1, n + 1): s = "" if i % 3 == 0: s += "Fizz" if i % 5 == 0: s += "Buzz" print(s if s else i) fizz_buzz(20) https://docs.bazel.build/versions/master/skylark/language.html

11. ΫΤϦ • ఆٛͨ͠Ϗϧυϧʔϧશମʹରͯ͠ΫΤϦΛ࣮ߦͰ͖Δ • ࣮ߦՄೳͳόΠφϦΛఆ͍ٛͯ͠Δύοέʔδ • bazel query 'kind(go_binary, //...)'

    • ͋Δύοέʔδʹґଘ͍ͯ͠ΔύοέʔδͷϦετ • bazel query 'rdeps(...,)' • ϦϙδτϦ഑ԼͷίϯςφΠϝʔδΛ͢΂ͯpush͢ΔϫϯϥΠφʔ • bazel query 'kind(container_push, //...)' | xargs -n1 bazel run 11

12. Sandboxing • Ϗϧυͷ࠷খ୯Ґ͝ͱʹαϯυϘοΫεʹΑͬͯ෼཭͞ΕΔ • αϯυϘοΫεʹด͡ࠐΊΒΕΔͷ͸ϏϧυʹඞཁͳϑΝΠϧͷΈ • ෆ༻ҙͳ҉໧తͳґଘΛ࡞ΓࠐΉՄೳੑ͕௿͍ • ϏϧυʹඞཁͳϑΝΠϧΛ͢΂ͯϦετΞοϓ͢Δ •

    Ϗϧυͷ࠶ݱੑΛߴΊΔ͜ͱ΋໨తͷҰͭʢେࣄʣ 12

13. Cache • ΩϟογϡͰ͖Δͱ͜Ζ͸Ωϟογϡ͢ΔΑ͏ʹͳ͍ͬͯΔ • Ϗϧυͷ݁Ռ͸ϩʔΧϧͷσΟεΫΩϟογϡͰ࠶ར༻͞ΕΔ • ϩʔΧϧͰͷΩϟογϡ͚ͩͰ͸ͳ͘ϦϞʔτͰΩϟογϡΛڞ༗Ͱ͖Δ • ྡͷਓ͕Ϗϧυͨ݁͠ՌΛར༻Ͱ͖Δ •

    αϯυϘοΫεͷ͓͔͛Ͱதؒੜ੒෺΋ΩϟογϡͰ͖Δ 13

14. Bazelͷྑ͍ͱ͜Ζ • ґଘؔ܎͕͢΂ͯهड़͞ΕΔ • ͦΕΛϏϧυ͢ΔͨΊʹԿ͕ඞཁͳͷ͔͸͖ͬΓهड़͞ΕΔ • ෳ਺ͷݴޠΛϏϧυ͢Δ͜ͱ͕Ͱ͖Δ • ϏϧυΛߴ଎ʹ͢Δ࢓૊Έ͕͋Δ •

    ΠϯΫϦϝϯλϧίϯύΠϧ • Remote Cache ΍ Remote Build Execution 14

15. ґଘؔ܎ͷهड़ 15 gocon20 lib/logger pflag cmd/gocon20 lib/http lib/fluentd lib/file •

    ੒Ռ෺ΛಘΔͨΊʹඞཁͳೖྗΛ໌ه͢Δ • ೖྗ͸੒Ռ෺Ͱ͋ͬͯ΋Α͍ • ॥؀ࢀরΛ࡞Δ͜ͱ͸Ͱ͖ͳ͍

16. ґଘؔ܎ͷهड़ 16 sh_binary( name = "sh", srcs = ["hello.sh"], deps

    = [":world"], ) sh_library( name = "world", srcs = ["world.sh"] ) Target ग़ྗͷ୯Ґ

17. ґଘؔ܎ͷهड़ 17 sh_binary( name = "sh", srcs = ["hello.sh"], deps

    = [":world"], ) sh_library( name = "world", srcs = ["world.sh"] ) λʔήοτ໊

18. ґଘؔ܎ͷهड़ 18 sh_binary( name = "sh", srcs = ["hello.sh"], deps

    = [":world"], ) sh_library( name = "world", srcs = ["world.sh"] ) ೖྗ

19. ґଘؔ܎ͷهड़ 19 sh_binary( name = "sh", srcs = ["hello.sh"], deps

    = [":world"], ) sh_library( name = "world", srcs = ["world.sh"] ) ґଘ

20. Visualize 20 //cmd/goimports:goimports //cmd/goimports:goimports_lib //cmd/goimports:goimports_gc.go //cmd/goimports:doc.go //cmd/goimports:goimports.go //internal/gocommand:gocommand //internal/imports:imports //vendor/golang.org/x/mod/semver:semver

    //internal/gocommand:vendor.go //internal/gocommand:invoke.go //internal/event:event //internal/imports:imports.go //internal/imports:zstdlib.go //internal/imports:mod.go //internal/imports:mod_cache.go //internal/imports:fix.go //internal/imports:sortimports.go //go/ast/astutil:astutil //internal/gopathwalk:gopathwalk //vendor/golang.org/x/mod/module:module //go/ast/astutil:rewrite.go //go/ast/astutil:util.go //go/ast/astutil:enclosing.go //go/ast/astutil:imports.go //internal/gopathwalk:walk.go //internal/fastwalk:fastwalk //vendor/golang.org/x/mod/module:module.go //vendor/golang.org/x/xerrors:xerrors //vendor/golang.org/x/mod/semver:semver.go //vendor/golang.org/x/xerrors:wrap.go //vendor/golang.org/x/xerrors:errors.go //vendor/golang.org/x/xerrors:doc.go //vendor/golang.org/x/xerrors:frame.go //vendor/golang.org/x/xerrors:format.go //vendor/golang.org/x/xerrors:adaptor.go //vendor/golang.org/x/xerrors:fmt.go //vendor/golang.org/x/xerrors/internal:internal //vendor/golang.org/x/xerrors/internal:internal.go //internal/fastwalk:fastwalk_dirent_namlen_bsd.go //internal/fastwalk:fastwalk_dirent_fileno.go //internal/fastwalk:fastwalk_dirent_ino.go //internal/fastwalk:fastwalk.go //internal/fastwalk:fastwalk_unix.go //internal/fastwalk:fastwalk_portable.go //internal/fastwalk:fastwalk_dirent_namlen_linux.go //internal/event:doc.go //internal/event:event.go //internal/event/core:core //internal/event/keys:keys //internal/event/label:label //internal/event/core:export.go //internal/event/core:event.go //internal/event/core:fast.go //internal/event/keys:keys.go //internal/event/keys:standard.go //internal/event/label:label.go

21. Bazelͷྑ͘ͳ͍ͱ͜Ζ • Starlark ͰϏϧυϧʔϧΛهड़͢Δͱ͍͏ͷ͕ಠಛ • ಋೖ࣌ͷֶशίετ͕ߴ͍ • νʔϜ಺ʹ͋Δఔ౓ਫ਼௨ͨ͠ਓ͕͍ͳ͚Ε͹ಋೖ΋೉͍͠ • Ϗϧυʹ͍ͭͯͷߟ͑ํΛैདྷ͔Βେ͖͘ม͑Δඞཁ͕͋Δ

    • Ϗϧυϓϩηεશମ΋ιϑτ΢ΣΞͷҰ෦ͱΈͳ͢͜ͱ͕Ͱ͖Δ • Bazel ྲྀͷ΍Γํ͕෼͔Δ·Ͱ͔ͳΓۤ௧Λ൐͏ 21

22. GoͱBazel

23. Ϩγϐ • ֎෦ґଘ • rules_go • gazelle • ϫϯϥΠφʔڞ༗πʔϧʢΦϓγϣϯɻͳͯ͘΋Α͍ʣ •

    IDEͷRun Configuration • Makefile 23

24. rules_go • https://github.com/bazelbuild/rules_go • Go ΛϏϧυ͢ΔͨΊͷ֦ு • go_binary: ࣮ߦϑΝΠϧΛఆٛ͢Δ •

    go_library: ଞύοέʔδ͔ΒϦϯΫ͞ΕΔύοέʔδΛఆٛ͢Δ • go_test: ςετΛఆٛ͢Δ • ͜ΕΒͷϧʔϧ͸ιʔεϑΝΠϧ͔Βࣗಈੜ੒Մೳ • GoͷϥϯλΠϜ͸Ϗϧυϧʔϧͷ಺෦Ͱऔಘ͢ΔͷͰೖΕΔඞཁ͸ͳ͍ 24

25. gazelle • ϏϧυϧʔϧΛࣗಈੜ੒͢Δπʔϧ • Go ͷ৔߹͸΄ͱΜͲ͜ͷࣗಈੜ੒Ͱ΍Γ͍ͨ͜ͱ͕Ͱ͖Δ • ॳճͷੜ੒͚ͩͰ͸ͳ͘ɺ2ճ໨Ҏ߱ͷߋ৽΋Ͱ͖Δ • ࣗಈߋ৽͞Εͨ͘ͳ͍ՕॴΛϚʔΫ΋Ͱ͖Δ

    25

26. BazelΛ࢖͍࢝ΊΔ·Ͱ • Step 1: WORKSPACE ʹґଘΛ௥Ճ͢Δ • Step 2: BUILD.bazel

    Λ࡞Δ • Step 3: ϏϧυϧʔϧΛੜ੒͢Δ 26

27. • ֎෦΁ͷґଘΛهड़͢ΔϑΝΠϧ Step 1: WORKSPACE 27 load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") http_archive( name

    = "io_bazel_rules_go", sha256 = "b725e6497741d7fc2d55fcc29a276627d10e43fa5d0bb692692890ae30d98d00", urls = [ "https://github.com/bazelbuild/rules_go/releases/download/v0.24.3/rules_go-v0.24.3.tar.gz", ], ) http_archive( name = "bazel_gazelle", sha256 = "72d339ff874a382f819aaea80669be049069f502d6c726a07759fdca99653c48", urls = [ "https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.22.1/bazel-gazelle-v0.22.1.tar.gz", ], ) load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies") load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies") go_rules_dependencies() go_register_toolchains() gazelle_dependencies()

28. • Ϗϧυͷ੒Ռ෺ʹ͍ͭͯఆٛ͢Δ • ϦϙδτϦͷτοϓʹ࣍ͷϑΝΠϧΛ࡞Δ • ͜ͷϑΝΠϧ͚ͩ͸ࣗΒͷखͰ࡞Δඞཁ͕͋Δ Step 2: BUILD.bazel 28

    load("@bazel_gazelle//:def.bzl", "gazelle") # gazelle:prefix go.f110.dev/bazel-example # gazelle:proto disable_global gazelle(name = "gazelle")

29. Step 3: gazelle update • BUILD.bazel Λࣗಈੜ੒ • bazel run

    //:gazelle -- update • ੜ੒͞Εͨ BUILD.bazel ͸͢΂ͯίϛοτ͢Δ 29

30. ։ൃΛଓ͚Δ • Step 1: Write code • Step 2: BUILD.bazel

    Λߋ৽͢Δ • bazel run //:gazelle -- update • ґଘؔ܎ʹมߋ͕ͳ͚Ε͹ෆཁ • Step 3: ࣮ߦ͢Δ • bazel run //cmd/gocon20 30

31. ґଘϥΠϒϥϦͷ؅ཧํ๏ • ؅ཧ͢Δํ๏͸2ͭ 1. WORKSPACE Ͱ؅ཧ͢Δํ๏ • gazelle update-repos -from_file=go.mod

    Ͱ WORKSPACE ϑΝΠϧΛߋ৽͢Δ 2. vendor σΟϨΫτϦΛϦϙδτϦʹؚΊΔ • vendor σΟϨΫτϦͰ؅ཧ͢Δํ๏͕͓͢͢Ί • bazel run @go_sdk//:bin/go -- mod vendor • WORKSPACE άϩʔόϧͳ֎෦ґଘ͕૿͑Δͱ؅ཧ͕େมʹͳΔ 31

32. BazelͰϏϧυ͢Δͱ͖ͷϙΠϯτ (1/2) • ೖྗʹରͯ͠ग़ྗ͕มΘΒͳ͍Α͏ʹ͢Δ • Ϗϧυ࣌ؒΛຒΊࠐΉͱϏϧυຖʹग़ྗ͕ҟͳΔͷͰ΍Βͳ͍ • Gitͷίϛοτϋογϡ΋ຒΊࠐΉඞཁ͸ͳ͍ • ಉ͡ίϛοτ͔Βಉ͡όΠφϦϑΝΠϧ͕ग़ྗ͞ΕΔͷͰಛఆ͢Δ͜ͱ͸Ͱ͖Δ

    • ιʔείʔυͷࣗಈੜ੒Λ͍ͯ͠Δ৔߹͸݁ՌΛϦϙδτϦʹίϛοτ͢Δ • αϯυϘοΫε಺Ͱੜ੒ͨ͠ίʔυʹରͯ͠IDE͕ΠϯσοΫε͠ͳ͍ͷͰิ׬Ͱ͖ͳ͍ • ґଘϥΠϒϥϦ͸ vendor σΟϨΫτϦ؅ཧʹ͢Δ 32

33. BazelͰϏϧυ͢Δͱ͖ͷϙΠϯτ (2/2) • ʢݱ࣌఺Ͱ͸ʣprotobuf ʹର͢Δϧʔϧͷࣗಈੜ੒Λ͠ͳ͍ • # gazelle:proto disable_global ΛτοϓͷBUILD.bazelʹॻ͍͓ͯ͘

    • Ϗϧυϧʔϧʹ͢ΔͱαϯυϘοΫε಺Ͱίʔυੜ੒͕ߦΘΕͯ͠·͍IDEͰิ׬Ͱ͖ͳ͍ 33

34. BazelͰίϯςφΛ࡞Δ

35. ίϯςφΛ࡞Δ • rules_docker Λ࢖͏ • ίϯςφΛ࡞Δࡍʹ docker ͕ඞཁͳ͍ • Docker

    Desktop on Mac ͔Βͷղ์ • ݡ͍Ωϟογϡ • rpm/deb ύοέʔδΛίϯςφʹೖΕΑ͏ͱࢥ͏ͱ͍͢͝େม 35

36. όΠφϦ͕1͚ͭͩͷίϯςφ • go_binary ͷ୅ΘΓʹ go_image Λ࢖͏ 36 go_binary( name =

    "gocon20", embed = [":gocon20_lib"], visibility = ["//visibility:public"], ) load("@io_bazel_rules_docker//go:image.bzl", "go_image") go_image( name = "gocon20_image", embed = [":gocon20_lib"], goos = "linux", goarch = "amd64", visibility = ["//visibility:public"], )

37. Ϗϧυ • bazel build //cmd/gocon20:gocon20_image.tar • dockerʹϩʔυ͢Δ • docker load

    -i bazel-bin/cmd/gocon20/gocon20_image.tar 37 $ docker images | grep gocon20 REPOSITORY TAG IMAGE ID CREATED SIZE bazel/cmd/gocon20 gocon20_image 6a76b205225f 50 years ago 23.3MB λΠϜελϯϓ͸ඞͣ0ʹͳΔ

38. όΠφϦҎ֎ΛؚΊ͍ͨ৔߹ • όΠφϦҎ֎ΛؚΜͩ tar ͱόΠφϦΛؚΜͩ tar ΛϚʔδ͢ΔΠϝʔδ 38 load("@rules_pkg//:pkg.bzl", "pkg_tar")

    load("@io_bazel_rules_docker//container:container.bzl", "container_image") pkg_tar( name = "files", srcs = ["gocon.txt"], package_dir = "/", ) pkg_tar( name = "bin", srcs = [ "//cmd/gocon20:gocon20", ], mode = "0755", package_dir = "/usr/local/bin", ) container_image( name = "image", base = "@com_google_distroless_base//image", tars = [ ":files", ":bin", ], )

39. push • ϓϥΠϕʔτϨδετϦʹରͯ͠ίϯςφΠϝʔδͷPush΋Ͱ͖Δ • ࣄલʹ docker login Λ͓ͯ͘͠ 39 load("@io_bazel_rules_docker//container:container.bzl",

    "container_push") container_push( name = "push", format = "Docker", image = ":image", registry = "quay.io", repository = "f110/bazel-example", tag = "latest", ) $ bazel run //:push $ cat bazel-bin/push.digest sha256:d6b76f53f79f030e117ac49323c294fff9dfebec5d036708e5f009189ed39104

40. ϙΠϯτ • GOOS ʹؾΛ͚ͭΔ • ΫϩείϯύΠϧΛ͢Δ৔߹͸ؾΛ͚ͭΔ • --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64 Λࢦఆ͢Δඞཁ͕͋Δ৔߹΋ •

    Mach-O ೖΓίϯςφΛ࡞ͬͯ Kubernetes Ͱಈ͔ͦ͏ͱͯͯ͠͠͠·͍͕ͪ 40

41. ·ͱΊ • Bazel ΛϏϧυπʔϧʹ͢Δͱґଘؔ܎͕ΫϦΞʹͳΔ • Bazel Ͱ Go ͷϓϩάϥϜ΋ίϯςφ΋Ϗϧυ͢Δ͜ͱ͕Ͱ͖Δ •

    ՝୊΍ߟ͑ํͷ੾Γସ͕͑ඞཁͳͱ͜Ζ΋͋Δ͕ͦΕΒΛղܾͰ͖Δͱշద • ֶशίετͷߴ͞ • ґଘϥΠϒϥϦͷ؅ཧ • Ϗϧυશମ΋ιϑτ΢ΣΞͱΈͳ͢ߟ͑ํ 41

42. Reference • https://bazel.build/ • https://www.pantsbuild.org/ • https://buck.build/ • https://please.build/index.html •

    https://blog.bazel.build/2017/03/21/design-of-skylark.html • https://github.com/bazelbuild/starlark • https://blog.bazel.build/2015/09/11/sandboxing.html • https://github.com/bazelbuild/rules_go • https://github.com/bazelbuild/bazel-gazelle • https://github.com/bazelbuild/rules_docker 42