Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Continuous Delivery with Bazel, Kubernes and Gitops

Continuous Delivery with Bazel, Kubernes and Gitops

Overview of a simple and effective deployment pipeline based off Bazel used to deploy to Kubernetes using a GitOps approach.



Ross Guarino

May 02, 2018


  1. CI/CD: Go/Bazel/Kubernetes A simple & effective deployment pipeline

  2. Deployment Automation

  3. Manual Deployments - Tedious/Toil - Delay the delivery of completed

    features or fixes - Easy to mess up - Difficult to debug intentions
  4. Automate Deployment - Goals - Fast - Idiot Proof -

    Remove all humans Bonus: - Work with local uncommitted changes - Need to have fast Edit -> Run -> Test cycle
  5. Automate Deployment - How? Two Key Changes 1. Switch to

    GitOps 2. Move to Bazel
  6. GitOps - Operations by Pull Request Git as the source

    of truth Works really well with Declarative orchestration (Kubernetes) Deploy to Production by simply merging a PR, machines take care of the rest github.com/box/kube-applier is a simple solution to mirror Git -> Kubernetes
  7. Bazel {fast, correct} - Choose Two Build and test software

    of any size, quickly and reliably
  8. Deployment Pipeline

  9. Why Bazel? - Reproducible - Fast - Polyglot - Flexible

  10. Deployment Pipeline

  11. Deployment Pipeline

  12. The Deployment Pipeline

  13. Where do the Kubernetes definitions live? If we keep code

    and .yaml separate we make it hard for local testing We include basic resources in the source repository itself
  14. Step 2 & 3: Build and Unit Test with Bazel

    Keeps track of every input over time Sandbox isolates build & tests from surrounding environment No more “The tests pass on my machine”
  15. Step 3: Packaging with Bazel - Supports a variety of

    packaging formats - Deb (pkg_deb) - Tar (pkg_tar) - Docker Images (image) - Easy to add support for your own - Produces identical packages months later
  16. Step 3: Docker Images pkg_tar( name = "bundle", srcs =

    [":profile"], package_dir = "/opt/pyli", ) image( name = "image", base = "@distroless_base//image", tars = [":bundle"], visibility = [ "//deployment/app:__pkg__", ], )
  17. Step 3: Docker Images Can lead to strange looking timestamps

    $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE bazel/cmd/profile image 78789f09c502 48 years ago 26.8MB
  18. Step 5: Deploy Deployment with GitOps is a PR to

    a operations repository Bazel is used to generate the Kubernetes files CD opens a PR to OPs repo with Bazel’s help
  19. Step 5: Deploy github.com/bazelbuild/rules_k8s Bazel Kubernetes rules allow you to

    build Kubernetes resource definitions.
  20. K8s_object rule Teaches Bazel about the relationship between a pod

    and a docker image rule
  21. Step 5: Deploy: Kubernetes Rule k8s_object( name = "profile", images

    = { "profile:dev": "//cmd/profile:image", }, template = ":profile.yaml", image_chroot = “gcr.io/<project>/”, kind = “nonsense”, cluster = “default”, ) apiVersion: apps/v1beta1 kind: Deployment metadata: name: profile spec: [...] template: [...] spec: containers: - name: profile image: profile:dev
  22. Step 5: Deploy: Kubernetes Rule $ bazel run //deployment:profile apiVersion:

    apps/v1beta1 kind: Deployment metadata: name: profile spec: [...] template: [...] spec: containers: - name: profile image: gcr.io/<project>/cmd/profile@sha256:<sha>
  23. Step 5: Deployment $ bazel run //deployment:profile > path/to/deployment/repo/deployment.yaml $

    cd path/to/deployment/repo $ git checkout -b deployment-branch $ git commit -m “deployment commit” $ git push origin deployment-branch
  24. Step 5: Human Conformation Once the PR is up a

    Human checks the PR and merges it
  25. Step 5: Local Development Every k8s_object rule with a kind

    and cluster comes with a .apply target $ bazel run //deployment:profile.apply [...] Edit some more code $ bazel run //deployment:profile.apply [...]
  26. Goals (Revisited) - Fast - Idiot Proof Bonus: - Work

    with local uncommitted changes - Need to have fast Edit -> Run -> Test cycle
  27. Bazel and Go Some Tips

  28. Tip 0: Use Gazelle $ cat BUILD load("@bazel_gazelle//:def.bzl", "gazelle") gazelle(

    name = "gazelle", prefix = "github.com/example/project", ) $ bazel run //:gazelle Project: github.com/bazelbuild/bazel-gazelle Auto-Generate Bazel build for existing go code
  29. Tip 1: Continue to use Dep Bazel does not understand

    semver. Use Gopkg.lock to populate your WORKSPACE $ bazel run //:gazelle -- update-repos -from_file=Gopkg.lock
  30. Tip 2: Use dep to update deps Bazel does not

    understand semver. Use dep update to update dependencies and the sync with Bazel $ dep ensure -update $ bazel run //:gazelle -- update-repos -from_file=Gopkg.lock
  31. Tip 3: Use Bazel Remote Caching Remote caching can dramatically

    speed up build times If you’re building vendoring large projects (like Kubernetes) this save a lot of time Google Cloud Storage caching is easy and works within 5 minutes
  32. Tip 4: Manually Cross Compile packages go_binary( name = "bin-linux-arm64",

    goarch = "arm64", goos = "linux", ) go_binary( name = "bin-darwin-amd64", goarch = "amd64", goos = "darwin", )
  33. Thank you! @0xRLG rlg.io

  34. Step 5: Deploy: Defining Defaults Kubernetes Rules let you define

    default in your WORKSPACE k8s_defaults( name = "k8s_deploy", kind = "nonsense", image_chroot = "gcr.io/<project>/", cluster = "default", )