Engineering a Continuous Delivery Pipeline

Engineering a Continuous Delivery Pipeline

How do you make the process of getting code into production seamless as a developer, without causing difficulties for operations? For example, developers would rather avoid gatekeeping in order to have shorter feedback loops. Ops teams would rather be well prepared for what’s going into production and when, in order to avoid surprises and prepare support training. In this talk you will learn about the process of creating a deployment pipeline for Kubernetes using Gitlab CI, what checks you can use to catch human error early, and ways of validating that changes won't cause the result to tip over. This talk will improve your ability to design deployment processes which avoid developer pain but builds operational trust.

Given at DevOps Days Edinburgh 2017


Charlotte Godley

October 24, 2017


  1. 2.

    @charwarz Aim & Objectives Describe how we engineered our continuous

    delivery pipeline into multiple Kubernetes environments @ Ocado Technology Objectives • Set the problem in context: some specifics of the Ocado use case • Define what a bad deployment process looks like • Define what a better deployment process looks like • Discuss what caused us pain, and how we addressed it • Talk about where we’re going next
  2. 11.

    @charwarz GitOps Git is the single source of truth for

    operations. Every merge to master starts a deployment pipeline, so what’s currently running in prod should be the HEAD of master branch. Operations by Pull Request - Weaveworks gives a good summary
  3. 12.
  4. 13.
  5. 15.

    @charwarz Write down all the things Documentation our team writes

    is... - In one, obvious place - Peer reviewed - Concise, but with lots of examples - Updated regularly - A reference guide for best practice, but avoids duplicating upstream documentation
  6. 16.

    @charwarz What A Good Manifests Repo Looks Like |-- .gitlab-ci.yml

    |-- |-- manifests | |--00-Namespace.yaml ...other useful examples... |-- hack | |
  7. 17.

    @charwarz What A Good Deployment Repo Looks Like |-- .gitlab-ci.yml

    |-- |-- teams | |--00-kubernetes | | |--manifests | | | |--secret.yaml.enc | | |--bundles
  8. 19.

    @charwarz Ground rules... - Every resource has a namespace -

    Every project namespace name is <appId>[-optional-suffix] - Every project namespace has the label appId: <appId> - The above 2 rules match each other
  9. 21.

    @charwarz Monitoring - Automated slack integrations of when pipelines start/stop

    - Document links to our own dashboards - End to end tests to highlight where apps could be affected by kubernetes issues
  10. 30.

    @charwarz Tidying up - What’s currently on the cluster is

    master branch - Add = new merge request - Rollback = revert - Delete = ??
  11. 33.

    @charwarz Summary “If users aren't finding success on their own

    it's not their fault. It's our fault. We didn't make it easy enough for them to fall into the pit of success.” Jeff Atwood