Tweet
Tweet

Slide 1

Slide 1 text

Engineering a Continuous Delivery Pipeline Charlotte Godley

Slide 2

Slide 2 text

@charwarz Aim & Objectives Describe how we engineered our continuous delivery pipeline into multiple Kubernetes environments @ Ocado Technology Objectives ● Set the problem in context: some specifics of the Ocado use case ● Define what a bad deployment process looks like ● Define what a better deployment process looks like ● Discuss what caused us pain, and how we addressed it ● Talk about where we’re going next

Slide 3

Slide 3 text

@charwarz Hello! github.com/godley blog.charlottegodley.co.uk ocadotechnology.com github.com/ocadotechnology

Slide 4

Slide 4 text

@charwarz Problem context Source: arstechnica.com

Slide 5

Slide 5 text

@charwarz Current (bad) Delivery Processes at Ocado Source: ocadotechnology.com

Slide 6

Slide 6 text

@charwarz Getting a big picture is hard. Source: gregatkinson.com

Slide 7

Slide 7 text

@charwarz Incomplete or Non-existent Documentation Source: dilbert.com

Slide 8

Slide 8 text

@charwarz One size doesn’t fit all Source: linkedin.com

Slide 9

Slide 9 text

@charwarz No Post-deployment Visibility Aka, did that work?!

Slide 10

Slide 10 text

The big picture

Slide 11

Slide 11 text

@charwarz GitOps Git is the single source of truth for operations. Every merge to master starts a deployment pipeline, so what’s currently running in prod should be the HEAD of master branch. Operations by Pull Request - Weaveworks gives a good summary

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

Documentation

Slide 15

Slide 15 text

@charwarz Write down all the things Documentation our team writes is... - In one, obvious place - Peer reviewed - Concise, but with lots of examples - Updated regularly - A reference guide for best practice, but avoids duplicating upstream documentation

Slide 16

Slide 16 text

@charwarz What A Good Manifests Repo Looks Like |-- .gitlab-ci.yml |-- README.md |-- manifests | |--00-Namespace.yaml ...other useful examples... |-- hack | |--minikube.sh

Slide 17

Slide 17 text

@charwarz What A Good Deployment Repo Looks Like |-- .gitlab-ci.yml |-- README.md |-- teams | |--00-kubernetes | | |--manifests | | | |--secret.yaml.enc | | |--bundles

Slide 18

Slide 18 text

Avoiding rules

Slide 19

Slide 19 text

@charwarz Ground rules... - Every resource has a namespace - Every project namespace name is [-optional-suffix] - Every project namespace has the label appId: - The above 2 rules match each other

Slide 20

Slide 20 text

Did that work?!

Slide 21

Slide 21 text

@charwarz Monitoring - Automated slack integrations of when pipelines start/stop - Document links to our own dashboards - End to end tests to highlight where apps could be affected by kubernetes issues

Slide 22

Slide 22 text

@charwarz Access - Read-only access - Developer access - Admin access

Slide 23

Slide 23 text

Pain points

Slide 24

Slide 24 text

@charwarz Workflow

Slide 25

Slide 25 text

@charwarz Workflow

Slide 26

Slide 26 text

@charwarz Workflow

Slide 27

Slide 27 text

@charwarz Workflow

Slide 28

Slide 28 text

@charwarz Workflow fix: reference updater bot

Slide 29

Slide 29 text

@charwarz Validation Source: foolishlego.com

Slide 30

Slide 30 text

@charwarz Tidying up - What’s currently on the cluster is master branch - Add = new merge request - Rollback = revert - Delete = ??

Slide 31

Slide 31 text

@charwarz Testing the pipeline Source: agilenutshell.com

Slide 32

Slide 32 text

What’s next?

Slide 33

Slide 33 text

@charwarz Summary “If users aren't finding success on their own it's not their fault. It's our fault. We didn't make it easy enough for them to fall into the pit of success.” Jeff Atwood

Slide 34

Slide 34 text

@charwarz Thanks for listening! (We’re hiring…) github.com/godley blog.charlottegodley.co.uk ocadotechnology.com github.com/ocadotechnology careers.ocado.com