Bristol DevOps Meetup Achieving CD with k8s & Design patterns for Containerised Deployments Takayuki Watanabe Cookpad Ltd. Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w )

Who? • Takayuki Watanabe • twitter: @takanabe_w / github: takanabe • Site Reliability Engineer@Cookpad Ltd. • Umami rich food & booster lover Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 2

Umami? A loanword from the Japanese (͏·ຯ). umami can be translated as "pleasant savory taste" — Wikipedia: Umami Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 3

Umami? Umami is a general term used mainly for substances combining the amino acid glutamate, and/or the nucleotides inosinate and guanylate, with minerals such as sodium and potassium. — Umami Information Center Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 4

You feel delicious and happy while you are eating Umami rich food!! Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 5

Do you want to add Umami to your dishes? Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 6

Japanese Umami Boost Magic Powder You can buy them on Amazon! Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 7

Japanese Umami Boost Magic Powder You can buy them on Amazon! Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 8

OK! we can boost Umami easily... Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 9

Then, let's move on to today's main topic ... Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 10

How can you boost your organization development scale? • There are many topics... • Organization Design View • Functional vs Project based organization • Monolithic/Microservice Architecture • DevOps/Autonomous team • And more... Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 11

How can you boost your organization development scale? • There are many topics... • Technology View • Container applications • Advanced monitoring / Observability • Continous delivery / Continous deployment (today's focus) • And more... Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 12

Practical Approaches to Achieve Continuous Deployment with Kubernetes Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 13

Today's menu • Motivation of Continuous Deployment • Overview of Continuous Deployment • Pipeline tool: A key component of Continuous Deployment • Introduction to Spinnaker's features • Achieve Continuous Deployment on Kubernetes with Spinnaker • Demo Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 14

Motivation of Continuous Deployment Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 15

Motivation of Continuous Deployment Rapid product development is a friend to succeed in bisiness. Development performance is cretical factor to track rapid product development. Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 16

Motivation of Continuous Deployment Which metrics become appropriate KPIs to track organization development performance? Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 17

Motivation of Continuous Deployment Which metrics become appropriate KPIs to track organization development performance? • Per capita and total Pull Request open period? • Per capita and total number of code commits? • Per capita and total product release count? • Per capita and total deployment count? Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 18

Motivation of Continuous Deployment Cookpad's SRE are tracking deployment frequency. Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 19

Motivation of Continuous Deployment Cookpad's SRE are tracking deployment frequency. We want to increase the number. Increate the number of deployment per capita is also important. Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 20

Motivation of Continuous Deployment • However, as our oranization grows, individual knowledge gaps around release engineering widens Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 21

Motivation of Continuous Deployment • However, as our oranization grows, individual knowledge gaps around release engineering widens • How can I ship my software to staging and production? • Run deploy command over ssh? • Deploy command from local environment? • Deploy command from chat bot? Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 22

Motivation of Continuous Deployment • However, as our oranization grows, individual knowledge gaps around release engineering widens • How can I test my codes with CI? • Which CI tools should we use? • Which environment should we use for test? Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 23

Motivation of Continuous Deployment • However, as our oranization grows, individual knowledge gaps around release engineering widens • How can I deploy my software safely? • What is our criteria of rollbacks? • How can we deploy software to multi regions? • Should we involve QA team for product releases? Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 24

Motivation of Continuous Deployment • Best practices prepared by release engineering experts for each organization improve software release flow. • Follow tracks prepared by release engineering experts • The number of deploy count could be increased Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 25

Overview of Continuous Deployment Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 26

Overview of Continuous Deployment(CD) • Continuous Deployment (CD) is the chain of process that validates and deploys software into production environment. • Every change that passes the automated tests is deployed to production automatically. Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 27

Overview of Continuous Deployment(CD) • We can include best practices into Continuous Deployment • Deployment strategies • Quality Assurance • Feature implementation tests Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 28

Clarify dependencies & optimize each step • Deploy strategies are just one of the components in CD • Clarification of each CD step is important Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 29

Clarify dependencies & optimize each step • We can analyze and improve release flow if we fully understand the steps Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 30

Clarify dependencies & optimize each step • How can we change the deploy step? Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 31

Deployment strategies • Choose basic deploy storategies based on team policy • Recreate deployment (a.k.a Highlander) • Blue/Green deployment (a.k.a Red/Black deployment) • Canary deployment • Rolling deployment Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 32

Recreate deployment (a.k.a Highlander) All instances running a version of an application are upgraded to the new version at the same time. Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 33

Recreate deployment (a.k.a Highlander) • Pros: • Simple • Budget/resource friendly (standby instances are not necessary) • Cons: • Server downtime during switching to new codes (hot reload is necessary) • Use case: • hobby & POC projects • staging environment Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 34

Blue/Green deployment (a.k.a Red/Black) A new server group with the new version of code is stood up with no traffic. Once all the servers are ready, all the traffic is directed from the old server group to the new server group. Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 35

Blue/Green deployment (a.k.a Red/Black) • Pros: • Simple • Easy to rollback workload and traffic from new servers to previous servers • Cons: • All or nothing traffic switch • Budget/resource consuming • Use case: • High availability production environment Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 36

Canary deployment Only a small portion of the available servers are deployed new version of code. Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 37

Canary deployment • Pros: • A safer pattern than recreate deployment • Enable developers to confirm code behavior in production • Budget/resource friendly (standby instances are not necessary) • Cons: • Users face multi version service • Use case: • Confirm new feature behavior before 100% release in production Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 38

Rolling deployment The rolling deploy is simply the continuation of the canary deploy. Update one server after another until server group has been upgraded. Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 39

Rolling deployment • Pros: • Enable developers to confirm code behavior in production • Limit user downtime and impact due to unexpected code behavior • Budget/resource friendly (standby instances are not necessary) • Cons: • Users face multi version service • Sophisticated monitoring and operation support tools are indispensable • Use case: • Confirm new feature behavior before 100% release in production Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 40

Clarify dependencies & optimize each step • How can we clarify CD chain? • Pipeline tools! Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 41

Pipeline tools: A key component of Continuous Deployment Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 42

Role of Pipeline tools • One of the main components to achieve CD • Deliver automatic product release flow • Consistency of release step chains • Close individual skills gap for product release • Visibility & Transparency • UI & Declarative specification Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 43

Diffs of Pipeline tools • Pipeline itself does not make much difference • Myriad tools exist ... • Choose the best one for your environment Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 44

Example of Pipeline tools • Pure pipeline & CD platform • CodePipeline: https://aws.amazon.com/documentation/codepipeline/ • Spinnaker: https://github.com/spinnaker • Drone: https://github.com/drone/drone • Pipeline & CD platform specialized for kubernetes • Argo: https://github.com/argoproj/argo • Brigade: https://github.com/Azure/brigade • skaffold: https://github.com/GoogleCloudPlatform/skaffold • Jenkins X: https://github.com/jenkins-x Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 45

CodePipeline AWS CodePipeline is a continuous delivery service you can use to model, visualize, and automate the steps required to release your software. Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 46

Spinnaker Spinnaker is an open source, multi-cloud continuous delivery platform for releasing software changes with high velocity and confidence. Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 47

Brigade Brigade is the tool for creating pipelines for Kubernetes. Script simple and complex workflows using JavaScript. Chain together containers, running them in parallel or serially. Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 48

My Impression • AWS CodePipeline is a reasonable choice for AWS users • Good for for simple software release • Spinnaker is a platfrom which bring us CD & immutable infrastructure • Not only just simple pipeline • CD platform is valuable for large oranizations Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 49

Introduction to Spinnaker's features Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 50

Things we can do by Spinnaker • Define flexible Pipeline Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 51

Things we can do by Spinnaker • Leverage an immutable infrastructure • Deployment with immutable images for faster rollouts, easier rollbacks • Elimination of hard to debug configuration drift issues Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 52

Things we can do by Spinnaker • Restrict execution to specific time windows Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 53

Things we can do by Spinnaker • Deployment based on basic deploy storategies • Recreate deployment (a.k.a Highlander) • Blue/Green deployment (a.k.a Red/Black deployment) • Canary deployment • Rolling deployment Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 54

Things we can do by Spinnaker • Abstraction layer of cloud platforms (Multi Cloud) • GCP • AWS • OpenStack • Kubernetes • ECS(under development) Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 55

Things we can do by Spinnaker • Mapping resources between Spinnaker and Kubernetes • Orchestrate Kubernetes resources via Spinnaker Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 56

Things we can do by Spinnaker • Automated Canary Analysis(ACA) Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 57

Things we can do by Spinnaker • Chaos engineering integration(Chaos Monkey integration) • Randomly terminates virtual machine instances and containers that run inside of your production environment Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 58

Spinnaker is the best option for CD? • Seems like well maintained for GCP and kubernetes • Core maintainers are from Google/Netflix • There are Cons. .. • Document is not well maintained • Support for AWS services is relatively bad... • Escpecially integration with ECR is terrible • Declaration of pipeline is not ready for use Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 59

Achieve Continuous deployment on Kubernetes with Spinnaker Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 60

Continous Deployment steps on k8s with Spinnaker • Divide each deploy target environment into pieces • Easy to optimize by chaging pipeline components Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 61

Continous Deployment steps on k8s with Spinnaker • Local development steps (without pipeline) Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 62

Continous Deployment steps on k8s with Spinnaker • Feature test env steps Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 63

Continous Deployment steps on k8s with Spinnaker • Feature test env steps Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 64

Continous Deployment steps on k8s with Spinnaker • Staging env steps Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 65

Continous Deployment steps on k8s with Spinnaker • Staging env steps Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 66

Continous Deployment steps on k8s with Spinnaker • Production env steps Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 67

Continous Deployment steps on k8s with Spinnaker • Staging env steps Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 68

Continous Deployment steps on k8s with Spinnaker • Build best practices for your team • Pipeline include database migration? • Pipeline include manual judgement? • Pipeline is restricted by deploy time window? Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 69

Demo Easy-Peasy Continuous Deployment on Kubernetes with Spinnaker Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 70

Today's Demo Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 71

Step1: edit code Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 72

Step2: Push commit to github Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 73

Step3: Build Docker image Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 74

Step4: Push Docker image to ECR Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 75

Step5: Invoke Spinnaker pipeline Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 76

Step6: Deploy green apps to k8s Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 77

Step7: Service out blue apps Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 78

Cleanup • Continous Deployment boosts your organization development • Pipeline tools are key component of CD • Spinnaker brings us variable features for CD with Kubernetes Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 79

Thanks! takayuki-watanabe@cookpad.com Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 80

References Explanation of Continuous Deployment and deployment strategies highly inspireed by the following sites: • Martinfowler.com BlueGreenDeployment: https://martinfowler.com/bliki/BlueGreenDeployment.html • Martinfowler.com DeploymentPipeline: https://martinfowler.com/bliki/DeploymentPipeline.html • Continuous Delivery Vs. Continuous Deployment: What's the Diff?: https://puppet.com/blog/continuous-delivery-vs- continuous-deployment-what-s-diff • Deployment Strategies and Release Best Practices: http://cgrant.io/article/deployment-strategies/ • Stop Gambling with Upgrades, Murphy’s Law Always Wins: https://rhelblog.redhat.com/2015/05/07/stop-gambling-with- upgrades-murphys-law-always-wins/#more-908 • SAFe: Continuous Deployment: https://www.scaledagileframework.com/continuous-deployment/ • Continuous Delivery vs Continuous Deployment vs Continuous Integration: Key Definitions: https://blog.assembla.com/ assemblablog/tabid/12618/bid/92411/continuous-delivery-vs-continuous-deployment-vs-continuous-integration-wait-huh.aspx • Kubernetes deployment strategies: http://container-solutions.com/kubernetes-deployment-strategies/ Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 81

References Pipeline tools • Spinnaker: https://docs.aws.amazon.com/codepipeline/latest/userguide/welcome.html • Brigate: https://azure.github.io/kashti/#!/build/01bzzn45f0p5bjtsffw9m8chts) • Chaos Monkey: https://github.com/Netflix/chaosmonkey • Automated Canary Analysis at Netflix with Kayenta: https://medium.com/netflix-techblog/automated- canary-analysis-at-netflix-with-kayenta-3260bc7acc69 Misc • Wikipedia Umami: https://en.wikipedia.org/wiki/Umami • Umami Information Center: https://www.umamiinfo.com/what/whatisumami/ Bristol DevOps Meetup @Cookpad Ltd. (April 25, 2018) | Takayuki Watanabe ( takanabe/@takanabe_w ) 82