Tweet
Tweet

Slide 1

Slide 1 text

LINE Developer Meetup in Tokyo #40 18 July 2018 Continuous Delivery with Spinnaker

Slide 2

Slide 2 text

@deeeet / @tcnksm Blog / deeeet.com Tech Lead at Mercari Microservices Platform Team

Slide 3

Slide 3 text

We are hiring !

Slide 4

Slide 4 text

4 How Mercari uses Spinnaker for Microservices Continuous Delivery ? ● What & Why Continuous Delivery? ● What & Why Spinnaker? ● How Mercari uses Spinnaker? (Current status & future) tl;dr

Slide 5

Slide 5 text

Microservices at Mercari

Slide 6

Slide 6 text

6 While we had lots of benefits of monolith architecture, its codebase is getting larger and complex... ● Difficult to understand change effects ● Difficult to test ● Difficult to on-boarding ● Difficult to isolate failure Why Microservices?

Slide 7

Slide 7 text

7 Even business and organization growth, we want to ● Avoid velocity stalled but rather make deployment cycle faster ● Increase each developer productivity Why Microservices? Started to migrate Microservices architecure

Slide 8

Slide 8 text

8 We use GCP for main infrastructure for Microservices platform ● Each services are deployed to Kubernetes (GKE) as container ● Each services uses GCP Managed DB (e.g., Cloud Spanner) Microservices Technical Stack

Slide 9

Slide 9 text

Continuous Delivery

Slide 10

Slide 10 text

10 Continuous Delivery is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. What is Continuous Delivery? https://en.wikipedia.org/wiki/Continuous_delivery

Slide 11

Slide 11 text

11 Continuous deployment is the next step of continuous delivery: Every change that passes the automated tests is deployed to production automatically. +Continuous Deployment https://puppet.com/blog/continuous-delivery-vs-continuous-deployment-what-s-diff

Slide 12

Slide 12 text

12 ● We can provide new feature faster to customers ● We can easily triage source of issue when something goes wrong ● We can focus on service development itself Why Continuous Delivery?

Slide 13

Slide 13 text

13 Challenges at Mercari We had some challenges for Continuous Delivery ● Continuous Delivery in Microservices architecture ● Continuous Delivery on Kubernetes environment

Slide 14

Slide 14 text

14 Challenges: Microservices “You build it, You run it!” principle is there ● In monolith ○ Central team (e.g., SRE) handles all deployment ● In microservices ○ Each service development team need to maintain and customize deployment based on their service characteristics

Slide 15

Slide 15 text

15 You can use kubectl apply but... ● To kick kubectl apply from CI is straightforward way but… ○ How to know a deployment “Finished”? ○ How to handle rollback? ○ How to implement complex deployment flow (e.g., canary)? Challenges: Kubernetes

Slide 16

Slide 16 text

16 Challenges at Mercari We need common tool or platform which allows developers easily customize their deployments and works with kubernetes

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

18 Spinnaker is an open source, multi-cloud continuous delivery platform for releasing software changes with high velocity and confidence. What is Spinnaker? https://www.spinnaker.io

Slide 19

Slide 19 text

19 Initially developed internally at Netflix (before Asgard) ● Google joined development from 2015 ● Open sourced at Nov. 2015 ● Reached 1.0 at Jun. 2017 ● Latest version is 1.8.1 (at Jul. 2018) What is Spinnaker?

Slide 20

Slide 20 text

20 Support Multi-Cloud like GCP and AWS and kubernetes ● Opinionated based on Netflix experience and lessons and learned ○ Immutable infrastructure, Red-Black deployment ● Becoming kinda de-fact tool for Cloud deployment What is Spinnaker?

Slide 21

Slide 21 text

21

Slide 22

Slide 22 text

22 Spinnaker structures deployments as Pipeline ● Each pipeline has configuration defining like ○ Stages ○ Triggers ○ Notifications How to use Spinnaker

Slide 23

Slide 23 text

23 Stage is Spinnaker action that form an atomic building block for a pipeline Spinnaker Pipeline: Stage ● Infrastructure stages: Bake, Deploy, Rollback ● External system integration: Jenkins, Webhook ● Controlling Flow: Manual judgement, Wait, Run pipeline

Slide 24

Slide 24 text

24 Triggers decides how the pipeline is started Spinnaker Pipeline: Trigger ● Time-based: Cron, Manual ● Event-based: Docker Registry, Git, CI, Pipeline, Pub/Sub

Slide 25

Slide 25 text

25 Workflows that are automatically run needs notifications to broadcast the status of event Spinnaker Pipeline: Notification ● When?: Pipeline started, success, failure ● Where?: Slack, Email, Pub/Sub

Slide 26

Slide 26 text

26 Spinnaker Pipeline: Example Trigger Stage

Slide 27

Slide 27 text

27 A big part of Continuous Delivery practice successful is to be able to new code without fear. Automation is great. But it’s better with proper safeguards is important Safeguards

Slide 28

Slide 28 text

28 Spinnaker provides a lot of safeguards ● For Deployment ○ Deploy strategies (red-black, canary) ○ Rollback ● For Pipeline ○ Pipeline concurrency Spinnaker Safeguards

Slide 29

Slide 29 text

29 Spinnaker Safeguards: Stage Each steps are considered to be remediated in failure scenarios

Slide 30

Slide 30 text

30 Spinnaker Safeguards: Deploy Window Block deployment when peektime

Slide 31

Slide 31 text

31 Spinnaker watches Kubernetes operation is actually “Finished” (manifest stability) Spinnaker Safeguards: Kubernetes ● Spinnaker considered stable when the Kubernetes contorller manager no longer needs to modify it ○ Deployment is stable when pods are updated and ready ○ Service is stable when underlying LB is created

Slide 32

Slide 32 text

Spinnaker at Mercari

Slide 33

Slide 33 text

33 We used it from 2017 (v1.0) ● Currently Using 1.8 ● Using Kubernetes V1 provider (GUI based) ● More than 30 applications are deployed via Spinnaker ● Handle deployment for 3 regions: JP, US and UK Spinnaker at Mercari

Slide 34

Slide 34 text

34 Spinnaker Architecture We prepared dedicated GKE cluster for Spinnaker

Slide 35

Slide 35 text

35 Architecture Spinnaker manages multiple region GKE cluster us-west1 europe-west1 asia-northwest1

Slide 36

Slide 36 text

36 Architecture Spinnaker manages both DEV and PROD cluser DEV cluster PROD cluster

Slide 37

Slide 37 text

37 We have various pipelines ● Red-Black (Blue-Green) deployment for Microservices ● Canary deployment (without ACA) ● Complex Machine Learning model deployment ● DB migration Job Example Pipelines

Slide 38

Slide 38 text

38 Example Pipelines: DB Migration

Slide 39

Slide 39 text

39 Example Pipelines: Canary

Slide 40

Slide 40 text

40 As admin ● Easy provide common best practice to developers as stage ○ Admin can encapsulate complex cloud operation ○ We want to build own custom stage in future! ● Everything is in one place The Good

Slide 41

Slide 41 text

41 As developers ● Easy to setup own customized pipeline (no custom scripting) ● Easy to refer other team practice ● No need to learn whole kubernetes YAML manifest The Good

Slide 42

Slide 42 text

42 As admin ● Too unstable (v1.7 was worst experience..) ● Spinnaker management itself by halyard ○ Difficult to review and automate ● Another cluster problem effect to the others .. ● Less documentation (need to ask via Slack or GitHub Issues) The Bad

Slide 43

Slide 43 text

43 As developers ● Too many hates about GUI ○ Not intuitive… difficult to know where to change ○ Impossible to review changes ○ Difficult to copy & paste from other team pipeline ○ Difficult to share knowledge... The Bad I wonder how Netflix does it well (they has 9000 pipelines!)

Slide 44

Slide 44 text

Next Steps

Slide 45

Slide 45 text

45 We want to manage everything by declarative way ● Introduce Kubernetes V2 provider ● Introduce Declarative Continuous Delivery (DCD spec) Declarative management Make GUI read-only!

Slide 46

Slide 46 text

46 We want to add more useful stages for developers ● Integration testing stage ● Squeeze testing stage ● Automated canary analysis (ACA) stage More stages

Slide 47

Slide 47 text

47 We want to provide a training session to developers ● What is Kubernetes? How to use? ● What is Spinnaker? How to use? On-boarding training Documentation documentation documentation...

Slide 48

Slide 48 text

48 Introduced how Mercari uses Spinnaker for Microservices Continuous Delivery Conclusion