Continuous Delivery for Microservices with Spinnaker at Mercari
At Mercari, we use Spinnaker for Continuous Delivery for Microservices on Kubernetes. In this talk, I will introduce how Mercari uses Spinnaker, current status and feature plan.
6 While we had lots of benefits of monolith architecture, its codebase is getting larger and complex... ● Difficult to understand change effects ● Difficult to test ● Difficult to on-boarding ● Difficult to isolate failure Why Microservices?
7 Even business and organization growth, we want to ● Avoid velocity stalled but rather make deployment cycle faster ● Increase each developer productivity Why Microservices? Started to migrate Microservices architecure
8 We use GCP for main infrastructure for Microservices platform ● Each services are deployed to Kubernetes (GKE) as container ● Each services uses GCP Managed DB (e.g., Cloud Spanner) Microservices Technical Stack
10 Continuous Delivery is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. What is Continuous Delivery? https://en.wikipedia.org/wiki/Continuous_delivery
11 Continuous deployment is the next step of continuous delivery: Every change that passes the automated tests is deployed to production automatically. +Continuous Deployment https://puppet.com/blog/continuous-delivery-vs-continuous-deployment-what-s-diff
12 ● We can provide new feature faster to customers ● We can easily triage source of issue when something goes wrong ● We can focus on service development itself Why Continuous Delivery?
13 Challenges at Mercari We had some challenges for Continuous Delivery ● Continuous Delivery in Microservices architecture ● Continuous Delivery on Kubernetes environment
14 Challenges: Microservices “You build it, You run it!” principle is there ● In monolith ○ Central team (e.g., SRE) handles all deployment ● In microservices ○ Each service development team need to maintain and customize deployment based on their service characteristics
15 You can use kubectl apply but... ● To kick kubectl apply from CI is straightforward way but… ○ How to know a deployment “Finished”? ○ How to handle rollback? ○ How to implement complex deployment flow (e.g., canary)? Challenges: Kubernetes
18 Spinnaker is an open source, multi-cloud continuous delivery platform for releasing software changes with high velocity and confidence. What is Spinnaker? https://www.spinnaker.io
19 Initially developed internally at Netflix (before Asgard) ● Google joined development from 2015 ● Open sourced at Nov. 2015 ● Reached 1.0 at Jun. 2017 ● Latest version is 1.8.1 (at Jul. 2018) What is Spinnaker?
20 Support Multi-Cloud like GCP and AWS and kubernetes ● Opinionated based on Netflix experience and lessons and learned ○ Immutable infrastructure, Red-Black deployment ● Becoming kinda de-fact tool for Cloud deployment What is Spinnaker?
22 Spinnaker structures deployments as Pipeline ● Each pipeline has configuration defining like ○ Stages ○ Triggers ○ Notifications How to use Spinnaker
23 Stage is Spinnaker action that form an atomic building block for a pipeline Spinnaker Pipeline: Stage ● Infrastructure stages: Bake, Deploy, Rollback ● External system integration: Jenkins, Webhook ● Controlling Flow: Manual judgement, Wait, Run pipeline
25 Workflows that are automatically run needs notifications to broadcast the status of event Spinnaker Pipeline: Notification ● When?: Pipeline started, success, failure ● Where?: Slack, Email, Pub/Sub
27 A big part of Continuous Delivery practice successful is to be able to new code without fear. Automation is great. But it’s better with proper safeguards is important Safeguards
28 Spinnaker provides a lot of safeguards ● For Deployment ○ Deploy strategies (red-black, canary) ○ Rollback ● For Pipeline ○ Pipeline concurrency Spinnaker Safeguards
31 Spinnaker watches Kubernetes operation is actually “Finished” (manifest stability) Spinnaker Safeguards: Kubernetes ● Spinnaker considered stable when the Kubernetes contorller manager no longer needs to modify it ○ Deployment is stable when pods are updated and ready ○ Service is stable when underlying LB is created
33 We used it from 2017 (v1.0) ● Currently Using 1.8 ● Using Kubernetes V1 provider (GUI based) ● More than 30 applications are deployed via Spinnaker ● Handle deployment for 3 regions: JP, US and UK Spinnaker at Mercari
37 We have various pipelines ● Red-Black (Blue-Green) deployment for Microservices ● Canary deployment (without ACA) ● Complex Machine Learning model deployment ● DB migration Job Example Pipelines
40 As admin ● Easy provide common best practice to developers as stage ○ Admin can encapsulate complex cloud operation ○ We want to build own custom stage in future! ● Everything is in one place The Good
41 As developers ● Easy to setup own customized pipeline (no custom scripting) ● Easy to refer other team practice ● No need to learn whole kubernetes YAML manifest The Good
42 As admin ● Too unstable (v1.7 was worst experience..) ● Spinnaker management itself by halyard ○ Difficult to review and automate ● Another cluster problem effect to the others .. ● Less documentation (need to ask via Slack or GitHub Issues) The Bad
43 As developers ● Too many hates about GUI ○ Not intuitive… difficult to know where to change ○ Impossible to review changes ○ Difficult to copy & paste from other team pipeline ○ Difficult to share knowledge... The Bad I wonder how Netflix does it well (they has 9000 pipelines!)
47 We want to provide a training session to developers ● What is Kubernetes? How to use? ● What is Spinnaker? How to use? On-boarding training Documentation documentation documentation...
tcnksm
takanorig
a_bicky
hhiroshell
akiroom
kazamori
k1low
fullfull
kosui
poteboy
karamem0
okamototakuyasr2
nishiuma
pauljervisheath
destraynor
zakiwarfel
brad_frost
orderedlist
carmenintech
dougneiner
chriscoyier
garrettdimon
kneath
kevinliebholz
hursman