Continuous Delivery for Microservices with Spinnaker at Mercari

At Mercari, we use Spinnaker for Continuous Delivery for Microservices on Kubernetes. In this talk, I will introduce how Mercari uses Spinnaker, current status and feature plan.

1. LINE Developer Meetup in Tokyo #40 18 July 2018 Continuous Delivery with Spinnaker
2. @deeeet / @tcnksm Blog / deeeet.com Tech Lead at Mercari Microservices Platform Team
3. We are hiring !
4. 4 How Mercari uses Spinnaker for Microservices Continuous Delivery ? • What & Why Continuous Delivery? • What & Why Spinnaker? • How Mercari uses Spinnaker? (Current status & future) tl;dr
5. Microservices at Mercari
6. 6 While we had lots of benefits of monolith architecture, its codebase is getting larger and complex... • Difficult to understand change effects • Difficult to test • Difficult to on-boarding • Difficult to isolate failure Why Microservices?
7. 7 Even business and organization growth, we want to • Avoid velocity stalled but rather make deployment cycle faster • Increase each developer productivity Why Microservices? Started to migrate Microservices architecure
8. 8 We use GCP for main infrastructure for Microservices platform • Each services are deployed to Kubernetes (GKE) as container • Each services uses GCP Managed DB (e.g., Cloud Spanner) Microservices Technical Stack
9. Continuous Delivery
10. 10 Continuous Delivery is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. What is Continuous Delivery? https://en.wikipedia.org/wiki/Continuous_delivery
11. 11 Continuous deployment is the next step of continuous delivery: Every change that passes the automated tests is deployed to production automatically. +Continuous Deployment https://puppet.com/blog/continuous-delivery-vs-continuous-deployment-what-s-diff
12. 12 • We can provide new feature faster to customers • We can easily triage source of issue when something goes wrong • We can focus on service development itself Why Continuous Delivery?
13. 13 Challenges at Mercari We had some challenges for Continuous Delivery • Continuous Delivery in Microservices architecture • Continuous Delivery on Kubernetes environment
14. 14 Challenges: Microservices “You build it, You run it!” principle is there • In monolith ◦ Central team (e.g., SRE) handles all deployment • In microservices ◦ Each service development team need to maintain and customize deployment based on their service characteristics
15. 15 You can use kubectl apply but... • To kick kubectl apply from CI is straightforward way but… ◦ How to know a deployment “Finished”? ◦ How to handle rollback? ◦ How to implement complex deployment flow (e.g., canary)? Challenges: Kubernetes
16. 16 Challenges at Mercari We need common tool or platform which allows developers easily customize their deployments and works with kubernetes
18. 18 Spinnaker is an open source, multi-cloud continuous delivery platform for releasing software changes with high velocity and confidence. What is Spinnaker? https://www.spinnaker.io
19. 19 Initially developed internally at Netflix (before Asgard) • Google joined development from 2015 • Open sourced at Nov. 2015 • Reached 1.0 at Jun. 2017 • Latest version is 1.8.1 (at Jul. 2018) What is Spinnaker?
20. 20 Support Multi-Cloud like GCP and AWS and kubernetes • Opinionated based on Netflix experience and lessons and learned ◦ Immutable infrastructure, Red-Black deployment • Becoming kinda de-fact tool for Cloud deployment What is Spinnaker?
22. 22 Spinnaker structures deployments as Pipeline • Each pipeline has configuration defining like ◦ Stages ◦ Triggers ◦ Notifications How to use Spinnaker
23. 23 Stage is Spinnaker action that form an atomic building block for a pipeline Spinnaker Pipeline: Stage • Infrastructure stages: Bake, Deploy, Rollback • External system integration: Jenkins, Webhook • Controlling Flow: Manual judgement, Wait, Run pipeline
24. 24 Triggers decides how the pipeline is started Spinnaker Pipeline: Trigger • Time-based: Cron, Manual • Event-based: Docker Registry, Git, CI, Pipeline, Pub/Sub
25. 25 Workflows that are automatically run needs notifications to broadcast the status of event Spinnaker Pipeline: Notification • When?: Pipeline started, success, failure • Where?: Slack, Email, Pub/Sub
26. 26 Spinnaker Pipeline: Example Trigger Stage
27. 27 A big part of Continuous Delivery practice successful is to be able to new code without fear. Automation is great. But it’s better with proper safeguards is important Safeguards
28. 28 Spinnaker provides a lot of safeguards • For Deployment ◦ Deploy strategies (red-black, canary) ◦ Rollback • For Pipeline ◦ Pipeline concurrency Spinnaker Safeguards
29. 29 Spinnaker Safeguards: Stage Each steps are considered to be remediated in failure scenarios
30. 30 Spinnaker Safeguards: Deploy Window Block deployment when peektime
31. 31 Spinnaker watches Kubernetes operation is actually “Finished” (manifest stability) Spinnaker Safeguards: Kubernetes • Spinnaker considered stable when the Kubernetes contorller manager no longer needs to modify it ◦ Deployment is stable when pods are updated and ready ◦ Service is stable when underlying LB is created
32. Spinnaker at Mercari
33. 33 We used it from 2017 (v1.0) • Currently Using 1.8 • Using Kubernetes V1 provider (GUI based) • More than 30 applications are deployed via Spinnaker • Handle deployment for 3 regions: JP, US and UK Spinnaker at Mercari
34. 34 Spinnaker Architecture We prepared dedicated GKE cluster for Spinnaker
35. 35 Architecture Spinnaker manages multiple region GKE cluster us-west1 europe-west1 asia-northwest1
36. 36 Architecture Spinnaker manages both DEV and PROD cluser DEV cluster PROD cluster
37. 37 We have various pipelines • Red-Black (Blue-Green) deployment for Microservices • Canary deployment (without ACA) • Complex Machine Learning model deployment • DB migration Job Example Pipelines
38. 38 Example Pipelines: DB Migration
39. 39 Example Pipelines: Canary
40. 40 As admin • Easy provide common best practice to developers as stage ◦ Admin can encapsulate complex cloud operation ◦ We want to build own custom stage in future! • Everything is in one place The Good
41. 41 As developers • Easy to setup own customized pipeline (no custom scripting) • Easy to refer other team practice • No need to learn whole kubernetes YAML manifest The Good
42. 42 As admin • Too unstable (v1.7 was worst experience..) • Spinnaker management itself by halyard ◦ Difficult to review and automate • Another cluster problem effect to the others .. • Less documentation (need to ask via Slack or GitHub Issues) The Bad
43. 43 As developers • Too many hates about GUI ◦ Not intuitive… difficult to know where to change ◦ Impossible to review changes ◦ Difficult to copy & paste from other team pipeline ◦ Difficult to share knowledge... The Bad I wonder how Netflix does it well (they has 9000 pipelines!)
44. Next Steps
45. 45 We want to manage everything by declarative way • Introduce Kubernetes V2 provider • Introduce Declarative Continuous Delivery (DCD spec) Declarative management Make GUI read-only!
46. 46 We want to add more useful stages for developers • Integration testing stage • Squeeze testing stage • Automated canary analysis (ACA) stage More stages
47. 47 We want to provide a training session to developers • What is Kubernetes? How to use? • What is Spinnaker? How to use? On-boarding training Documentation documentation documentation...
48. 48 Introduced how Mercari uses Spinnaker for Microservices Continuous Delivery Conclusion