Continuous Delivery for Microservices with Spinnaker at Mercari

Continuous Delivery for Microservices with Spinnaker at Mercari

At Mercari, we use Spinnaker for Continuous Delivery for Microservices on Kubernetes. In this talk, I will introduce how Mercari uses Spinnaker, current status and feature plan.

Ecb3acc2d246962361a4f8b3f7a6dd12?s=128

taichi nakashima

July 18, 2018
Tweet

Transcript

  1. 4.

    4 How Mercari uses Spinnaker for Microservices Continuous Delivery ?

    • What & Why Continuous Delivery? • What & Why Spinnaker? • How Mercari uses Spinnaker? (Current status & future) tl;dr
  2. 6.

    6 While we had lots of benefits of monolith architecture,

    its codebase is getting larger and complex... • Difficult to understand change effects • Difficult to test • Difficult to on-boarding • Difficult to isolate failure Why Microservices?
  3. 7.

    7 Even business and organization growth, we want to •

    Avoid velocity stalled but rather make deployment cycle faster • Increase each developer productivity Why Microservices? Started to migrate Microservices architecure
  4. 8.

    8 We use GCP for main infrastructure for Microservices platform

    • Each services are deployed to Kubernetes (GKE) as container • Each services uses GCP Managed DB (e.g., Cloud Spanner) Microservices Technical Stack
  5. 10.

    10 Continuous Delivery is a software engineering approach in which

    teams produce software in short cycles, ensuring that the software can be reliably released at any time. What is Continuous Delivery? https://en.wikipedia.org/wiki/Continuous_delivery
  6. 11.

    11 Continuous deployment is the next step of continuous delivery:

    Every change that passes the automated tests is deployed to production automatically. +Continuous Deployment https://puppet.com/blog/continuous-delivery-vs-continuous-deployment-what-s-diff
  7. 12.

    12 • We can provide new feature faster to customers

    • We can easily triage source of issue when something goes wrong • We can focus on service development itself Why Continuous Delivery?
  8. 13.

    13 Challenges at Mercari We had some challenges for Continuous

    Delivery • Continuous Delivery in Microservices architecture • Continuous Delivery on Kubernetes environment
  9. 14.

    14 Challenges: Microservices “You build it, You run it!” principle

    is there • In monolith ◦ Central team (e.g., SRE) handles all deployment • In microservices ◦ Each service development team need to maintain and customize deployment based on their service characteristics
  10. 15.

    15 You can use kubectl apply but... • To kick

    kubectl apply from CI is straightforward way but… ◦ How to know a deployment “Finished”? ◦ How to handle rollback? ◦ How to implement complex deployment flow (e.g., canary)? Challenges: Kubernetes
  11. 16.

    16 Challenges at Mercari We need common tool or platform

    which allows developers easily customize their deployments and works with kubernetes
  12. 17.
  13. 18.

    18 Spinnaker is an open source, multi-cloud continuous delivery platform

    for releasing software changes with high velocity and confidence. What is Spinnaker? https://www.spinnaker.io
  14. 19.

    19 Initially developed internally at Netflix (before Asgard) • Google

    joined development from 2015 • Open sourced at Nov. 2015 • Reached 1.0 at Jun. 2017 • Latest version is 1.8.1 (at Jul. 2018) What is Spinnaker?
  15. 20.

    20 Support Multi-Cloud like GCP and AWS and kubernetes •

    Opinionated based on Netflix experience and lessons and learned ◦ Immutable infrastructure, Red-Black deployment • Becoming kinda de-fact tool for Cloud deployment What is Spinnaker?
  16. 21.

    21

  17. 22.

    22 Spinnaker structures deployments as Pipeline • Each pipeline has

    configuration defining like ◦ Stages ◦ Triggers ◦ Notifications How to use Spinnaker
  18. 23.

    23 Stage is Spinnaker action that form an atomic building

    block for a pipeline Spinnaker Pipeline: Stage • Infrastructure stages: Bake, Deploy, Rollback • External system integration: Jenkins, Webhook • Controlling Flow: Manual judgement, Wait, Run pipeline
  19. 24.

    24 Triggers decides how the pipeline is started Spinnaker Pipeline:

    Trigger • Time-based: Cron, Manual • Event-based: Docker Registry, Git, CI, Pipeline, Pub/Sub
  20. 25.

    25 Workflows that are automatically run needs notifications to broadcast

    the status of event Spinnaker Pipeline: Notification • When?: Pipeline started, success, failure • Where?: Slack, Email, Pub/Sub
  21. 27.

    27 A big part of Continuous Delivery practice successful is

    to be able to new code without fear. Automation is great. But it’s better with proper safeguards is important Safeguards
  22. 28.

    28 Spinnaker provides a lot of safeguards • For Deployment

    ◦ Deploy strategies (red-black, canary) ◦ Rollback • For Pipeline ◦ Pipeline concurrency Spinnaker Safeguards
  23. 31.

    31 Spinnaker watches Kubernetes operation is actually “Finished” (manifest stability)

    Spinnaker Safeguards: Kubernetes • Spinnaker considered stable when the Kubernetes contorller manager no longer needs to modify it ◦ Deployment is stable when pods are updated and ready ◦ Service is stable when underlying LB is created
  24. 33.

    33 We used it from 2017 (v1.0) • Currently Using

    1.8 • Using Kubernetes V1 provider (GUI based) • More than 30 applications are deployed via Spinnaker • Handle deployment for 3 regions: JP, US and UK Spinnaker at Mercari
  25. 37.

    37 We have various pipelines • Red-Black (Blue-Green) deployment for

    Microservices • Canary deployment (without ACA) • Complex Machine Learning model deployment • DB migration Job Example Pipelines
  26. 40.

    40 As admin • Easy provide common best practice to

    developers as stage ◦ Admin can encapsulate complex cloud operation ◦ We want to build own custom stage in future! • Everything is in one place The Good
  27. 41.

    41 As developers • Easy to setup own customized pipeline

    (no custom scripting) • Easy to refer other team practice • No need to learn whole kubernetes YAML manifest The Good
  28. 42.

    42 As admin • Too unstable (v1.7 was worst experience..)

    • Spinnaker management itself by halyard ◦ Difficult to review and automate • Another cluster problem effect to the others .. • Less documentation (need to ask via Slack or GitHub Issues) The Bad
  29. 43.

    43 As developers • Too many hates about GUI ◦

    Not intuitive… difficult to know where to change ◦ Impossible to review changes ◦ Difficult to copy & paste from other team pipeline ◦ Difficult to share knowledge... The Bad I wonder how Netflix does it well (they has 9000 pipelines!)
  30. 45.

    45 We want to manage everything by declarative way •

    Introduce Kubernetes V2 provider • Introduce Declarative Continuous Delivery (DCD spec) Declarative management Make GUI read-only!
  31. 46.

    46 We want to add more useful stages for developers

    • Integration testing stage • Squeeze testing stage • Automated canary analysis (ACA) stage More stages
  32. 47.

    47 We want to provide a training session to developers

    • What is Kubernetes? How to use? • What is Spinnaker? How to use? On-boarding training Documentation documentation documentation...