Continuous Delivery for Microservices with Spinnaker at Mercari

Continuous Delivery for Microservices with Spinnaker at Mercari

At Mercari, we use Spinnaker for Continuous Delivery for Microservices on Kubernetes. In this talk, I will introduce how Mercari uses Spinnaker, current status and feature plan.

Ecb3acc2d246962361a4f8b3f7a6dd12?s=128

taichi nakashima

July 18, 2018
Tweet

Transcript

  1. LINE Developer Meetup in Tokyo #40 18 July 2018 Continuous

    Delivery with Spinnaker
  2. @deeeet / @tcnksm Blog / deeeet.com Tech Lead at Mercari

    Microservices Platform Team
  3. We are hiring !

  4. 4 How Mercari uses Spinnaker for Microservices Continuous Delivery ?

    • What & Why Continuous Delivery? • What & Why Spinnaker? • How Mercari uses Spinnaker? (Current status & future) tl;dr
  5. Microservices at Mercari

  6. 6 While we had lots of benefits of monolith architecture,

    its codebase is getting larger and complex... • Difficult to understand change effects • Difficult to test • Difficult to on-boarding • Difficult to isolate failure Why Microservices?
  7. 7 Even business and organization growth, we want to •

    Avoid velocity stalled but rather make deployment cycle faster • Increase each developer productivity Why Microservices? Started to migrate Microservices architecure
  8. 8 We use GCP for main infrastructure for Microservices platform

    • Each services are deployed to Kubernetes (GKE) as container • Each services uses GCP Managed DB (e.g., Cloud Spanner) Microservices Technical Stack
  9. Continuous Delivery

  10. 10 Continuous Delivery is a software engineering approach in which

    teams produce software in short cycles, ensuring that the software can be reliably released at any time. What is Continuous Delivery? https://en.wikipedia.org/wiki/Continuous_delivery
  11. 11 Continuous deployment is the next step of continuous delivery:

    Every change that passes the automated tests is deployed to production automatically. +Continuous Deployment https://puppet.com/blog/continuous-delivery-vs-continuous-deployment-what-s-diff
  12. 12 • We can provide new feature faster to customers

    • We can easily triage source of issue when something goes wrong • We can focus on service development itself Why Continuous Delivery?
  13. 13 Challenges at Mercari We had some challenges for Continuous

    Delivery • Continuous Delivery in Microservices architecture • Continuous Delivery on Kubernetes environment
  14. 14 Challenges: Microservices “You build it, You run it!” principle

    is there • In monolith ◦ Central team (e.g., SRE) handles all deployment • In microservices ◦ Each service development team need to maintain and customize deployment based on their service characteristics
  15. 15 You can use kubectl apply but... • To kick

    kubectl apply from CI is straightforward way but… ◦ How to know a deployment “Finished”? ◦ How to handle rollback? ◦ How to implement complex deployment flow (e.g., canary)? Challenges: Kubernetes
  16. 16 Challenges at Mercari We need common tool or platform

    which allows developers easily customize their deployments and works with kubernetes
  17. None
  18. 18 Spinnaker is an open source, multi-cloud continuous delivery platform

    for releasing software changes with high velocity and confidence. What is Spinnaker? https://www.spinnaker.io
  19. 19 Initially developed internally at Netflix (before Asgard) • Google

    joined development from 2015 • Open sourced at Nov. 2015 • Reached 1.0 at Jun. 2017 • Latest version is 1.8.1 (at Jul. 2018) What is Spinnaker?
  20. 20 Support Multi-Cloud like GCP and AWS and kubernetes •

    Opinionated based on Netflix experience and lessons and learned ◦ Immutable infrastructure, Red-Black deployment • Becoming kinda de-fact tool for Cloud deployment What is Spinnaker?
  21. 21

  22. 22 Spinnaker structures deployments as Pipeline • Each pipeline has

    configuration defining like ◦ Stages ◦ Triggers ◦ Notifications How to use Spinnaker
  23. 23 Stage is Spinnaker action that form an atomic building

    block for a pipeline Spinnaker Pipeline: Stage • Infrastructure stages: Bake, Deploy, Rollback • External system integration: Jenkins, Webhook • Controlling Flow: Manual judgement, Wait, Run pipeline
  24. 24 Triggers decides how the pipeline is started Spinnaker Pipeline:

    Trigger • Time-based: Cron, Manual • Event-based: Docker Registry, Git, CI, Pipeline, Pub/Sub
  25. 25 Workflows that are automatically run needs notifications to broadcast

    the status of event Spinnaker Pipeline: Notification • When?: Pipeline started, success, failure • Where?: Slack, Email, Pub/Sub
  26. 26 Spinnaker Pipeline: Example Trigger Stage

  27. 27 A big part of Continuous Delivery practice successful is

    to be able to new code without fear. Automation is great. But it’s better with proper safeguards is important Safeguards
  28. 28 Spinnaker provides a lot of safeguards • For Deployment

    ◦ Deploy strategies (red-black, canary) ◦ Rollback • For Pipeline ◦ Pipeline concurrency Spinnaker Safeguards
  29. 29 Spinnaker Safeguards: Stage Each steps are considered to be

    remediated in failure scenarios
  30. 30 Spinnaker Safeguards: Deploy Window Block deployment when peektime

  31. 31 Spinnaker watches Kubernetes operation is actually “Finished” (manifest stability)

    Spinnaker Safeguards: Kubernetes • Spinnaker considered stable when the Kubernetes contorller manager no longer needs to modify it ◦ Deployment is stable when pods are updated and ready ◦ Service is stable when underlying LB is created
  32. Spinnaker at Mercari

  33. 33 We used it from 2017 (v1.0) • Currently Using

    1.8 • Using Kubernetes V1 provider (GUI based) • More than 30 applications are deployed via Spinnaker • Handle deployment for 3 regions: JP, US and UK Spinnaker at Mercari
  34. 34 Spinnaker Architecture We prepared dedicated GKE cluster for Spinnaker

  35. 35 Architecture Spinnaker manages multiple region GKE cluster us-west1 europe-west1

    asia-northwest1
  36. 36 Architecture Spinnaker manages both DEV and PROD cluser DEV

    cluster PROD cluster
  37. 37 We have various pipelines • Red-Black (Blue-Green) deployment for

    Microservices • Canary deployment (without ACA) • Complex Machine Learning model deployment • DB migration Job Example Pipelines
  38. 38 Example Pipelines: DB Migration

  39. 39 Example Pipelines: Canary

  40. 40 As admin • Easy provide common best practice to

    developers as stage ◦ Admin can encapsulate complex cloud operation ◦ We want to build own custom stage in future! • Everything is in one place The Good
  41. 41 As developers • Easy to setup own customized pipeline

    (no custom scripting) • Easy to refer other team practice • No need to learn whole kubernetes YAML manifest The Good
  42. 42 As admin • Too unstable (v1.7 was worst experience..)

    • Spinnaker management itself by halyard ◦ Difficult to review and automate • Another cluster problem effect to the others .. • Less documentation (need to ask via Slack or GitHub Issues) The Bad
  43. 43 As developers • Too many hates about GUI ◦

    Not intuitive… difficult to know where to change ◦ Impossible to review changes ◦ Difficult to copy & paste from other team pipeline ◦ Difficult to share knowledge... The Bad I wonder how Netflix does it well (they has 9000 pipelines!)
  44. Next Steps

  45. 45 We want to manage everything by declarative way •

    Introduce Kubernetes V2 provider • Introduce Declarative Continuous Delivery (DCD spec) Declarative management Make GUI read-only!
  46. 46 We want to add more useful stages for developers

    • Integration testing stage • Squeeze testing stage • Automated canary analysis (ACA) stage More stages
  47. 47 We want to provide a training session to developers

    • What is Kubernetes? How to use? • What is Spinnaker? How to use? On-boarding training Documentation documentation documentation...
  48. 48 Introduced how Mercari uses Spinnaker for Microservices Continuous Delivery

    Conclusion