$30 off During Our Annual Pro Sale. View Details »

Continuous Delivery for Microservices with Spinnaker at Mercari

Continuous Delivery for Microservices with Spinnaker at Mercari

At Mercari, we use Spinnaker for Continuous Delivery for Microservices on Kubernetes. In this talk, I will introduce how Mercari uses Spinnaker, current status and feature plan.

taichi nakashima

July 18, 2018
Tweet

More Decks by taichi nakashima

Other Decks in Technology

Transcript

  1. LINE Developer Meetup in Tokyo #40
    18 July 2018
    Continuous Delivery
    with Spinnaker

    View Slide

  2. @deeeet / @tcnksm
    Blog / deeeet.com
    Tech Lead at Mercari
    Microservices Platform Team

    View Slide

  3. We are hiring !

    View Slide

  4. 4
    How Mercari uses Spinnaker for Microservices
    Continuous Delivery ?
    ● What & Why Continuous Delivery?
    ● What & Why Spinnaker?
    ● How Mercari uses Spinnaker? (Current status & future)
    tl;dr

    View Slide

  5. Microservices at Mercari

    View Slide

  6. 6
    While we had lots of benefits of monolith architecture,
    its codebase is getting larger and complex...
    ● Difficult to understand change effects
    ● Difficult to test
    ● Difficult to on-boarding
    ● Difficult to isolate failure
    Why Microservices?

    View Slide

  7. 7
    Even business and organization growth, we want to
    ● Avoid velocity stalled but rather make deployment cycle faster
    ● Increase each developer productivity
    Why Microservices?
    Started to migrate Microservices architecure

    View Slide

  8. 8
    We use GCP for main infrastructure for Microservices
    platform
    ● Each services are deployed to Kubernetes (GKE) as container
    ● Each services uses GCP Managed DB (e.g., Cloud Spanner)
    Microservices Technical Stack

    View Slide

  9. Continuous Delivery

    View Slide

  10. 10
    Continuous Delivery is a software engineering approach in
    which teams produce software in short cycles, ensuring
    that the software can be reliably released at any time.
    What is Continuous Delivery?
    https://en.wikipedia.org/wiki/Continuous_delivery

    View Slide

  11. 11
    Continuous deployment is the next step of continuous
    delivery: Every change that passes the automated tests is
    deployed to production automatically.
    +Continuous Deployment
    https://puppet.com/blog/continuous-delivery-vs-continuous-deployment-what-s-diff

    View Slide

  12. 12
    ● We can provide new feature faster to customers
    ● We can easily triage source of issue when something goes wrong
    ● We can focus on service development itself
    Why Continuous Delivery?

    View Slide

  13. 13
    Challenges at Mercari
    We had some challenges for Continuous Delivery
    ● Continuous Delivery in Microservices architecture
    ● Continuous Delivery on Kubernetes environment

    View Slide

  14. 14
    Challenges: Microservices
    “You build it, You run it!” principle is there
    ● In monolith
    ○ Central team (e.g., SRE) handles all deployment
    ● In microservices
    ○ Each service development team need to maintain and
    customize deployment based on their service characteristics

    View Slide

  15. 15
    You can use kubectl apply but...
    ● To kick kubectl apply from CI is straightforward way but…
    ○ How to know a deployment “Finished”?
    ○ How to handle rollback?
    ○ How to implement complex deployment flow (e.g., canary)?
    Challenges: Kubernetes

    View Slide

  16. 16
    Challenges at Mercari
    We need common tool or platform which allows
    developers easily customize their deployments and
    works with kubernetes

    View Slide

  17. View Slide

  18. 18
    Spinnaker is an open source, multi-cloud continuous
    delivery platform for releasing software changes with
    high velocity and confidence.
    What is Spinnaker?
    https://www.spinnaker.io

    View Slide

  19. 19
    Initially developed internally at Netflix (before Asgard)
    ● Google joined development from 2015
    ● Open sourced at Nov. 2015
    ● Reached 1.0 at Jun. 2017
    ● Latest version is 1.8.1 (at Jul. 2018)
    What is Spinnaker?

    View Slide

  20. 20
    Support Multi-Cloud like GCP and AWS and kubernetes
    ● Opinionated based on Netflix experience and lessons and learned
    ○ Immutable infrastructure, Red-Black deployment
    ● Becoming kinda de-fact tool for Cloud deployment
    What is Spinnaker?

    View Slide

  21. 21

    View Slide

  22. 22
    Spinnaker structures deployments as Pipeline
    ● Each pipeline has configuration defining like
    ○ Stages
    ○ Triggers
    ○ Notifications
    How to use Spinnaker

    View Slide

  23. 23
    Stage is Spinnaker action that form an atomic building
    block for a pipeline
    Spinnaker Pipeline: Stage
    ● Infrastructure stages: Bake, Deploy, Rollback
    ● External system integration: Jenkins, Webhook
    ● Controlling Flow: Manual judgement, Wait, Run pipeline

    View Slide

  24. 24
    Triggers decides how the pipeline is started
    Spinnaker Pipeline: Trigger
    ● Time-based: Cron, Manual
    ● Event-based: Docker Registry, Git, CI, Pipeline, Pub/Sub

    View Slide

  25. 25
    Workflows that are automatically run needs notifications
    to broadcast the status of event
    Spinnaker Pipeline: Notification
    ● When?: Pipeline started, success, failure
    ● Where?: Slack, Email, Pub/Sub

    View Slide

  26. 26
    Spinnaker Pipeline: Example
    Trigger Stage

    View Slide

  27. 27
    A big part of Continuous Delivery practice successful is
    to be able to new code without fear.
    Automation is great. But it’s better with proper
    safeguards is important
    Safeguards

    View Slide

  28. 28
    Spinnaker provides a lot of safeguards
    ● For Deployment
    ○ Deploy strategies (red-black, canary)
    ○ Rollback
    ● For Pipeline
    ○ Pipeline concurrency
    Spinnaker Safeguards

    View Slide

  29. 29
    Spinnaker Safeguards: Stage
    Each steps are considered to be remediated in failure scenarios

    View Slide

  30. 30
    Spinnaker Safeguards: Deploy Window
    Block deployment when peektime

    View Slide

  31. 31
    Spinnaker watches Kubernetes operation is actually
    “Finished” (manifest stability)
    Spinnaker Safeguards: Kubernetes
    ● Spinnaker considered stable when the Kubernetes contorller
    manager no longer needs to modify it
    ○ Deployment is stable when pods are updated and ready
    ○ Service is stable when underlying LB is created

    View Slide

  32. Spinnaker at Mercari

    View Slide

  33. 33
    We used it from 2017 (v1.0)
    ● Currently Using 1.8
    ● Using Kubernetes V1 provider (GUI based)
    ● More than 30 applications are deployed via Spinnaker
    ● Handle deployment for 3 regions: JP, US and UK
    Spinnaker at Mercari

    View Slide

  34. 34
    Spinnaker Architecture
    We prepared dedicated GKE cluster for Spinnaker

    View Slide

  35. 35
    Architecture
    Spinnaker manages multiple region GKE cluster
    us-west1
    europe-west1
    asia-northwest1

    View Slide

  36. 36
    Architecture
    Spinnaker manages both DEV and PROD cluser
    DEV cluster
    PROD cluster

    View Slide

  37. 37
    We have various pipelines
    ● Red-Black (Blue-Green) deployment for Microservices
    ● Canary deployment (without ACA)
    ● Complex Machine Learning model deployment
    ● DB migration Job
    Example Pipelines

    View Slide

  38. 38
    Example Pipelines: DB Migration

    View Slide

  39. 39
    Example Pipelines: Canary

    View Slide

  40. 40
    As admin
    ● Easy provide common best practice to developers as stage
    ○ Admin can encapsulate complex cloud operation
    ○ We want to build own custom stage in future!
    ● Everything is in one place
    The Good

    View Slide

  41. 41
    As developers
    ● Easy to setup own customized pipeline (no custom scripting)
    ● Easy to refer other team practice
    ● No need to learn whole kubernetes YAML manifest
    The Good

    View Slide

  42. 42
    As admin
    ● Too unstable (v1.7 was worst experience..)
    ● Spinnaker management itself by halyard
    ○ Difficult to review and automate
    ● Another cluster problem effect to the others ..
    ● Less documentation (need to ask via Slack or GitHub Issues)
    The Bad

    View Slide

  43. 43
    As developers
    ● Too many hates about GUI
    ○ Not intuitive… difficult to know where to change
    ○ Impossible to review changes
    ○ Difficult to copy & paste from other team pipeline
    ○ Difficult to share knowledge...
    The Bad
    I wonder how Netflix does it well (they has 9000 pipelines!)

    View Slide

  44. Next Steps

    View Slide

  45. 45
    We want to manage everything by declarative way
    ● Introduce Kubernetes V2 provider
    ● Introduce Declarative Continuous Delivery (DCD spec)
    Declarative management
    Make GUI read-only!

    View Slide

  46. 46
    We want to add more useful stages for developers
    ● Integration testing stage
    ● Squeeze testing stage
    ● Automated canary analysis (ACA) stage
    More stages

    View Slide

  47. 47
    We want to provide a training session to developers
    ● What is Kubernetes? How to use?
    ● What is Spinnaker? How to use?
    On-boarding training
    Documentation documentation documentation...

    View Slide

  48. 48
    Introduced how Mercari uses Spinnaker for
    Microservices Continuous Delivery
    Conclusion

    View Slide