Building and operating service mesh at mid-size company

44e6e0e9bcc3d8279020aad563f16f34?s=47 taiki45
September 07, 2018

Building and operating service mesh at mid-size company

44e6e0e9bcc3d8279020aad563f16f34?s=128

taiki45

September 07, 2018
Tweet

Transcript

  1. 4.

    Cookpad • "Make everyday cooking fun!" • Originally started in

    Japan in 1997 • Operate in over 23 languages, 68 countries
  2. 7.

    Technology stack • Ruby on Rails for both web frontend

    and backend apps • Python for ML apps • Go for backend app • Rust, Swift, Java etc.. for internal apps
  3. 8.
  4. 10.

    Operational problems • Decrease in system reliability • Hard to

    trouble shoot and debug ‣ Increase of time detect root causes of incidents ‣ Capacity planing
  5. 11.

    Solutions • Expeditor ‣ Ruby library inspired by Netflix's Hystrix

    • aws-xray ‣ Ruby library for distributed tracing using AWS's X-Ray service https://github.com/cookpad
  6. 13.
  7. 14.

    GoPythonRustJavaSwift apps? • Limitation of library model approach ‣ More

    for product development ‣ Controlling library versions is hard • Planning to develop our proxy and mixed with consul-template
  8. 20.

    Timeline • Early 2017: making plan • Late 2017: building

    MVP • Early 2018: generally available
  9. 21.

    Envoy • Publicity released at mid 2016 • Lightweight •

    Graceful reloading • gRPC support https://github.com/envoyproxy/envoy
  10. 22.

    Plan: in-house • Early 2017: no Istio • We use

    Amazon ECS • Not using full features of Envoy • Resiliency and observability
  11. 23.

    Goals • Control resiliency settings by Ops ‣ Centrally managed

    ‣ Review flow • All metrics should go into Prometheus • Low operation cost ‣ Less components, use of managed services
  12. 24.
  13. 25.

    Configuration contents • Jsonnet • Route config ‣ Retry, timeouts

    for paths, domains • Cluster config ‣ DNS name of internal ELB ‣ Circuit breaker settings https://github.com/cookpad/kumonos
  14. 26.
  15. 27.

    Drop statsd-relay • Adding tags to metrics with DogStatsd format

    • Less component is preferable ‣ Send PRs to Envoy ‣ dog_statsd sink and fixed tag configuration are available
  16. 30.

    The hard point of ECS • Copy current ECS service

    • Wait then switch • Delete old one
  17. 33.

    Dashboards • Prometheus • Grafana ‣ Per service ‣ Per

    servie-to-service ‣ Envoy instances • Vizceral ‣ promviz, promviz-front
  18. 34.
  19. 35.
  20. 36.
  21. 37.
  22. 38.
  23. 39.

    Envoy on EC2 • Build and distribute as a in-house

    deb package • Manage as a systemd service • Use hot-restarter.py ‣ Generate starter script for each host role
  24. 40.

    wait-side-car • Sidecar Envoy containers need a few seconds to

    be up ‣ For background jobs • Wrapper command-line tool ‣ cookpad/wait-side-car https://github.com/cookpad/wait-side-car
  25. 43.

    Resiliency • Eliminates temporal burst of errors from backend services

    • Speed of reviewing settings and deployment • Fault isolation: not yet remarkable result
  26. 44.

    Observability • Decrease of time to detect root causes around

    service communication issues • Visualization of how resilience mechanism is working • One of sources of Service Level Indicator
  27. 45.
  28. 47.

    Next challenges • v2 xDS migration / Istio • Chaos

    engineering platform • Distributed tracing • Auth[z, n]
  29. 48.
  30. 49.

    Wrap up • Issues around service communications • Introducing service

    mesh instead of doing library approach • Key results: resiliency, observability, platform improvement