Building and operating service mesh at mid-size company

For builderscon tokyo 2018: https://builderscon.io/tokyo/2018/session/4bf634b9-cc92-4499-ba50-3ff6304e4657

Transcript

1. Building and operating service mesh at mid-size company Taiki Ono, Cookpad Inc.
2. Agenda • Background • Problems • Introducing and operations • Key results • Next challenges
3. Background
4. Cookpad • "Make everyday cooking fun!" • Originally started in Japan in 1997 • Operate in over 23 languages, 68 countries
5. Scale • 200+ product developers • 100+ production services • 90M Monthly Average User
6. Organization structure
7. Technology stack • Ruby on Rails for both web frontend and backend apps • Python for ML apps • Go for backend app • Rust, Swift, Java etc.. for internal apps
9. Problems
10. Operational problems • Decrease in system reliability • Hard to trouble shoot and debug ‣ Increase of time detect root causes of incidents ‣ Capacity planing
11. Solutions • Expeditor ‣ Ruby library inspired by Netflix's Hystrix • aws-xray ‣ Ruby library for distributed tracing using AWS's X-Ray service https://github.com/cookpad
12. http://techlife.cookpad.com/entry/2017/09/06/115710
14. GoPythonRustJavaSwift apps? • Limitation of library model approach ‣ More for product development ‣ Controlling library versions is hard • Planning to develop our proxy and mixed with consul-template
15. Service mesh to the rescue
16. at SRECON America 2017 "Lyft's Envoy: Experiences Operating a Large Service Mesh"
17. Replacing libraries to proxy
18. control-plane
19. Introducing and operating service mesh
20. Timeline • Early 2017: making plan • Late 2017: building MVP • Early 2018: generally available
21. Envoy • Publicity released at mid 2016 • Lightweight • Graceful reloading • gRPC support https://github.com/envoyproxy/envoy
22. Plan: in-house • Early 2017: no Istio • We use Amazon ECS • Not using full features of Envoy • Resiliency and observability
23. Goals • Control resiliency settings by Ops ‣ Centrally managed ‣ Review flow • All metrics should go into Prometheus • Low operation cost ‣ Less components, use of managed services
25. Configuration contents • Jsonnet • Route config ‣ Retry, timeouts for paths, domains • Cluster config ‣ DNS name of internal ELB ‣ Circuit breaker settings https://github.com/cookpad/kumonos
27. Drop statsd-relay • Adding tags to metrics with DogStatsd format • Less component is preferable ‣ Send PRs to Envoy ‣ dog_statsd sink and fixed tag configuration are available
28. gRPC infrastructure • Need L7 proxy for HTTP/2 traffic • Let's extend control-plane
29. ServiceDiscoveryService API • lyft/discovery ‣ Reference implementation of SDS API • Moved to cookpad/sds
30. The hard point of ECS • Copy current ECS service • Wait then switch • Delete old one
31. Generally available
32. Operations
33. Dashboards • Prometheus • Grafana ‣ Per service ‣ Per servie-to-service ‣ Envoy instances • Vizceral ‣ promviz, promviz-front
39. Envoy on EC2 • Build and distribute as a in-house deb package • Manage as a systemd service • Use hot-restarter.py ‣ Generate starter script for each host role
40. wait-side-car • Sidecar Envoy containers need a few seconds to be up ‣ For background jobs • Wrapper command-line tool ‣ cookpad/wait-side-car https://github.com/cookpad/wait-side-car
41. https://techlife.cookpad.com/entry/2018/04/02/140846
42. Key results
43. Resiliency • Eliminates temporal burst of errors from backend services • Speed of reviewing settings and deployment • Fault isolation: not yet remarkable result
44. Observability • Decrease of time to detect root causes around service communication issues • Visualization of how resilience mechanism is working • One of sources of Service Level Indicator
45. Growth of platform • Improve application platform without application deployment • Increase velocity of platform development team
46. Next challenges
47. Next challenges • v2 xDS migration / Istio • Chaos engineering platform • Distributed tracing • Auth[z, n]
48. Wrap up
49. Wrap up • Issues around service communications • Introducing service mesh instead of doing library approach • Key results: resiliency, observability, platform improvement
50. Q&A • Twitter: @taiki45 • Publish this slide later • http://techlife.cookpad.com/