Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Cookpad Tech Kitchen #20 クックパッドでのサービスメッシュについて
Search
taiki45
November 28, 2018
Technology
1
2.4k
Cookpad Tech Kitchen #20 クックパッドでのサービスメッシュについて
Cookpad Tech Kitchen #20 クックパッドのマイクロサービスプラットフォーム現状
https://cookpad.connpass.com/event/106913/
taiki45
November 28, 2018
Tweet
Share
More Decks by taiki45
See All by taiki45
Mocking in Rust Applications
taiki45
2
590
Error Handling in Rust Applications
taiki45
3
630
Efficient Platform for Security and Compliance
taiki45
4
1.4k
RustでAWS Lambda functionをいい感じに書く
taiki45
2
650
SPIFFE Meetup Tokyo #2 LT: Envoy SDS
taiki45
0
750
builderscon Tokyo 2019: Intro Service Mesh
taiki45
6
3.4k
NoOps Meetup Tokyo #7: 入門サービスメッシュ
taiki45
4
1.9k
CloudNative Days Tokyo 2019: Understanding Envoy
taiki45
3
3.5k
Cloud Native Meetup Tokyo #8 ServiceMesh Day Recap
taiki45
2
380
Other Decks in Technology
See All in Technology
「Chatwork」の認証基盤の移行とログ活用によるプロダクト改善
kubell_hr
1
200
AIのAIによるAIのための出力評価と改善
chocoyama
2
580
Amazon ECS & AWS Fargate 運用アーキテクチャ2025 / Amazon ECS and AWS Fargate Ops Architecture 2025
iselegant
17
5.7k
本が全く読めなかった過去の自分へ
genshun9
0
580
Delegating the chores of authenticating users to Keycloak
ahus1
0
130
製造業からパッケージ製品まで、あらゆる領域をカバー!生成AIを利用したテストシナリオ生成 / 20250627 Suguru Ishii
shift_evolve
PRO
1
140
Leveraging Open-Source Tools for Creating 3D Tiles in the Urban Environment
simboss
PRO
0
100
第9回情シス転職ミートアップ_テックタッチ株式会社
forester3003
0
250
Understanding_Thread_Tuning_for_Inference_Servers_of_Deep_Models.pdf
lycorptech_jp
PRO
0
140
Oracle Audit Vault and Database Firewall 20 概要
oracle4engineer
PRO
3
1.7k
Github Copilot エージェントモードで試してみた
ochtum
0
110
GitHub Copilot の概要
tomokusaba
1
140
Featured
See All Featured
Typedesign – Prime Four
hannesfritz
42
2.7k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
8
670
Optimizing for Happiness
mojombo
379
70k
Thoughts on Productivity
jonyablonski
69
4.7k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.8k
Facilitating Awesome Meetings
lara
54
6.4k
Agile that works and the tools we love
rasmusluckow
329
21k
For a Future-Friendly Web
brad_frost
179
9.8k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
Faster Mobile Websites
deanohume
307
31k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Transcript
クックパッドでの サービスメッシュについて Taiki Ono, Cookpad Inc.
Agenda • Background • Problems • Introducing and operations •
Key results • Next challenges
Background
Cookpad • "Make everyday cooking fun!" • Originally started in
Japan in 1997 • Operate in over 23 languages, 68 countries
Scale • 200+ product developers • 100+ production services •
90M Monthly Average User
Organization structure Service Team SRE team etc
Technology stack • Ruby on Rails for both web frontend
and backend apps • Python for ML apps • Go for backend app • Rust, Swift, Java etc.. for internal apps
None
Problems
Operational problems • Decrease in system reliability • Hard to
trouble shoot and debug ‣ Increase of time detect root causes of incidents ‣ Capacity planing
Solutions • Expeditor ‣ Ruby library inspired by Netflix's Hystrix
• aws-xray ‣ Ruby library for distributed tracing using AWS's X-Ray service https://github.com/cookpad
http://techlife.cookpad.com/entry/2017/09/06/115710
None
GoPythonRustJavaSwift apps? • Limitation of library model approach ‣ More
for product development ‣ Controlling library versions is hard • Planning to develop our proxy and mixed with consul-template
Service mesh to the rescue
at SRECON America 2017 "Lyft's Envoy: Experiences Operating a Large
Service Mesh"
Replacing libraries to proxy
control-plane
Introducing and operating service mesh
Timeline • Early 2017: making plan • Late 2017: building
MVP • Early 2018: generally available
Envoy • Publicity released at mid 2016 • Lightweight •
Graceful reloading • gRPC support https://github.com/envoyproxy/envoy
Plan: in-house • Early 2017: no Istio • We use
Amazon ECS • Not using full features of Envoy • Resiliency and observability parts
Goals • Control resiliency settings by Ops ‣ Centrally managed
‣ Review flow • All metrics should go into Prometheus • Low operation cost ‣ Less components, use of managed services
None
Configuration contents • Jsonnet • Route config ‣ Retry, timeouts
for paths, domains • Cluster config ‣ DNS name of internal ELB ‣ Circuit breaker settings https://github.com/cookpad/kumonos
Drop statsd-relay • Adding tags to metrics with DogStatsd format
• Less component is preferable ‣ Send PRs to Envoy ‣ dog_statsd sink and fixed tag configuration are available
gRPC infrastructure • Need L7 proxy for HTTP/2 traffic •
Let's extend control-plane
ServiceDiscoveryService API • lyft/discovery ‣ Reference implementation of SDS API
• Moved to cookpad/sds
The hard point of ECS • Copy current ECS service
• Wait then switch • Delete old one
Generally available
Operations
Dashboards • Prometheus • Grafana ‣ Per service ‣ Per
servie-to-service ‣ Envoy instances • Vizceral ‣ promviz, promviz-front
None
None
None
None
None
Envoy on EC2 • Build and distribute as a in-house
deb package • Manage as a systemd service • Use hot-restarter.py ‣ Generate starter script for each host role
wait-side-car • Sidecar Envoy containers need a few seconds to
be up ‣ For background jobs • Wrapper command-line tool ‣ cookpad/wait-side-car https://github.com/cookpad/wait-side-car
https://techlife.cookpad.com/entry/2018/04/02/140846
Key results
Resiliency • Eliminates temporal burst of errors from backend services
• Speed of reviewing settings and deployment • Fault isolation: not yet remarkable result
Observability • Decrease of time to detect root causes around
service communication issues • Visualization of how resilience mechanism is working • One of sources of Service Level Indicator
Continuous Growth of platform • Improve application platform without application
deployment • Increase velocity of platform development team
Next challenges
Next challenges • v2 xDS migration • More effective traffic
control • Chaos engineering platform • Distributed tracing • Auth[z, n]
Wrap up
Wrap up • Issues around service communications • Introducing service
mesh instead of doing library approach • Key results: resiliency, observability, platform improvement
Q&A • Twitter: @taiki45 • http://techlife.cookpad.com/ • EnvoyCon 2018 https://
events.linuxfoundation.org/events/ kubecon-cloudnativecon-north- america-2018/co-located-events/ envoycon/