Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Cookpad Tech Kitchen #20 クックパッドでのサービスメッシュについて
Search
taiki45
November 28, 2018
Technology
1
2.2k
Cookpad Tech Kitchen #20 クックパッドでのサービスメッシュについて
Cookpad Tech Kitchen #20 クックパッドのマイクロサービスプラットフォーム現状
https://cookpad.connpass.com/event/106913/
taiki45
November 28, 2018
Tweet
Share
More Decks by taiki45
See All by taiki45
SPIFFE Meetup Tokyo #2 LT: Envoy SDS
taiki45
0
540
builderscon Tokyo 2019: Intro Service Mesh
taiki45
6
3k
NoOps Meetup Tokyo #7: 入門サービスメッシュ
taiki45
4
1.7k
CloudNative Days Tokyo 2019: Understanding Envoy
taiki45
3
3.2k
Cloud Native Meetup Tokyo #8 ServiceMesh Day Recap
taiki45
2
300
EnvoyCon 2018: Building and operating service mesh at mid-size company
taiki45
4
4.2k
Building and operating service mesh at mid-size company
taiki45
2
2.2k
Observability, Service Mesh and Microservices
taiki45
25
6.2k
Draft: Observability, Service Mesh and Microservices
taiki45
1
580
Other Decks in Technology
See All in Technology
ルーターでプレゼンする
puhitaku
1
3.4k
One engineer company with Ruby on Rails
rstankov
2
460
Kaggleで学ぶ系列データのための深層学習モデリング
yu4u
6
820
モーダル間の変換後の一致性とジャンル表を用いた解釈可能性の考察 ~Text-to-MusicとText-To-ImageかつImage-to-Musicを例に~
otanet
0
320
Cloud Service Mesh に触れ合う
phaya72
1
300
LLM開発・活用の舞台裏@2024.04.25
yushin_n
3
1.4k
LayerXにおけるLLMプロダクト開発の今までとこれから
layerx
PRO
4
1.4k
IaCジェネレーターとBedrockで詳細設計書を生成してみた
tsukasa_ishimaru
4
950
今年のRubyKaigiはProfiler Year🤘
osyoyu
0
510
20分で完全に理解するGrafanaダッシュボード
hamadakoji
5
990
IaCからAWSに入門した初心者が CloudFormationを通して考えた「AWS操作」の使い分け
maimyyym
3
620
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
5
37k
Featured
See All Featured
VelocityConf: Rendering Performance Case Studies
addyosmani
321
23k
Embracing the Ebb and Flow
colly
80
4.2k
Building Adaptive Systems
keathley
32
1.9k
How to name files
jennybc
65
93k
Designing on Purpose - Digital PM Summit 2013
jponch
111
6.5k
Writing Fast Ruby
sferik
622
60k
Being A Developer After 40
akosma
67
580k
Designing for Performance
lara
601
67k
The Invisible Customer
myddelton
114
12k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
245
20k
The MySQL Ecosystem @ GitHub 2015
samlambert
244
12k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
155
14k
Transcript
クックパッドでの サービスメッシュについて Taiki Ono, Cookpad Inc.
Agenda • Background • Problems • Introducing and operations •
Key results • Next challenges
Background
Cookpad • "Make everyday cooking fun!" • Originally started in
Japan in 1997 • Operate in over 23 languages, 68 countries
Scale • 200+ product developers • 100+ production services •
90M Monthly Average User
Organization structure Service Team SRE team etc
Technology stack • Ruby on Rails for both web frontend
and backend apps • Python for ML apps • Go for backend app • Rust, Swift, Java etc.. for internal apps
None
Problems
Operational problems • Decrease in system reliability • Hard to
trouble shoot and debug ‣ Increase of time detect root causes of incidents ‣ Capacity planing
Solutions • Expeditor ‣ Ruby library inspired by Netflix's Hystrix
• aws-xray ‣ Ruby library for distributed tracing using AWS's X-Ray service https://github.com/cookpad
http://techlife.cookpad.com/entry/2017/09/06/115710
None
GoPythonRustJavaSwift apps? • Limitation of library model approach ‣ More
for product development ‣ Controlling library versions is hard • Planning to develop our proxy and mixed with consul-template
Service mesh to the rescue
at SRECON America 2017 "Lyft's Envoy: Experiences Operating a Large
Service Mesh"
Replacing libraries to proxy
control-plane
Introducing and operating service mesh
Timeline • Early 2017: making plan • Late 2017: building
MVP • Early 2018: generally available
Envoy • Publicity released at mid 2016 • Lightweight •
Graceful reloading • gRPC support https://github.com/envoyproxy/envoy
Plan: in-house • Early 2017: no Istio • We use
Amazon ECS • Not using full features of Envoy • Resiliency and observability parts
Goals • Control resiliency settings by Ops ‣ Centrally managed
‣ Review flow • All metrics should go into Prometheus • Low operation cost ‣ Less components, use of managed services
None
Configuration contents • Jsonnet • Route config ‣ Retry, timeouts
for paths, domains • Cluster config ‣ DNS name of internal ELB ‣ Circuit breaker settings https://github.com/cookpad/kumonos
Drop statsd-relay • Adding tags to metrics with DogStatsd format
• Less component is preferable ‣ Send PRs to Envoy ‣ dog_statsd sink and fixed tag configuration are available
gRPC infrastructure • Need L7 proxy for HTTP/2 traffic •
Let's extend control-plane
ServiceDiscoveryService API • lyft/discovery ‣ Reference implementation of SDS API
• Moved to cookpad/sds
The hard point of ECS • Copy current ECS service
• Wait then switch • Delete old one
Generally available
Operations
Dashboards • Prometheus • Grafana ‣ Per service ‣ Per
servie-to-service ‣ Envoy instances • Vizceral ‣ promviz, promviz-front
None
None
None
None
None
Envoy on EC2 • Build and distribute as a in-house
deb package • Manage as a systemd service • Use hot-restarter.py ‣ Generate starter script for each host role
wait-side-car • Sidecar Envoy containers need a few seconds to
be up ‣ For background jobs • Wrapper command-line tool ‣ cookpad/wait-side-car https://github.com/cookpad/wait-side-car
https://techlife.cookpad.com/entry/2018/04/02/140846
Key results
Resiliency • Eliminates temporal burst of errors from backend services
• Speed of reviewing settings and deployment • Fault isolation: not yet remarkable result
Observability • Decrease of time to detect root causes around
service communication issues • Visualization of how resilience mechanism is working • One of sources of Service Level Indicator
Continuous Growth of platform • Improve application platform without application
deployment • Increase velocity of platform development team
Next challenges
Next challenges • v2 xDS migration • More effective traffic
control • Chaos engineering platform • Distributed tracing • Auth[z, n]
Wrap up
Wrap up • Issues around service communications • Introducing service
mesh instead of doing library approach • Key results: resiliency, observability, platform improvement
Q&A • Twitter: @taiki45 • http://techlife.cookpad.com/ • EnvoyCon 2018 https://
events.linuxfoundation.org/events/ kubecon-cloudnativecon-north- america-2018/co-located-events/ envoycon/