Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
SRE-Lounge-8-Cookpad-Microservice-Architecture-...
Search
rrreeeyyy
March 13, 2019
Technology
5
5.2k
SRE-Lounge-8-Cookpad-Microservice-Architecture-Overview
SRE Lounge #8 でお話させていただきました
rrreeeyyy
March 13, 2019
Tweet
Share
More Decks by rrreeeyyy
See All by rrreeeyyy
An Efficient Incident Response Training with AI / SRE NEXT 2024 Sponsor Session
rrreeeyyy
1
2.7k
カンファレンスから見る SRE トレンド 2024 / SRE Trends from Conferences in 2024 #SRE_Findy
rrreeeyyy
4
2.1k
信頼性の育て方 / mackerel-meetup-15
rrreeeyyy
9
2.3k
SRE の歩き方・進め方 / sre-walk-through-procedure
rrreeeyyy
0
8.5k
「信頼性」を保ちつつ大規模サービスをリニューアルする / cookpad-tech-kitchen-service-embedded-sres
rrreeeyyy
11
12k
Cookpad and Prometheus
rrreeeyyy
6
20k
A survey of anomaly detection methodologies for web system
rrreeeyyy
5
1.2k
エンジニアリングをちゃんとやる あるいは 人類の平和 について / wsa02-rrreeeyyy
rrreeeyyy
13
3k
「自立」したWebシステムを創る。自分の好きなことをする世界を目指して。/ ipsj-one-2018-rrreeeyyy
rrreeeyyy
3
1.8k
Other Decks in Technology
See All in Technology
可視化により内部品質をあげるAIドキュメントリバース/20240910 Hiromitsu Akiba
shift_evolve
0
220
ついに出た!OpenAIの最新モデル「o1」って何がすごいの?
minorun365
PRO
3
1k
開発者の定量・定性データを組み合わせて開発者体験を把握するための取り組み
ham0215
1
130
言葉は感情の近似値である。その感情と言葉の誤差を最小化しよう ~コミュニケーションにおけるアナログ/デジタル変換の課題に立ち向かう~
nktamago
0
220
DevRelの始め方
moongift
PRO
1
390
PdMはどのように全てのスピードを上げられるか ~ 非連続進化のための具体的な取り組み ~
sansantech
PRO
4
1.3k
忙しい人のためのLangGraph概要まとめ
__ymgc__
1
190
開発生産性を始める前に開発チームができること / optim-improve-development-productivity.pdf
optim
0
110
eBPFのこれまでとこれから
yutarohayakawa
10
3.2k
Tricentisにおけるテスト自動化へのAI活用ご紹介/20240910Shunsuke Katakura
shift_evolve
0
200
持続可能なソフトウェア開発を支える『GitHub CI/CD実践ガイド』
tmknom
6
1.4k
Agile in Automotive Industry, puzzles and lights.
hiranabe
3
1.4k
Featured
See All Featured
Design by the Numbers
sachag
277
19k
Imperfection Machines: The Place of Print at Facebook
scottboms
263
13k
The Invisible Customer
myddelton
119
13k
Unsuck your backbone
ammeep
667
57k
Fantastic passwords and where to find them - at NoRuKo
philnash
48
2.8k
GitHub's CSS Performance
jonrohan
1030
450k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
27
7.4k
Pencils Down: Stop Designing & Start Developing
hursman
119
11k
Fireside Chat
paigeccino
31
2.9k
The Invisible Side of Design
smashingmag
295
50k
RailsConf 2023
tenderlove
28
810
5 minutes of I Can Smell Your CMS
philhawksworth
202
19k
Transcript
$PPLQBE.JDSPTFSWJDF "SDIJUFDUVSF0WFSWJFX ؙحؙػحس吳䒭⠓爡䪮遭鿇43&ؚٕ٦ف し䊛畆㣕 !SSSFFFZZZ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
!SSSFFFZZZ !SSSFFFZZZ IUUQTSSSFFFZZZDPN :PTIJLBXB3ZPUB .F ˖ :PTIJLBXB3ZPUB !SSSFFFZZZ[reɪ] ˖ ؙحؙػحس吳䒭⠓爡
։ ˖ 䪮遭鿇43&ؚٕ٦ف ˖ 莆걄㚖 ˖ 0CTFSWBCJMJUZ 5JNFTFSJFTEBUBCBTF ˖ 馯 ˖ QVZP FPS[FB MFBHVFPGMFHFOET HCG ˖ TFJZV SFTUBHF XVH 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE43&T ˖ 秈せ +1 ˖ 㛇劤涸ח傈劤㕂ⰻד ˖ 秈せ (MPCBM ˖
؎ؘٔأ׃ֻכ傈劤㕂ⰻד 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE +1(MPCBM ˖ +BQBO ˖ IUUQTDPPLQBEDPN ˖ (MPCBM $PVOUSJFT-BOHVBHFT ˖
IUUQTDPPLQBEDPNVT ˖ IUUQTDPPLQBEDPNVL ˖ IUUQTDPPLQBEDPNJE ˖ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE +1(MPCBM ˖ +BQBO ˖ IUUQTDPPLQBEDPN ˖ (MPCBM ˖ ֿך《穈כGPPUOPUFTך⚛ןח撑׃גְֻׁ
IUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSFOETFBTPO IUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE43&Tך顑⟣眔㔲 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
؎ً٦آ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
植㹋 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.JDSPTFSWJDFTBOE$PPLQBE 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.JDSPTFSWJDFTBOE$PPLQBE ˖ 䎃׀ַو؎ؙٗ؟٦ؽأ⻉ח《穈דֹ ˖ ؝ىُص؛٦ءّٝ٥顑⟣眔㔲٥涪⸬桦זו㼎ׅ㉏겗䠐陎 ˖ 㣐ֹז⽃♧ך؝٦سك٦أ٥؟٦ؽأ㔿剣ך㉏겗ָֻׁ֮ ˖ و؎ؙٗ؟٦ؽأ⻉ָ鹌ֿהח״㣐ֹזًٔحزך❦「
˖ 涪鸞䏝ךぢ♳װ僇然ז顑⟣眔㔲٥ٔٔ٦أ鸞䏝ךぢ♳זו ˖ 㼭ְׁ؝٦سك٦أ٥؟٦ؽأזדכך兛鸐ך涪ָ遤ִ״ֲח 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.JDSPTFSWJDFTBOE$PPLQBE ˖ ♧倯דծو؎ؙٗ؟٦ؽأ⻉鹌ֿהד饯ֹ㉏겗֮ ˖ 43&ָⰋגך؟٦ؽأ鋅הأ؛٦ٕ׃זְ ˖ ؟٦ؽأ鸐⥋ך㉏겗٥鸬ꓲ涸זꥺ㹱٥〳錁庠䚍ך㉏겗 ˖ 43&ך⚺噟ך♧אכְֲֿ㉏겗ח㼎ׅ،فٗ٦ث
˖ ֿתדוך״ֲח׃ג،فٗ٦ث׃גַֹ稱➜ ˖ 4DBMBCJMJUZ "WBJMBCJMJUZך錁挿ַ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
4DBMBCJMJUZ ˖ أ؛٦ٓؽٔذ؍ծה鎉ג圫ղז⽃⡘ָ֮ ˖ ➂ ˖ ؟٦غ ˖ ؝أز 43&-PVOHF
]:PTIJLBXB3ZPUB !SSSFFFZZZ
4DBMBCJMJUZ ˖ ➂ ˖ إٕؿ؟٦ؽأ⻉鹌 ˖ ؟٦غ ˖ "VUP4DBMJOHך崞欽 ˖
؝أز ˖ Ⱏ鸐㛇湍ך4QPU⻉ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
إٕؿ؟٦ؽأ⻉ ˖ ؙحؙػحسדכ"NB[PO&$4ⵃ欽׃ג؝ٝذشךⰟ鸐㛇湍⡲גְ ˖ ،فٔ؛٦ءّٝך㹀纏װرفٗ؎חכIBLPⵃ欽׃גְ ˖ 㛇劤涸ח؟٦ؽأך涪罏ָ㹀纏剅ֹرفٗ؎דֹ״ֲחזגְ ˖ 㹀纏ؿ؋؎ٕך剅䒭חכ+TPOOFUⵃ欽׃גְ ˖
"84ך؟٦ؽأDPEFOJ[FUPPMTװ5FSSBGPSN⢪ג؝٦س⻉׃גְ ˖ 涪罏כ䗳銲ד֮ל荈歋ח13⳿ֿׅהָדֹ״ֲחזגְ IUUQTDPEFOJ[FUPPMT IUUQTHJUIVCDPNFBHMFUNUIBLP 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
؝ٝذشⰟ鸐㛇湍 ˖ &$4؎ٝأةٝأכ4QPU'MFFU"VUP4DBMJOH(SPVQד盖椚ׁגְ ˖ &$4؎ٝأةٝأךקהוכ4QPUד⹛ְגְ ˖ ؝أز٥鎘皾项彁ד㣐ֹזًٔحزָ欰תגְ ˖ ؎ٝأةٝأךٔا٦أ٥&$4؟٦ؽأך朐䡾זו然钠׃גأ؛٦ٕ،ؐز׃גְ ˖
ⰻ醡ד؟٦ؽأך朐䡾然钠דֹ8FC؝ٝا٦ٕ⡲גְ ˖ ؟٦ؽأ荈⡤ךًزؙٔأװٗ٦سغٓٝ؟ךؙٔؒأز٥ٖأهٝأךًزؙٔأזו ˖ ،فٔ؛٦ءّٝךؚٗךꠘ鋮٥嗚稊זו IUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF IUUQTTQFBLFSEFDLDPNFBHMFUNUCVJMEJOHBTUFBEZFDTJOGSBTUSVDUVSF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ scheduler: { type: 'ecs', cluster: 'eagletmt', desired_count: 2, role:
'ecsServiceRole', elb_v2: { vpc_id: 'vpc-WWWWWWWW', health_check_path: '/site/sha', listeners: [ { port: 443, protocol: 'HTTPS', certificate_arn: 'arn:aws:iam::012345678901:server-certificate/hello-lb-v2.example.com', }, ], subnets: ['subnet-XXXXXXXX', 'subnet-YYYYYYYY'], security_groups: ['sg-ZZZZZZZZ'], : }, }, app: { image: 'ryotarai/hello-sinatra', : secrets: [{ name: 'MESSAGE', value_from: 'arn:aws:ssm:ap-northeast-1:012345678901:parameter/hako/hello-lb-v2/secret-message', }], }, sidecars: { front: { image_tag: 'hako-nginx', : }, }, : } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
"WBJMBCJMJUZ ˖ ؟٦ؽأך鸐⥋חֶֽ醱꧟䚍〳腉זꣲ㼰זֻׅ ˖ 4FSWJDFNFTIך㼪Ⰵ ˖ 黝ⴖזٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ ˖ ꥺ㹱ָ饯ֹהֹח.553ָ〳腉זꣲ瀉ְ״ֲחׅ ˖
.POJUPSJOH 0CTFSWBCJMJUZך然⥂זו ˖ $IBPT&OHJOFFSJOHפך《穈 ˖ "-JGFPG$IBPT&OHJOFFSJOH4UBSUJOHXJUI3FTJMJFODF IUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
4FSWJDFNFTIך《穈 ˖ ؟٦ؽأ鸐⥋ךקרⰋגד&OWPZָ欽ְגְ ˖ ،فٔ؛٦ءّٝכ➭ך؟٦ؽأח鸐⥋ׅ儗&OWPZ鸐ׅ״ֲחׅ ˖ 黝ⴖחٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ָ涪⹛ׅ✲דِ٦ؠ䕦갟ך֮♧儗涸זؒٓ٦ָ慧幾 ˖ ؟٦ؽأ鸐⥋ך➭ח鸐䌢ך5$11SPYZה׃ג⢪גְ ˖
&OWPZך$POUSPM1MBOFך؝ٝه٦طٝزה׃גDPPLQBEJUBDIP DPPLQBETET欽ְגְ ˖ ٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ך鏣㹀זו♧⯋ד盖椚׃גְ ˖ 鏣㹀ך鎸岀חכ+TPOOFUⵃ欽׃גְ IUUQTHJUIVCDPNDPPLQBEJUBDIP IUUQTHJUIVCDPNDPPLQBETET IUUQTUFDIMJGFDPPLQBEDPNFOUSZ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.POJUPSJOH 0CTFSWBCJMJUZפך《穈 ˖ 䖞勻ך؎ٝأةٝأך湊鋔כ׆ה֮;BCCJY⢪גְ ˖ 㢩䕎湊鋔ה׃גך4UBUVT$BLFװ鎉铂׀הך鑫稢ז"1.ה׃גך/FX3FMJDזו֮ ˖ ⟃♴ך״ֲזًزؙٔأכ1SPNFUIFVTח꧊ג鑫稢ח鋅״ֲח׃גְ ˖ ؎ٝأةٝأ٥ىسٕؐؑ،ך㛇劤涸זًزؙٔأ
node_exporter ぐىسٕؐؑ،׀הךFYQPSUFS ˖ ؝ٝذش橆㞮ך㛇劤涸זًزؙٔأ cAdvisor ٥&OWPZַ《؟٦ؽأ鸐⥋ךًزؙٔأ ˖ 1SPNFUIFVTח꧊ًزؙٔأַ،ٓ٦زׅ➬穈 ˖ +TPOOFU欽ְג3VMFך㹀纏"MFSUNBOBHFSך鏣㹀剅ֽ״ֲח׃גְ ˖ "8493BZװ7J[DFSBMזו欽ְ%JTUSJCVUFE5SBDJOH 7JTVBMJ[BUJPOך➬穈 IUUQTHJUIVCDPNSSSFFFZZZEPDLFSQSPNFUIFVTNJYJOUSFFNBTUFSFYBNQMF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ prometheusAlerts+:: { groups+: [ { name: 'example-alerts', rules: [
{ alert: 'ExampleCriticalAlertPrometheusDownFor1min', // PromQL expr: ||| up{job="prometheus"} == 0 ||| % $._config, 'for': '1m', labels: { pager: 'sre', severity: 'critical', }, annotations: { title: '{{$labels.job}} is unreachable', description: '{{$labels.job}} at {{$labels.instance}} could not be scraped for over 1 minutes.', runbook: 'https://wiki.example.com/pages/runbook/ExampleCriticalAlertPrometheusDownFor1min' }, }, ], }, ], }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ alertmanagerConfig+:: { route+: { routes+: [{ match_re: { pager:
'^(sre)$' }, receiver: 'sre-slack', routes: [{ match: { severity: 'critical' }, receiver: 'sre-pagerduty', }], }], }, }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ alertmanagerConfig+:: { receivers+: [ { name: 'sre-slack', slack_configs: [
{ send_resolved: true, channel: '#sre', }, ], }, ], }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ alertmanagerConfig+:: { receivers+: [ { name: 'sre-pagerduty', pagerduty_configs: [
{ service_key: '{{ template "secrets.pagerduty.sre.service_key" }}' }, ], }, ], }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
תתדֹֿה٥װְֿהָ֮ ˖ 刿זإٕؿ؟٦ؽأ⻉ ˖ ぐ؟٦ؽأפךث٦ي䋆ֹ鴥4-*4-0ך㼪Ⰵ ˖ 刿ז؟٦ؽأًحءُפך《穈 ˖ 刿ז؝ٝذشⰟ鸐㛇湍ך何㊣פך《穈 ˖
H31$זוך31$'SBNFXPSLפך《穈 ˖ 0CTFSWBCJMJUZ%JTUSJCVUFE5SBDJOHפך《穈 ˖ $IBPT&OHJOFFSJOHפך《穈 ˖ 婍גְٖؖء٦橆㞮ךٌتٝ⻉ ˖ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
8FBSFIJSJOH ˖ IUUQTDPPLQBEKPCT ˖ IUUQTDPPLQBEXPSLBCMFDPN 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ