Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
SRE-Lounge-8-Cookpad-Microservice-Architecture-...
Search
rrreeeyyy
March 13, 2019
Technology
5
5.5k
SRE-Lounge-8-Cookpad-Microservice-Architecture-Overview
SRE Lounge #8 でお話させていただきました
rrreeeyyy
March 13, 2019
Tweet
Share
More Decks by rrreeeyyy
See All by rrreeeyyy
Rethinking Incident Response: Context-Aware AI in Practice
rrreeeyyy
1
390
Incident Response Practices: Waroom's Features and Future Challenges
rrreeeyyy
0
230
An Efficient Incident Response Training with AI / SRE NEXT 2024 Sponsor Session
rrreeeyyy
1
5.2k
カンファレンスから見る SRE トレンド 2024 / SRE Trends from Conferences in 2024 #SRE_Findy
rrreeeyyy
4
2.4k
信頼性の育て方 / mackerel-meetup-15
rrreeeyyy
10
2.6k
SRE の歩き方・進め方 / sre-walk-through-procedure
rrreeeyyy
0
8.8k
「信頼性」を保ちつつ大規模サービスをリニューアルする / cookpad-tech-kitchen-service-embedded-sres
rrreeeyyy
11
12k
Cookpad and Prometheus
rrreeeyyy
6
21k
A survey of anomaly detection methodologies for web system
rrreeeyyy
5
1.3k
Other Decks in Technology
See All in Technology
United™️ Airlines®️ Customer®️ USA Contact Numbers: Complete 2025 Support Guide
flyunitedguide
0
780
20250708オープンエンドな探索と知識発見
sakana_ai
PRO
4
860
全部AI、全員Cursor、ドキュメント駆動開発 〜DevinやGeminiも添えて〜
rinchsan
2
2.2k
american airlines®️ USA Contact Numbers: Complete 2025 Support Guide
supportflight
1
120
オーティファイ会社紹介資料 / Autify Company Deck
autifyhq
10
130k
ClaudeCodeにキレない技術
gtnao
0
560
事例で学ぶ!B2B SaaSにおけるSREの実践例/SRE for B2B SaaS: A Real-World Case Study
bitkey
1
340
【あのMCPって、どんな処理してるの?】 AWS CDKでの開発で便利なAWS MCP Servers特集
yoshimi0227
6
730
TLSから見るSREの未来
atpons
2
240
敢えて生成AIを使わないマネジメント業務
kzkmaeda
2
510
[SRE NEXT] ARR150億円_エンジニア140名_27チーム_17プロダクトから始めるSLO.pdf
satos
5
2.1k
united airlines ™®️ USA Contact Numbers: Complete 2025 Support Guide
flyunitedhelp
1
470
Featured
See All Featured
The Art of Programming - Codeland 2020
erikaheidi
54
13k
Statistics for Hackers
jakevdp
799
220k
KATA
mclloyd
30
14k
The World Runs on Bad Software
bkeepers
PRO
69
11k
Visualization
eitanlees
146
16k
Code Review Best Practice
trishagee
69
19k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.4k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2.1k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
Balancing Empowerment & Direction
lara
1
440
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
The Language of Interfaces
destraynor
158
25k
Transcript
$PPLQBE.JDSPTFSWJDF "SDIJUFDUVSF0WFSWJFX ؙحؙػحس吳䒭⠓爡䪮遭鿇43&ؚٕ٦ف し䊛畆㣕 !SSSFFFZZZ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
!SSSFFFZZZ !SSSFFFZZZ IUUQTSSSFFFZZZDPN :PTIJLBXB3ZPUB .F ˖ :PTIJLBXB3ZPUB !SSSFFFZZZ[reɪ] ˖ ؙحؙػحس吳䒭⠓爡
։ ˖ 䪮遭鿇43&ؚٕ٦ف ˖ 莆걄㚖 ˖ 0CTFSWBCJMJUZ 5JNFTFSJFTEBUBCBTF ˖ 馯 ˖ QVZP FPS[FB MFBHVFPGMFHFOET HCG ˖ TFJZV SFTUBHF XVH 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE43&T ˖ 秈せ +1 ˖ 㛇劤涸ח傈劤㕂ⰻד ˖ 秈せ (MPCBM ˖
؎ؘٔأ׃ֻכ傈劤㕂ⰻד 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE +1(MPCBM ˖ +BQBO ˖ IUUQTDPPLQBEDPN ˖ (MPCBM $PVOUSJFT-BOHVBHFT ˖
IUUQTDPPLQBEDPNVT ˖ IUUQTDPPLQBEDPNVL ˖ IUUQTDPPLQBEDPNJE ˖ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE +1(MPCBM ˖ +BQBO ˖ IUUQTDPPLQBEDPN ˖ (MPCBM ˖ ֿך《穈כGPPUOPUFTך⚛ןח撑׃גְֻׁ
IUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSFOETFBTPO IUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE43&Tך顑⟣眔㔲 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
؎ً٦آ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
植㹋 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.JDSPTFSWJDFTBOE$PPLQBE 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.JDSPTFSWJDFTBOE$PPLQBE ˖ 䎃׀ַو؎ؙٗ؟٦ؽأ⻉ח《穈דֹ ˖ ؝ىُص؛٦ءّٝ٥顑⟣眔㔲٥涪⸬桦זו㼎ׅ㉏겗䠐陎 ˖ 㣐ֹז⽃♧ך؝٦سك٦أ٥؟٦ؽأ㔿剣ך㉏겗ָֻׁ֮ ˖ و؎ؙٗ؟٦ؽأ⻉ָ鹌ֿהח״㣐ֹזًٔحزך❦「
˖ 涪鸞䏝ךぢ♳װ僇然ז顑⟣眔㔲٥ٔٔ٦أ鸞䏝ךぢ♳זו ˖ 㼭ְׁ؝٦سك٦أ٥؟٦ؽأזדכך兛鸐ך涪ָ遤ִ״ֲח 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.JDSPTFSWJDFTBOE$PPLQBE ˖ ♧倯דծو؎ؙٗ؟٦ؽأ⻉鹌ֿהד饯ֹ㉏겗֮ ˖ 43&ָⰋגך؟٦ؽأ鋅הأ؛٦ٕ׃זְ ˖ ؟٦ؽأ鸐⥋ך㉏겗٥鸬ꓲ涸זꥺ㹱٥〳錁庠䚍ך㉏겗 ˖ 43&ך⚺噟ך♧אכְֲֿ㉏겗ח㼎ׅ،فٗ٦ث
˖ ֿתדוך״ֲח׃ג،فٗ٦ث׃גַֹ稱➜ ˖ 4DBMBCJMJUZ "WBJMBCJMJUZך錁挿ַ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
4DBMBCJMJUZ ˖ أ؛٦ٓؽٔذ؍ծה鎉ג圫ղז⽃⡘ָ֮ ˖ ➂ ˖ ؟٦غ ˖ ؝أز 43&-PVOHF
]:PTIJLBXB3ZPUB !SSSFFFZZZ
4DBMBCJMJUZ ˖ ➂ ˖ إٕؿ؟٦ؽأ⻉鹌 ˖ ؟٦غ ˖ "VUP4DBMJOHך崞欽 ˖
؝أز ˖ Ⱏ鸐㛇湍ך4QPU⻉ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
إٕؿ؟٦ؽأ⻉ ˖ ؙحؙػحسדכ"NB[PO&$4ⵃ欽׃ג؝ٝذشךⰟ鸐㛇湍⡲גְ ˖ ،فٔ؛٦ءّٝך㹀纏װرفٗ؎חכIBLPⵃ欽׃גְ ˖ 㛇劤涸ח؟٦ؽأך涪罏ָ㹀纏剅ֹرفٗ؎דֹ״ֲחזגְ ˖ 㹀纏ؿ؋؎ٕך剅䒭חכ+TPOOFUⵃ欽׃גְ ˖
"84ך؟٦ؽأDPEFOJ[FUPPMTװ5FSSBGPSN⢪ג؝٦س⻉׃גְ ˖ 涪罏כ䗳銲ד֮ל荈歋ח13⳿ֿׅהָדֹ״ֲחזגְ IUUQTDPEFOJ[FUPPMT IUUQTHJUIVCDPNFBHMFUNUIBLP 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
؝ٝذشⰟ鸐㛇湍 ˖ &$4؎ٝأةٝأכ4QPU'MFFU"VUP4DBMJOH(SPVQד盖椚ׁגְ ˖ &$4؎ٝأةٝأךקהוכ4QPUד⹛ְגְ ˖ ؝أز٥鎘皾项彁ד㣐ֹזًٔحزָ欰תגְ ˖ ؎ٝأةٝأךٔا٦أ٥&$4؟٦ؽأך朐䡾זו然钠׃גأ؛٦ٕ،ؐز׃גְ ˖
ⰻ醡ד؟٦ؽأך朐䡾然钠דֹ8FC؝ٝا٦ٕ⡲גְ ˖ ؟٦ؽأ荈⡤ךًزؙٔأװٗ٦سغٓٝ؟ךؙٔؒأز٥ٖأهٝأךًزؙٔأזו ˖ ،فٔ؛٦ءّٝךؚٗךꠘ鋮٥嗚稊זו IUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF IUUQTTQFBLFSEFDLDPNFBHMFUNUCVJMEJOHBTUFBEZFDTJOGSBTUSVDUVSF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ scheduler: { type: 'ecs', cluster: 'eagletmt', desired_count: 2, role:
'ecsServiceRole', elb_v2: { vpc_id: 'vpc-WWWWWWWW', health_check_path: '/site/sha', listeners: [ { port: 443, protocol: 'HTTPS', certificate_arn: 'arn:aws:iam::012345678901:server-certificate/hello-lb-v2.example.com', }, ], subnets: ['subnet-XXXXXXXX', 'subnet-YYYYYYYY'], security_groups: ['sg-ZZZZZZZZ'], : }, }, app: { image: 'ryotarai/hello-sinatra', : secrets: [{ name: 'MESSAGE', value_from: 'arn:aws:ssm:ap-northeast-1:012345678901:parameter/hako/hello-lb-v2/secret-message', }], }, sidecars: { front: { image_tag: 'hako-nginx', : }, }, : } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
"WBJMBCJMJUZ ˖ ؟٦ؽأך鸐⥋חֶֽ醱꧟䚍〳腉זꣲ㼰זֻׅ ˖ 4FSWJDFNFTIך㼪Ⰵ ˖ 黝ⴖזٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ ˖ ꥺ㹱ָ饯ֹהֹח.553ָ〳腉זꣲ瀉ְ״ֲחׅ ˖
.POJUPSJOH 0CTFSWBCJMJUZך然⥂זו ˖ $IBPT&OHJOFFSJOHפך《穈 ˖ "-JGFPG$IBPT&OHJOFFSJOH4UBSUJOHXJUI3FTJMJFODF IUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
4FSWJDFNFTIך《穈 ˖ ؟٦ؽأ鸐⥋ךקרⰋגד&OWPZָ欽ְגְ ˖ ،فٔ؛٦ءّٝכ➭ך؟٦ؽأח鸐⥋ׅ儗&OWPZ鸐ׅ״ֲחׅ ˖ 黝ⴖחٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ָ涪⹛ׅ✲דِ٦ؠ䕦갟ך֮♧儗涸זؒٓ٦ָ慧幾 ˖ ؟٦ؽأ鸐⥋ך➭ח鸐䌢ך5$11SPYZה׃ג⢪גְ ˖
&OWPZך$POUSPM1MBOFך؝ٝه٦طٝزה׃גDPPLQBEJUBDIP DPPLQBETET欽ְגְ ˖ ٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ך鏣㹀זו♧⯋ד盖椚׃גְ ˖ 鏣㹀ך鎸岀חכ+TPOOFUⵃ欽׃גְ IUUQTHJUIVCDPNDPPLQBEJUBDIP IUUQTHJUIVCDPNDPPLQBETET IUUQTUFDIMJGFDPPLQBEDPNFOUSZ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.POJUPSJOH 0CTFSWBCJMJUZפך《穈 ˖ 䖞勻ך؎ٝأةٝأך湊鋔כ׆ה֮;BCCJY⢪גְ ˖ 㢩䕎湊鋔ה׃גך4UBUVT$BLFװ鎉铂׀הך鑫稢ז"1.ה׃גך/FX3FMJDזו֮ ˖ ⟃♴ך״ֲזًزؙٔأכ1SPNFUIFVTח꧊ג鑫稢ח鋅״ֲח׃גְ ˖ ؎ٝأةٝأ٥ىسٕؐؑ،ך㛇劤涸זًزؙٔأ
node_exporter ぐىسٕؐؑ،׀הךFYQPSUFS ˖ ؝ٝذش橆㞮ך㛇劤涸זًزؙٔأ cAdvisor ٥&OWPZַ《؟٦ؽأ鸐⥋ךًزؙٔأ ˖ 1SPNFUIFVTח꧊ًزؙٔأַ،ٓ٦زׅ➬穈 ˖ +TPOOFU欽ְג3VMFך㹀纏"MFSUNBOBHFSך鏣㹀剅ֽ״ֲח׃גְ ˖ "8493BZװ7J[DFSBMזו欽ְ%JTUSJCVUFE5SBDJOH 7JTVBMJ[BUJPOך➬穈 IUUQTHJUIVCDPNSSSFFFZZZEPDLFSQSPNFUIFVTNJYJOUSFFNBTUFSFYBNQMF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ prometheusAlerts+:: { groups+: [ { name: 'example-alerts', rules: [
{ alert: 'ExampleCriticalAlertPrometheusDownFor1min', // PromQL expr: ||| up{job="prometheus"} == 0 ||| % $._config, 'for': '1m', labels: { pager: 'sre', severity: 'critical', }, annotations: { title: '{{$labels.job}} is unreachable', description: '{{$labels.job}} at {{$labels.instance}} could not be scraped for over 1 minutes.', runbook: 'https://wiki.example.com/pages/runbook/ExampleCriticalAlertPrometheusDownFor1min' }, }, ], }, ], }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ alertmanagerConfig+:: { route+: { routes+: [{ match_re: { pager:
'^(sre)$' }, receiver: 'sre-slack', routes: [{ match: { severity: 'critical' }, receiver: 'sre-pagerduty', }], }], }, }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ alertmanagerConfig+:: { receivers+: [ { name: 'sre-slack', slack_configs: [
{ send_resolved: true, channel: '#sre', }, ], }, ], }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ alertmanagerConfig+:: { receivers+: [ { name: 'sre-pagerduty', pagerduty_configs: [
{ service_key: '{{ template "secrets.pagerduty.sre.service_key" }}' }, ], }, ], }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
תתדֹֿה٥װְֿהָ֮ ˖ 刿זإٕؿ؟٦ؽأ⻉ ˖ ぐ؟٦ؽأפךث٦ي䋆ֹ鴥4-*4-0ך㼪Ⰵ ˖ 刿ז؟٦ؽأًحءُפך《穈 ˖ 刿ז؝ٝذشⰟ鸐㛇湍ך何㊣פך《穈 ˖
H31$זוך31$'SBNFXPSLפך《穈 ˖ 0CTFSWBCJMJUZ%JTUSJCVUFE5SBDJOHפך《穈 ˖ $IBPT&OHJOFFSJOHפך《穈 ˖ 婍גְٖؖء٦橆㞮ךٌتٝ⻉ ˖ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
8FBSFIJSJOH ˖ IUUQTDPPLQBEKPCT ˖ IUUQTDPPLQBEXPSLBCMFDPN 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ