Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
SRE-Lounge-8-Cookpad-Microservice-Architecture-Overview
Search
rrreeeyyy
March 13, 2019
Technology
5
5k
SRE-Lounge-8-Cookpad-Microservice-Architecture-Overview
SRE Lounge #8 でお話させていただきました
rrreeeyyy
March 13, 2019
Tweet
Share
More Decks by rrreeeyyy
See All by rrreeeyyy
カンファレンスから見る SRE トレンド 2024 / SRE Trends from Conferences in 2024 #SRE_Findy
rrreeeyyy
3
1.7k
信頼性の育て方 / mackerel-meetup-15
rrreeeyyy
9
2k
SRE の歩き方・進め方 / sre-walk-through-procedure
rrreeeyyy
0
8.3k
「信頼性」を保ちつつ大規模サービスをリニューアルする / cookpad-tech-kitchen-service-embedded-sres
rrreeeyyy
11
11k
Cookpad and Prometheus
rrreeeyyy
6
20k
A survey of anomaly detection methodologies for web system
rrreeeyyy
5
1.1k
エンジニアリングをちゃんとやる あるいは 人類の平和 について / wsa02-rrreeeyyy
rrreeeyyy
13
2.8k
「自立」したWebシステムを創る。自分の好きなことをする世界を目指して。/ ipsj-one-2018-rrreeeyyy
rrreeeyyy
3
1.7k
Web サービスの信頼性と運用の自動化について / iot40-rrreeeyyy
rrreeeyyy
12
7.4k
Other Decks in Technology
See All in Technology
ServiceNow Knowledge 24の歩き方 EYストラテジー・アンド・コンサルティング
manarobot
0
200
Gitlab本から学んだこと - そーだいなるプレイバック / gitlab-book
soudai
4
440
自己改善からチームを動かす! 「セルフエンジニアリングマネージャー」のすゝめ
shoota
6
830
エンジニアのキャリアをちょっと楽しくする3本の軸/Three Pillars to Make an Engineer's Career More Enjoyable
kwappa
0
2.7k
IaCジェネレーターとBedrockで詳細設計書を生成してみた
tsukasa_ishimaru
3
310
今年のRubyKaigiはProfiler Year🤘
osyoyu
0
190
AWSに詳しくない人でも始められるコスト最適化ガイド
yuhta28
1
250
KubeConにproposalを送りたい人へのアドバイス
sat
PRO
3
260
エンジニア候補者向け資料2024.04.24.pdf
macloud
0
3.3k
Java EE/Jakarta EEの現状と将来―クラウドネイティブ時代にJava EEは対応できるのか?―
takakiyo
1
170
アクセシビリティを考慮したUI/CSSフレームワーク・ライブラリ選定
yajihum
2
1k
LLM開発・活用の舞台裏@2024.04.25
yushin_n
2
740
Featured
See All Featured
No one is an island. Learnings from fostering a developers community.
thoeni
16
2.1k
Done Done
chrislema
178
15k
Scaling GitHub
holman
457
140k
Fashionably flexible responsive web design (full day workshop)
malarkey
398
65k
The Cost Of JavaScript in 2023
addyosmani
16
3.9k
Unsuck your backbone
ammeep
663
57k
A Philosophy of Restraint
colly
197
16k
Adopting Sorbet at Scale
ufuk
68
8.6k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
14
1.5k
Six Lessons from altMBA
skipperchong
21
3k
Pencils Down: Stop Designing & Start Developing
hursman
117
11k
Designing with Data
zakiwarfel
96
4.8k
Transcript
$PPLQBE.JDSPTFSWJDF "SDIJUFDUVSF0WFSWJFX ؙحؙػحس吳䒭⠓爡䪮遭鿇43&ؚٕ٦ف し䊛畆㣕 !SSSFFFZZZ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
!SSSFFFZZZ !SSSFFFZZZ IUUQTSSSFFFZZZDPN :PTIJLBXB3ZPUB .F ˖ :PTIJLBXB3ZPUB !SSSFFFZZZ[reɪ] ˖ ؙحؙػحس吳䒭⠓爡
։ ˖ 䪮遭鿇43&ؚٕ٦ف ˖ 莆걄㚖 ˖ 0CTFSWBCJMJUZ 5JNFTFSJFTEBUBCBTF ˖ 馯 ˖ QVZP FPS[FB MFBHVFPGMFHFOET HCG ˖ TFJZV SFTUBHF XVH 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE43&T ˖ 秈せ +1 ˖ 㛇劤涸ח傈劤㕂ⰻד ˖ 秈せ (MPCBM ˖
؎ؘٔأ׃ֻכ傈劤㕂ⰻד 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE +1(MPCBM ˖ +BQBO ˖ IUUQTDPPLQBEDPN ˖ (MPCBM $PVOUSJFT-BOHVBHFT ˖
IUUQTDPPLQBEDPNVT ˖ IUUQTDPPLQBEDPNVL ˖ IUUQTDPPLQBEDPNJE ˖ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE +1(MPCBM ˖ +BQBO ˖ IUUQTDPPLQBEDPN ˖ (MPCBM ˖ ֿך《穈כGPPUOPUFTך⚛ןח撑׃גְֻׁ
IUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSFOETFBTPO IUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE43&Tך顑⟣眔㔲 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
؎ً٦آ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
植㹋 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.JDSPTFSWJDFTBOE$PPLQBE 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.JDSPTFSWJDFTBOE$PPLQBE ˖ 䎃׀ַو؎ؙٗ؟٦ؽأ⻉ח《穈דֹ ˖ ؝ىُص؛٦ءّٝ٥顑⟣眔㔲٥涪⸬桦זו㼎ׅ㉏겗䠐陎 ˖ 㣐ֹז⽃♧ך؝٦سك٦أ٥؟٦ؽأ㔿剣ך㉏겗ָֻׁ֮ ˖ و؎ؙٗ؟٦ؽأ⻉ָ鹌ֿהח״㣐ֹזًٔحزך❦「
˖ 涪鸞䏝ךぢ♳װ僇然ז顑⟣眔㔲٥ٔٔ٦أ鸞䏝ךぢ♳זו ˖ 㼭ְׁ؝٦سك٦أ٥؟٦ؽأזדכך兛鸐ך涪ָ遤ִ״ֲח 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.JDSPTFSWJDFTBOE$PPLQBE ˖ ♧倯דծو؎ؙٗ؟٦ؽأ⻉鹌ֿהד饯ֹ㉏겗֮ ˖ 43&ָⰋגך؟٦ؽأ鋅הأ؛٦ٕ׃זְ ˖ ؟٦ؽأ鸐⥋ך㉏겗٥鸬ꓲ涸זꥺ㹱٥〳錁庠䚍ך㉏겗 ˖ 43&ך⚺噟ך♧אכְֲֿ㉏겗ח㼎ׅ،فٗ٦ث
˖ ֿתדוך״ֲח׃ג،فٗ٦ث׃גַֹ稱➜ ˖ 4DBMBCJMJUZ "WBJMBCJMJUZך錁挿ַ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
4DBMBCJMJUZ ˖ أ؛٦ٓؽٔذ؍ծה鎉ג圫ղז⽃⡘ָ֮ ˖ ➂ ˖ ؟٦غ ˖ ؝أز 43&-PVOHF
]:PTIJLBXB3ZPUB !SSSFFFZZZ
4DBMBCJMJUZ ˖ ➂ ˖ إٕؿ؟٦ؽأ⻉鹌 ˖ ؟٦غ ˖ "VUP4DBMJOHך崞欽 ˖
؝أز ˖ Ⱏ鸐㛇湍ך4QPU⻉ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
إٕؿ؟٦ؽأ⻉ ˖ ؙحؙػحسדכ"NB[PO&$4ⵃ欽׃ג؝ٝذشךⰟ鸐㛇湍⡲גְ ˖ ،فٔ؛٦ءّٝך㹀纏װرفٗ؎חכIBLPⵃ欽׃גְ ˖ 㛇劤涸ח؟٦ؽأך涪罏ָ㹀纏剅ֹرفٗ؎דֹ״ֲחזגְ ˖ 㹀纏ؿ؋؎ٕך剅䒭חכ+TPOOFUⵃ欽׃גְ ˖
"84ך؟٦ؽأDPEFOJ[FUPPMTװ5FSSBGPSN⢪ג؝٦س⻉׃גְ ˖ 涪罏כ䗳銲ד֮ל荈歋ח13⳿ֿׅהָדֹ״ֲחזגְ IUUQTDPEFOJ[FUPPMT IUUQTHJUIVCDPNFBHMFUNUIBLP 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
؝ٝذشⰟ鸐㛇湍 ˖ &$4؎ٝأةٝأכ4QPU'MFFU"VUP4DBMJOH(SPVQד盖椚ׁגְ ˖ &$4؎ٝأةٝأךקהוכ4QPUד⹛ְגְ ˖ ؝أز٥鎘皾项彁ד㣐ֹזًٔحزָ欰תגְ ˖ ؎ٝأةٝأךٔا٦أ٥&$4؟٦ؽأך朐䡾זו然钠׃גأ؛٦ٕ،ؐز׃גְ ˖
ⰻ醡ד؟٦ؽأך朐䡾然钠דֹ8FC؝ٝا٦ٕ⡲גְ ˖ ؟٦ؽأ荈⡤ךًزؙٔأװٗ٦سغٓٝ؟ךؙٔؒأز٥ٖأهٝأךًزؙٔأזו ˖ ،فٔ؛٦ءّٝךؚٗךꠘ鋮٥嗚稊זו IUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF IUUQTTQFBLFSEFDLDPNFBHMFUNUCVJMEJOHBTUFBEZFDTJOGSBTUSVDUVSF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ scheduler: { type: 'ecs', cluster: 'eagletmt', desired_count: 2, role:
'ecsServiceRole', elb_v2: { vpc_id: 'vpc-WWWWWWWW', health_check_path: '/site/sha', listeners: [ { port: 443, protocol: 'HTTPS', certificate_arn: 'arn:aws:iam::012345678901:server-certificate/hello-lb-v2.example.com', }, ], subnets: ['subnet-XXXXXXXX', 'subnet-YYYYYYYY'], security_groups: ['sg-ZZZZZZZZ'], : }, }, app: { image: 'ryotarai/hello-sinatra', : secrets: [{ name: 'MESSAGE', value_from: 'arn:aws:ssm:ap-northeast-1:012345678901:parameter/hako/hello-lb-v2/secret-message', }], }, sidecars: { front: { image_tag: 'hako-nginx', : }, }, : } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
"WBJMBCJMJUZ ˖ ؟٦ؽأך鸐⥋חֶֽ醱꧟䚍〳腉זꣲ㼰זֻׅ ˖ 4FSWJDFNFTIך㼪Ⰵ ˖ 黝ⴖזٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ ˖ ꥺ㹱ָ饯ֹהֹח.553ָ〳腉זꣲ瀉ְ״ֲחׅ ˖
.POJUPSJOH 0CTFSWBCJMJUZך然⥂זו ˖ $IBPT&OHJOFFSJOHפך《穈 ˖ "-JGFPG$IBPT&OHJOFFSJOH4UBSUJOHXJUI3FTJMJFODF IUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
4FSWJDFNFTIך《穈 ˖ ؟٦ؽأ鸐⥋ךקרⰋגד&OWPZָ欽ְגְ ˖ ،فٔ؛٦ءّٝכ➭ך؟٦ؽأח鸐⥋ׅ儗&OWPZ鸐ׅ״ֲחׅ ˖ 黝ⴖחٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ָ涪⹛ׅ✲דِ٦ؠ䕦갟ך֮♧儗涸זؒٓ٦ָ慧幾 ˖ ؟٦ؽأ鸐⥋ך➭ח鸐䌢ך5$11SPYZה׃ג⢪גְ ˖
&OWPZך$POUSPM1MBOFך؝ٝه٦طٝزה׃גDPPLQBEJUBDIP DPPLQBETET欽ְגְ ˖ ٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ך鏣㹀זו♧⯋ד盖椚׃גְ ˖ 鏣㹀ך鎸岀חכ+TPOOFUⵃ欽׃גְ IUUQTHJUIVCDPNDPPLQBEJUBDIP IUUQTHJUIVCDPNDPPLQBETET IUUQTUFDIMJGFDPPLQBEDPNFOUSZ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.POJUPSJOH 0CTFSWBCJMJUZפך《穈 ˖ 䖞勻ך؎ٝأةٝأך湊鋔כ׆ה֮;BCCJY⢪גְ ˖ 㢩䕎湊鋔ה׃גך4UBUVT$BLFװ鎉铂׀הך鑫稢ז"1.ה׃גך/FX3FMJDזו֮ ˖ ⟃♴ך״ֲזًزؙٔأכ1SPNFUIFVTח꧊ג鑫稢ח鋅״ֲח׃גְ ˖ ؎ٝأةٝأ٥ىسٕؐؑ،ך㛇劤涸זًزؙٔأ
node_exporter ぐىسٕؐؑ،׀הךFYQPSUFS ˖ ؝ٝذش橆㞮ך㛇劤涸זًزؙٔأ cAdvisor ٥&OWPZַ《؟٦ؽأ鸐⥋ךًزؙٔأ ˖ 1SPNFUIFVTח꧊ًزؙٔأַ،ٓ٦زׅ➬穈 ˖ +TPOOFU欽ְג3VMFך㹀纏"MFSUNBOBHFSך鏣㹀剅ֽ״ֲח׃גְ ˖ "8493BZװ7J[DFSBMזו欽ְ%JTUSJCVUFE5SBDJOH 7JTVBMJ[BUJPOך➬穈 IUUQTHJUIVCDPNSSSFFFZZZEPDLFSQSPNFUIFVTNJYJOUSFFNBTUFSFYBNQMF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ prometheusAlerts+:: { groups+: [ { name: 'example-alerts', rules: [
{ alert: 'ExampleCriticalAlertPrometheusDownFor1min', // PromQL expr: ||| up{job="prometheus"} == 0 ||| % $._config, 'for': '1m', labels: { pager: 'sre', severity: 'critical', }, annotations: { title: '{{$labels.job}} is unreachable', description: '{{$labels.job}} at {{$labels.instance}} could not be scraped for over 1 minutes.', runbook: 'https://wiki.example.com/pages/runbook/ExampleCriticalAlertPrometheusDownFor1min' }, }, ], }, ], }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ alertmanagerConfig+:: { route+: { routes+: [{ match_re: { pager:
'^(sre)$' }, receiver: 'sre-slack', routes: [{ match: { severity: 'critical' }, receiver: 'sre-pagerduty', }], }], }, }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ alertmanagerConfig+:: { receivers+: [ { name: 'sre-slack', slack_configs: [
{ send_resolved: true, channel: '#sre', }, ], }, ], }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ alertmanagerConfig+:: { receivers+: [ { name: 'sre-pagerduty', pagerduty_configs: [
{ service_key: '{{ template "secrets.pagerduty.sre.service_key" }}' }, ], }, ], }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
תתדֹֿה٥װְֿהָ֮ ˖ 刿זإٕؿ؟٦ؽأ⻉ ˖ ぐ؟٦ؽأפךث٦ي䋆ֹ鴥4-*4-0ך㼪Ⰵ ˖ 刿ז؟٦ؽأًحءُפך《穈 ˖ 刿ז؝ٝذشⰟ鸐㛇湍ך何㊣פך《穈 ˖
H31$זוך31$'SBNFXPSLפך《穈 ˖ 0CTFSWBCJMJUZ%JTUSJCVUFE5SBDJOHפך《穈 ˖ $IBPT&OHJOFFSJOHפך《穈 ˖ 婍גְٖؖء٦橆㞮ךٌتٝ⻉ ˖ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
8FBSFIJSJOH ˖ IUUQTDPPLQBEKPCT ˖ IUUQTDPPLQBEXPSLBCMFDPN 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ