SRE-Lounge-8-Cookpad-Microservice-Architecture-Overview

Slide 1

Slide 1 text

$PPLQBE.JDSPTFSWJDF "SDIJUFDUVSF0WFSWJFX ؙحؙػحس吳䒭⠓爡䪮遭鿇43&ؚٕ٦ف し䊛畆㣕 !SSSFFFZZZ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 2

Slide 2 text

!SSSFFFZZZ !SSSFFFZZZ IUUQTSSSFFFZZZDPN :PTIJLBXB3ZPUB .F ˖ :PTIJLBXB3ZPUB !SSSFFFZZZ[reɪ] ˖ ؙحؙػحس吳䒭⠓爡 ։ ˖ 䪮遭鿇43&ؚٕ٦ف ˖ 莆㄂걄㚖 ˖ 0CTFSWBCJMJUZ 5JNFTFSJFTEBUBCBTF ˖ 馯㄂ ˖ QVZP FPS[FB MFBHVFPGMFHFOET HCG ˖ TFJZV SFTUBHF XVH 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 3

Slide 3 text

$PPLQBE43&T ˖ 秈せ +1 ˖ 㛇劤涸ח傈劤㕂ⰻד⹶⹡ ˖ 秈せ (MPCBM ˖ ؎ؘٔأ׮׃ֻכ傈劤㕂ⰻד⹶⹡ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 4

Slide 4 text

$PPLQBE +1(MPCBM ˖ +BQBO ˖ IUUQTDPPLQBEDPN ˖ (MPCBM $PVOUSJFT-BOHVBHFT ˖ IUUQTDPPLQBEDPNVT ˖ IUUQTDPPLQBEDPNVL ˖ IUUQTDPPLQBEDPNJE ˖ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 5

Slide 5 text

$PPLQBE +1(MPCBM ˖ +BQBO ˖ IUUQTDPPLQBEDPN ˖ (MPCBM ˖ ֿ׍׵ך《׶穈׫כGPPUOPUFTך⚛ןח׾⿫撑׃גֻ׌ְׁ IUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSFOETFBTPO IUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 6

Slide 6 text

$PPLQBE43&Tך顑⟣眔㔲 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 7

Slide 7 text

؎ً٦آ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 8

Slide 8 text

43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 9

Slide 9 text

植㹋 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 10

Slide 10 text

43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 11

Slide 11 text

.JDSPTFSWJDFTBOE$PPLQBE 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 12

Slide 12 text

.JDSPTFSWJDFTBOE$PPLQBE ˖ 䎃׀׹ַ׵و؎ؙٗ؟٦ؽأ⻉ח《׶穈׿דֹ׋ ˖ ؝ىُص؛٦ءّٝ٥顑⟣眔㔲٥Ꟛ涪⸬桦זו㼎ׅ׷㉏겗䠐陎 ˖ 㣐ֹז⽃♧ך؝٦سك٦أ٥؟٦ؽأ㔿剣ך㉏겗ָ׋ֻׁ׿֮׷ ˖ و؎ؙٗ؟٦ؽأ⻉ָ鹌׬ֿהח״׷㣐ֹזًٔحزך❦「 ˖ Ꟛ涪鸞䏝ךぢ♳װ僇然ז顑⟣眔㔲٥ٔٔ٦أ鸞䏝ךぢ♳זו ˖ 㼭ְׁ؝٦سك٦أ٥؟٦ؽأז׵דכך兛鸐ךꟚ涪ָ遤ִ׷״ֲח 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 13

Slide 13 text

.JDSPTFSWJDFTBOE$PPLQBE ˖ ♧倯דծو؎ؙٗ؟٦ؽأ⻉׾鹌׭׷ֿהד饯ֹ׷㉏겗׮֮׷ ˖ 43&ָⰋגך؟٦ؽأ׾鋅׷הأ؛٦ٕ׃זְ ˖ ؟٦ؽأ꟦鸐⥋ך㉏겗٥鸬ꓲ涸זꥺ㹱٥〳錁庠䚍ך㉏겗 ˖ 43&ך⚺噟⹡ך♧אכְֲֿ׏׋㉏겗ח㼎ׅ׷،فٗ٦ث ˖ ֿ׸תדוך״ֲח׃ג،فٗ٦ث׃גֹ׋ַ׾稱➜ ˖ 4DBMBCJMJUZ "WBJMBCJMJUZך錁挿ַ׵ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 14

Slide 14 text

4DBMBCJMJUZ ˖ أ؛٦ٓؽٔذ؍ծה鎉׏ג׮圫ղז⽃⡘ָ֮׷ ˖ ➂꟦ ˖ ؟٦غ ˖ ؝أز 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 15

Slide 15 text

4DBMBCJMJUZ ˖ ➂꟦ ˖ إٕؿ؟٦ؽأ⻉׾鹌׭׷ ˖ ؟٦غ ˖ "VUP4DBMJOHך崞欽 ˖ ؝أز ˖ Ⱏ鸐㛇湍ך4QPU⻉ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 16

Slide 16 text

إٕؿ؟٦ؽأ⻉ ˖ ؙحؙػحسדכ"NB[PO&$4׾ⵃ欽׃ג؝ٝذشךⰟ鸐㛇湍׾⡲׏גְ׷ ˖ ،فٔ؛٦ءّٝך㹀纏װرفٗ؎חכIBLP׾ⵃ欽׃גְ׷ ˖ 㛇劤涸ח؟٦ؽأךꟚ涪罏ָ㹀纏׾剅ֹرفٗ؎דֹ׷״ֲחז׏גְ׷ ˖ 㹀纏ؿ؋؎ٕך剅䒭חכ+TPOOFU׾ⵃ欽׃גְ׷ ˖ "84ך؟٦ؽأ׮DPEFOJ[FUPPMTװ5FSSBGPSN׾⢪׏ג؝٦س⻉׃גְ׷ ˖ Ꟛ涪罏כ䗳銲ד֮׸ל荈歋ח13׾⳿ֿׅהָדֹ׷״ֲחז׏גְ׷ IUUQTDPEFOJ[FUPPMT IUUQTHJUIVCDPNFBHMFUNUIBLP 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 17

Slide 17 text

؝ٝذشⰟ鸐㛇湍 ˖ &$4؎ٝأةٝأכ4QPU'MFFU"VUP4DBMJOH(SPVQד盖椚ׁ׸גְ׷ ˖ &$4؎ٝأةٝأךקה׿וכ4QPUד⹛ְגְ׷ ˖ ؝أز꬗٥鎘皾项彁꬗ד㣐ֹזًٔحزָ欰ת׸גְ׷ ˖ ؎ٝأةٝأךٔا٦أ٥&$4؟٦ؽأך朐䡾זו׾然钠׃גأ؛٦ٕ،ؐز׃גְ׷ ˖ ⰻ醡ד؟٦ؽأך朐䡾׾然钠דֹ׷8FC؝ٝا٦ٕ׾⡲׏גְ׷ ˖ ؟٦ؽأ荈⡤ךًزؙٔأװٗ٦سغٓٝ؟ךؙٔؒأز٥ٖأهٝأךًزؙٔأזו ˖ ،فٔ؛٦ءّٝךؚٗךꠘ鋮٥嗚稊זו IUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF IUUQTTQFBLFSEFDLDPNFBHMFUNUCVJMEJOHBTUFBEZFDTJOGSBTUSVDUVSF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 18

Slide 18 text

43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 19

Slide 19 text

43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 20

Slide 20 text

{ scheduler: { type: 'ecs', cluster: 'eagletmt', desired_count: 2, role: 'ecsServiceRole', elb_v2: { vpc_id: 'vpc-WWWWWWWW', health_check_path: '/site/sha', listeners: [ { port: 443, protocol: 'HTTPS', certificate_arn: 'arn:aws:iam::012345678901:server-certificate/hello-lb-v2.example.com', }, ], subnets: ['subnet-XXXXXXXX', 'subnet-YYYYYYYY'], security_groups: ['sg-ZZZZZZZZ'], : }, }, app: { image: 'ryotarai/hello-sinatra', : secrets: [{ name: 'MESSAGE', value_from: 'arn:aws:ssm:ap-northeast-1:012345678901:parameter/hako/hello-lb-v2/secret-message', }], }, sidecars: { front: { image_tag: 'hako-nginx', : }, }, : } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 21

Slide 21 text

43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 22

Slide 22 text

43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 23

Slide 23 text

"WBJMBCJMJUZ ˖ ؟٦ؽأ꟦ך鸐⥋חֶֽ׷醱꧟䚍׾〳腉זꣲ׶㼰זֻׅ׷ ˖ 4FSWJDFNFTIך㼪Ⰵ ˖ 黝ⴖזٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ ˖ ꥺ㹱ָ饯ֹ׋הֹח.553ָ〳腉זꣲ׶瀉ְ״ֲחׅ׷ ˖ .POJUPSJOH 0CTFSWBCJMJUZך然⥂זו ˖ $IBPT&OHJOFFSJOHפך《׶穈׫ ˖ "-JGFPG$IBPT&OHJOFFSJOH4UBSUJOHXJUI3FTJMJFODF IUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 24

Slide 24 text

4FSWJDFNFTIך《׶穈׫ ˖ ؟٦ؽأ꟦鸐⥋ךקרⰋגד&OWPZָ欽ְ׵׸גְ׷ ˖ ،فٔ؛٦ءّٝכ➭ך؟٦ؽأח鸐⥋ׅ׷儗&OWPZ׾鸐ׅ״ֲחׅ׷ ˖ 黝ⴖחٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ָ涪⹛ׅ׷✲דِ٦ؠ䕦갟ך֮׷♧儗涸זؒٓ٦ָ慧幾 ˖ ؟٦ؽأ꟦鸐⥋ך➭ח鸐䌢ך5$11SPYZה׃ג׮⢪׏גְ׷ ˖ &OWPZך$POUSPM1MBOFך؝ٝه٦طٝزה׃גDPPLQBEJUBDIP DPPLQBETET׾欽ְגְ׷ ˖ ٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ך鏣㹀זו׾♧⯋ד盖椚׃גְ׷ ˖ 鏣㹀ך鎸岀חכ+TPOOFU׾ⵃ欽׃גְ׷ IUUQTHJUIVCDPNDPPLQBEJUBDIP IUUQTHJUIVCDPNDPPLQBETET IUUQTUFDIMJGFDPPLQBEDPNFOUSZ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 25

Slide 25 text

43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 26

Slide 26 text

.POJUPSJOH 0CTFSWBCJMJUZפך《׶穈׫ ˖ 䖞勻ך؎ٝأةٝأך湊鋔כ׆׏ה֮׷;BCCJY׮⢪׻׸גְ׷ ˖ 㢩䕎湊鋔ה׃גך4UBUVT$BLFװ鎉铂׀הך鑫稢ז"1.ה׃גך/FX3FMJDזו׮֮׷ ˖ ⟃♴ך״ֲזًزؙٔأכ1SPNFUIFVTח꧊׭ג鑫稢ח鋅׸׷״ֲח׃גְ׷ ˖ ؎ٝأةٝأ٥ىسٕؐؑ،ך㛇劤涸זًزؙٔأ node_exporter ぐىسٕؐؑ،׀הךFYQPSUFS ˖ ؝ٝذش橆㞮ך㛇劤涸זًزؙٔأ cAdvisor ٥&OWPZַ׵《׸׷؟٦ؽأ꟦鸐⥋ךًزؙٔأ ˖ 1SPNFUIFVTח꧊׭׋ًزؙٔأַ׵،ٓ٦زׅ׷➬穈׫ ˖ +TPOOFU׾欽ְג3VMFך㹀纏"MFSUNBOBHFSך鏣㹀׾剅ֽ׷״ֲח׃גְ׷ ˖ "8493BZװ7J[DFSBMזו׾欽ְ׋%JTUSJCVUFE5SBDJOH 7JTVBMJ[BUJPOך➬穈׫ IUUQTHJUIVCDPNSSSFFFZZZEPDLFSQSPNFUIFVTNJYJOUSFFNBTUFSFYBNQMF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 27

Slide 27 text

43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 28

Slide 28 text

{ prometheusAlerts+:: { groups+: [ { name: 'example-alerts', rules: [ { alert: 'ExampleCriticalAlertPrometheusDownFor1min', // PromQL expr: ||| up{job="prometheus"} == 0 ||| % $._config, 'for': '1m', labels: { pager: 'sre', severity: 'critical', }, annotations: { title: '{{$labels.job}} is unreachable', description: '{{$labels.job}} at {{$labels.instance}} could not be scraped for over 1 minutes.', runbook: 'https://wiki.example.com/pages/runbook/ExampleCriticalAlertPrometheusDownFor1min' }, }, ], }, ], }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 29

Slide 29 text

{ alertmanagerConfig+:: { route+: { routes+: [{ match_re: { pager: '^(sre)$' }, receiver: 'sre-slack', routes: [{ match: { severity: 'critical' }, receiver: 'sre-pagerduty', }], }], }, }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 30

Slide 30 text

{ alertmanagerConfig+:: { receivers+: [ { name: 'sre-slack', slack_configs: [ { send_resolved: true, channel: '#sre', }, ], }, ], }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 31

Slide 31 text

{ alertmanagerConfig+:: { receivers+: [ { name: 'sre-pagerduty', pagerduty_configs: [ { service_key: '{{ template "secrets.pagerduty.sre.service_key" }}' }, ], }, ], }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 32

Slide 32 text

43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 33

Slide 33 text

43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 34

Slide 34 text

43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 35

Slide 35 text

ת׌ת׌דֹ׷ֿה٥װ׶׋ְֿהָ֮׷ ˖ 刿ז׷إٕؿ؟٦ؽأ⻉ ˖ ぐ؟٦ؽأפךث٦ي׾䋆ֹ鴥׿׌4-*4-0ך㼪Ⰵ ˖ 刿ז׷؟٦ؽأًحءُפך《׶穈׫ ˖ 刿ז׷؝ٝذشⰟ鸐㛇湍ך何㊣פך《׶穈׫ ˖ H31$זוך31$'SBNFXPSLפך《׶穈׫ ˖ 0CTFSWBCJMJUZ%JTUSJCVUFE5SBDJOHפך《׶穈׫ ˖ $IBPT&OHJOFFSJOHפך《׶穈׫ ˖ 婍׏גְ׷ٖؖء٦橆㞮ךٌتٝ⻉ ˖ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ

Slide 36

Slide 36 text

8FBSFIJSJOH ˖ IUUQTDPPLQBEKPCT ˖ IUUQTDPPLQBEXPSLBCMFDPN 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ