SRE Lounge #8 でお話させていただきました
$PPLQBE.JDSPTFSWJDF"SDIJUFDUVSF0WFSWJFXؙحؙػحس吳䒭⠓爡䪮遭鿇43&ؚٕ٦فし䊛畆㣕 !SSSFFFZZZ43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
View Slide
!SSSFFFZZZ!SSSFFFZZZIUUQTSSSFFFZZZDPN:PTIJLBXB3ZPUB.F˖ :PTIJLBXB3ZPUB !SSSFFFZZZ[reɪ]˖ ؙحؙػحس吳䒭⠓爡 ։˖ 䪮遭鿇43&ؚٕ٦ف˖ 莆걄㚖˖ 0CTFSWBCJMJUZ 5JNFTFSJFTEBUBCBTF˖ 馯˖ QVZP FPS[FB MFBHVFPGMFHFOET HCG˖ TFJZV SFTUBHF XVH43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE43&T˖ 秈せ +1˖ 㛇劤涸ח傈劤㕂ⰻד˖ 秈せ (MPCBM˖ ؎ؘٔأ׃ֻכ傈劤㕂ⰻד43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE +1(MPCBM˖ +BQBO˖ IUUQTDPPLQBEDPN˖ (MPCBM $PVOUSJFT-BOHVBHFT˖ IUUQTDPPLQBEDPNVT˖ IUUQTDPPLQBEDPNVL˖ IUUQTDPPLQBEDPNJE˖ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE +1(MPCBM˖ +BQBO˖ IUUQTDPPLQBEDPN˖ (MPCBM˖ ֿך《穈כGPPUOPUFTך⚛ןח撑׃גְֻׁIUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSFOETFBTPOIUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSF43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE43&Tך顑⟣眔㔲43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
؎ً٦آ43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
植㹋43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.JDSPTFSWJDFTBOE$PPLQBE43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.JDSPTFSWJDFTBOE$PPLQBE ˖ 䎃׀ַو؎ؙٗ؟٦ؽأ⻉ח《穈דֹ˖ ؝ىُص؛٦ءّٝ٥顑⟣眔㔲٥涪⸬桦זו㼎ׅ㉏겗䠐陎˖ 㣐ֹז⽃♧ך؝٦سك٦أ٥؟٦ؽأ㔿剣ך㉏겗ָֻׁ֮˖ و؎ؙٗ؟٦ؽأ⻉ָ鹌ֿהח״㣐ֹזًٔحزך❦「˖ 涪鸞䏝ךぢ♳װ僇然ז顑⟣眔㔲٥ٔٔ٦أ鸞䏝ךぢ♳זו˖ 㼭ְׁ؝٦سك٦أ٥؟٦ؽأזדכך兛鸐ך涪ָ遤ִ״ֲח43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.JDSPTFSWJDFTBOE$PPLQBE ˖ ♧倯דծو؎ؙٗ؟٦ؽأ⻉鹌ֿהד饯ֹ㉏겗֮˖ 43&ָⰋגך؟٦ؽأ鋅הأ؛٦ٕ׃זְ˖ ؟٦ؽأ鸐⥋ך㉏겗٥鸬ꓲ涸זꥺ㹱٥〳錁庠䚍ך㉏겗˖ 43&ך⚺噟ך♧אכְֲֿ㉏겗ח㼎ׅ،فٗ٦ث˖ ֿתדוך״ֲח׃ג،فٗ٦ث׃גַֹ稱➜˖ 4DBMBCJMJUZ "WBJMBCJMJUZך錁挿ַ43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
4DBMBCJMJUZ˖ أ؛٦ٓؽٔذ؍ծה鎉ג圫ղז⽃⡘ָ֮˖ ➂˖ ؟٦غ˖ ؝أز43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
4DBMBCJMJUZ˖ ➂˖ إٕؿ؟٦ؽأ⻉鹌˖ ؟٦غ˖ "VUP4DBMJOHך崞欽˖ ؝أز˖ Ⱏ鸐㛇湍ך4QPU⻉43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
إٕؿ؟٦ؽأ⻉˖ ؙحؙػحسדכ"NB[PO&$4ⵃ欽׃ג؝ٝذشךⰟ鸐㛇湍⡲גְ˖ ،فٔ؛٦ءّٝך㹀纏װرفٗ؎חכIBLPⵃ欽׃גְ˖ 㛇劤涸ח؟٦ؽأך涪罏ָ㹀纏剅ֹرفٗ؎דֹ״ֲחזגְ˖ 㹀纏ؿ؋؎ٕך剅䒭חכ+TPOOFUⵃ欽׃גְ˖ "84ך؟٦ؽأDPEFOJ[FUPPMTװ5FSSBGPSN⢪ג؝٦س⻉׃גְ˖ 涪罏כ䗳銲ד֮ל荈歋ח13⳿ֿׅהָדֹ״ֲחזגְIUUQTDPEFOJ[FUPPMTIUUQTHJUIVCDPNFBHMFUNUIBLP43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
؝ٝذشⰟ鸐㛇湍˖ &$4؎ٝأةٝأכ4QPU'MFFU"VUP4DBMJOH(SPVQד盖椚ׁגְ˖ &$4؎ٝأةٝأךקהוכ4QPUד⹛ְגְ˖ ؝أز٥鎘皾项彁ד㣐ֹזًٔحزָ欰תגְ˖ ؎ٝأةٝأךٔا٦أ٥&$4؟٦ؽأך朐䡾זו然钠׃גأ؛٦ٕ،ؐز׃גְ˖ ⰻ醡ד؟٦ؽأך朐䡾然钠דֹ8FC؝ٝا٦ٕ⡲גְ˖ ؟٦ؽأ荈⡤ךًزؙٔأװٗ٦سغٓٝ؟ךؙٔؒأز٥ٖأهٝأךًزؙٔأזו˖ ،فٔ؛٦ءّٝךؚٗךꠘ鋮٥嗚稊זוIUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODFIUUQTTQFBLFSEFDLDPNFBHMFUNUCVJMEJOHBTUFBEZFDTJOGSBTUSVDUVSF43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{scheduler: {type: 'ecs',cluster: 'eagletmt',desired_count: 2,role: 'ecsServiceRole',elb_v2: {vpc_id: 'vpc-WWWWWWWW',health_check_path: '/site/sha',listeners: [{port: 443,protocol: 'HTTPS',certificate_arn: 'arn:aws:iam::012345678901:server-certificate/hello-lb-v2.example.com',},],subnets: ['subnet-XXXXXXXX', 'subnet-YYYYYYYY'],security_groups: ['sg-ZZZZZZZZ'],:},},app: {image: 'ryotarai/hello-sinatra',:secrets: [{name: 'MESSAGE',value_from: 'arn:aws:ssm:ap-northeast-1:012345678901:parameter/hako/hello-lb-v2/secret-message',}],},sidecars: {front: {image_tag: 'hako-nginx',:},},:}43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
"WBJMBCJMJUZ˖ ؟٦ؽأך鸐⥋חֶֽ醱꧟䚍〳腉זꣲ㼰זֻׅ˖ 4FSWJDFNFTIך㼪Ⰵ˖ 黝ⴖזٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦˖ ꥺ㹱ָ饯ֹהֹח.553ָ〳腉זꣲ瀉ְ״ֲחׅ˖ .POJUPSJOH 0CTFSWBCJMJUZך然⥂זו˖ $IBPT&OHJOFFSJOHפך《穈˖ "-JGFPG$IBPT&OHJOFFSJOH4UBSUJOHXJUI3FTJMJFODFIUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
4FSWJDFNFTIך《穈˖ ؟٦ؽأ鸐⥋ךקרⰋגד&OWPZָ欽ְגְ˖ ،فٔ؛٦ءّٝכ➭ך؟٦ؽأח鸐⥋ׅ儗&OWPZ鸐ׅ״ֲחׅ˖ 黝ⴖחٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ָ涪⹛ׅ✲דِ٦ؠ䕦갟ך֮♧儗涸זؒٓ٦ָ慧幾˖ ؟٦ؽأ鸐⥋ך➭ח鸐䌢ך5$11SPYZה׃ג⢪גְ˖ &OWPZך$POUSPM1MBOFך؝ٝه٦طٝزה׃גDPPLQBEJUBDIP DPPLQBETET欽ְגְ˖ ٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ך鏣㹀זו♧⯋ד盖椚׃גְ˖ 鏣㹀ך鎸岀חכ+TPOOFUⵃ欽׃גְIUUQTHJUIVCDPNDPPLQBEJUBDIP IUUQTHJUIVCDPNDPPLQBETETIUUQTUFDIMJGFDPPLQBEDPNFOUSZ43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.POJUPSJOH 0CTFSWBCJMJUZפך《穈˖ 䖞勻ך؎ٝأةٝأך湊鋔כ׆ה֮;BCCJY⢪גְ˖ 㢩䕎湊鋔ה׃גך4UBUVT$BLFװ鎉铂׀הך鑫稢ז"1.ה׃גך/FX3FMJDזו֮˖ ⟃♴ך״ֲזًزؙٔأכ1SPNFUIFVTח꧊ג鑫稢ח鋅״ֲח׃גְ˖ ؎ٝأةٝأ٥ىسٕؐؑ،ך㛇劤涸זًزؙٔأ node_exporter ぐىسٕؐؑ،׀הךFYQPSUFS˖ ؝ٝذش橆㞮ך㛇劤涸זًزؙٔأ cAdvisor٥&OWPZַ《؟٦ؽأ鸐⥋ךًزؙٔأ˖ 1SPNFUIFVTח꧊ًزؙٔأַ،ٓ٦زׅ➬穈˖ +TPOOFU欽ְג3VMFך㹀纏"MFSUNBOBHFSך鏣㹀剅ֽ״ֲח׃גְ˖ "8493BZװ7J[DFSBMזו欽ְ%JTUSJCVUFE5SBDJOH 7JTVBMJ[BUJPOך➬穈IUUQTHJUIVCDPNSSSFFFZZZEPDLFSQSPNFUIFVTNJYJOUSFFNBTUFSFYBNQMF43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{prometheusAlerts+:: {groups+: [{name: 'example-alerts',rules: [{alert: 'ExampleCriticalAlertPrometheusDownFor1min',// PromQLexpr: |||up{job="prometheus"} == 0||| % $._config,'for': '1m',labels: {pager: 'sre',severity: 'critical',},annotations: {title: '{{$labels.job}} is unreachable',description: '{{$labels.job}} at {{$labels.instance}} could not be scraped for over 1 minutes.',runbook: 'https://wiki.example.com/pages/runbook/ExampleCriticalAlertPrometheusDownFor1min'},},],},],},}43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{alertmanagerConfig+:: {route+: {routes+: [{match_re: { pager: '^(sre)$' },receiver: 'sre-slack',routes: [{match: { severity: 'critical' },receiver: 'sre-pagerduty',}],}],},},}43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{alertmanagerConfig+:: {receivers+: [{name: 'sre-slack',slack_configs: [{send_resolved: true,channel: '#sre',},],},],},}43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{alertmanagerConfig+:: {receivers+: [{name: 'sre-pagerduty',pagerduty_configs: [{service_key: '{{ template "secrets.pagerduty.sre.service_key" }}'},],},],},}43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
תתדֹֿה٥װְֿהָ֮˖ 刿זإٕؿ؟٦ؽأ⻉˖ ぐ؟٦ؽأפךث٦ي䋆ֹ鴥4-*4-0ך㼪Ⰵ˖ 刿ז؟٦ؽأًحءُפך《穈˖ 刿ז؝ٝذشⰟ鸐㛇湍ך何㊣פך《穈˖ H31$זוך31$'SBNFXPSLפך《穈˖ 0CTFSWBCJMJUZ%JTUSJCVUFE5SBDJOHפך《穈˖ $IBPT&OHJOFFSJOHפך《穈˖ 婍גְٖؖء٦橆㞮ךٌتٝ⻉˖ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
8FBSFIJSJOH˖ IUUQTDPPLQBEKPCT˖ IUUQTDPPLQBEXPSLBCMFDPN43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ