Pro Yearly is on sale from $80 to $50! »

SRE-Lounge-8-Cookpad-Microservice-Architecture-Overview

28e154e6e0351c70091997d2f574295a?s=47 rrreeeyyy
March 13, 2019

 SRE-Lounge-8-Cookpad-Microservice-Architecture-Overview

SRE Lounge #8 でお話させていただきました

28e154e6e0351c70091997d2f574295a?s=128

rrreeeyyy

March 13, 2019
Tweet

Transcript

  1. $PPLQBE.JDSPTFSWJDF "SDIJUFDUVSF0WFSWJFX ؙحؙػحس吳䒭⠓爡䪮遭鿇43&ؚٕ٦ف し䊛畆㣕 !SSSFFFZZZ 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 

  2. !SSSFFFZZZ !SSSFFFZZZ IUUQTSSSFFFZZZDPN :PTIJLBXB3ZPUB .F ˖ :PTIJLBXB3ZPUB !SSSFFFZZZ[reɪ] ˖ ؙحؙػحس吳䒭⠓爡

    ։ ˖ 䪮遭鿇43&ؚٕ٦ف ˖ 莆㄂걄㚖 ˖ 0CTFSWBCJMJUZ 5JNFTFSJFTEBUBCBTF ˖ 馯㄂ ˖ QVZP FPS[FB MFBHVFPGMFHFOET HCG ˖ TFJZV SFTUBHF XVH 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  3. $PPLQBE43&T ˖ 秈せ +1 ˖ 㛇劤涸ח傈劤㕂ⰻד⹶⹡ ˖ 秈せ (MPCBM ˖

    ؎ؘٔأ׮׃ֻכ傈劤㕂ⰻד⹶⹡ 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  4. $PPLQBE +1(MPCBM ˖ +BQBO ˖ IUUQTDPPLQBEDPN ˖ (MPCBM $PVOUSJFT-BOHVBHFT ˖

    IUUQTDPPLQBEDPNVT ˖ IUUQTDPPLQBEDPNVL ˖ IUUQTDPPLQBEDPNJE ˖  43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  5. $PPLQBE +1(MPCBM ˖ +BQBO ˖ IUUQTDPPLQBEDPN ˖ (MPCBM ˖ ֿ׍׵ך《׶穈׫כGPPUOPUFTך⚛ןח׾⿫撑׃גֻ׌ְׁ

    IUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSFOETFBTPO IUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSF 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  6. $PPLQBE43&Tך顑⟣眔㔲 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 

  7. ؎ً٦آ 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 

  8. 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 

  9. 植㹋 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 

  10. 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 

  11. .JDSPTFSWJDFTBOE$PPLQBE 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 

  12. .JDSPTFSWJDFTBOE$PPLQBE  ˖ 䎃׀׹ַ׵و؎ؙٗ؟٦ؽأ⻉ח《׶穈׿דֹ׋ ˖ ؝ىُص؛٦ءّٝ٥顑⟣眔㔲٥Ꟛ涪⸬桦זו㼎ׅ׷㉏겗䠐陎 ˖ 㣐ֹז⽃♧ך؝٦سك٦أ٥؟٦ؽأ㔿剣ך㉏겗ָ׋ֻׁ׿֮׷ ˖ و؎ؙٗ؟٦ؽأ⻉ָ鹌׬ֿהח״׷㣐ֹזًٔحزך❦「

    ˖ Ꟛ涪鸞䏝ךぢ♳װ僇然ז顑⟣眔㔲٥ٔٔ٦أ鸞䏝ךぢ♳זו ˖ 㼭ְׁ؝٦سك٦أ٥؟٦ؽأז׵דכך兛鸐ךꟚ涪ָ遤ִ׷״ֲח 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  13. .JDSPTFSWJDFTBOE$PPLQBE  ˖ ♧倯דծو؎ؙٗ؟٦ؽأ⻉׾鹌׭׷ֿהד饯ֹ׷㉏겗׮֮׷ ˖ 43&ָⰋגך؟٦ؽأ׾鋅׷הأ؛٦ٕ׃זְ ˖ ؟٦ؽأ꟦鸐⥋ך㉏겗٥鸬ꓲ涸זꥺ㹱٥〳錁庠䚍ך㉏겗 ˖ 43&ך⚺噟⹡ך♧אכְֲֿ׏׋㉏겗ח㼎ׅ׷،فٗ٦ث

    ˖ ֿ׸תדוך״ֲח׃ג،فٗ٦ث׃גֹ׋ַ׾稱➜ ˖ 4DBMBCJMJUZ "WBJMBCJMJUZך錁挿ַ׵ 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  14. 4DBMBCJMJUZ ˖ أ؛٦ٓؽٔذ؍ծה鎉׏ג׮圫ղז⽃⡘ָ֮׷ ˖ ➂꟦ ˖ ؟٦غ ˖ ؝أز 43&-PVOHF

     ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  15. 4DBMBCJMJUZ ˖ ➂꟦ ˖ إٕؿ؟٦ؽأ⻉׾鹌׭׷ ˖ ؟٦غ ˖ "VUP4DBMJOHך崞欽 ˖

    ؝أز ˖ Ⱏ鸐㛇湍ך4QPU⻉ 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  16. إٕؿ؟٦ؽأ⻉ ˖ ؙحؙػحسדכ"NB[PO&$4׾ⵃ欽׃ג؝ٝذشךⰟ鸐㛇湍׾⡲׏גְ׷ ˖ ،فٔ؛٦ءّٝך㹀纏װرفٗ؎חכIBLP׾ⵃ欽׃גְ׷ ˖ 㛇劤涸ח؟٦ؽأךꟚ涪罏ָ㹀纏׾剅ֹرفٗ؎דֹ׷״ֲחז׏גְ׷ ˖ 㹀纏ؿ؋؎ٕך剅䒭חכ+TPOOFU׾ⵃ欽׃גְ׷ ˖

    "84ך؟٦ؽأ׮DPEFOJ[FUPPMTװ5FSSBGPSN׾⢪׏ג؝٦س⻉׃גְ׷ ˖ Ꟛ涪罏כ䗳銲ד֮׸ל荈歋ח13׾⳿ֿׅהָדֹ׷״ֲחז׏גְ׷ IUUQTDPEFOJ[FUPPMT IUUQTHJUIVCDPNFBHMFUNUIBLP 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  17. ؝ٝذشⰟ鸐㛇湍 ˖ &$4؎ٝأةٝأכ4QPU'MFFU"VUP4DBMJOH(SPVQד盖椚ׁ׸גְ׷ ˖ &$4؎ٝأةٝأךקה׿וכ4QPUד⹛ְגְ׷ ˖ ؝أز꬗٥鎘皾项彁꬗ד㣐ֹזًٔحزָ欰ת׸גְ׷ ˖ ؎ٝأةٝأךٔا٦أ٥&$4؟٦ؽأך朐䡾זו׾然钠׃גأ؛٦ٕ،ؐز׃גְ׷ ˖

    ⰻ醡ד؟٦ؽأך朐䡾׾然钠דֹ׷8FC؝ٝا٦ٕ׾⡲׏גְ׷ ˖ ؟٦ؽأ荈⡤ךًزؙٔأװٗ٦سغٓٝ؟ךؙٔؒأز٥ٖأهٝأךًزؙٔأזו ˖ ،فٔ؛٦ءّٝךؚٗךꠘ鋮٥嗚稊זו IUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF IUUQTTQFBLFSEFDLDPNFBHMFUNUCVJMEJOHBTUFBEZFDTJOGSBTUSVDUVSF 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  18. 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 

  19. 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 

  20. { scheduler: { type: 'ecs', cluster: 'eagletmt', desired_count: 2, role:

    'ecsServiceRole', elb_v2: { vpc_id: 'vpc-WWWWWWWW', health_check_path: '/site/sha', listeners: [ { port: 443, protocol: 'HTTPS', certificate_arn: 'arn:aws:iam::012345678901:server-certificate/hello-lb-v2.example.com', }, ], subnets: ['subnet-XXXXXXXX', 'subnet-YYYYYYYY'], security_groups: ['sg-ZZZZZZZZ'], : }, }, app: { image: 'ryotarai/hello-sinatra', : secrets: [{ name: 'MESSAGE', value_from: 'arn:aws:ssm:ap-northeast-1:012345678901:parameter/hako/hello-lb-v2/secret-message', }], }, sidecars: { front: { image_tag: 'hako-nginx', : }, }, : } 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  21. 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 

  22. 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 

  23. "WBJMBCJMJUZ ˖ ؟٦ؽأ꟦ך鸐⥋חֶֽ׷醱꧟䚍׾〳腉זꣲ׶㼰זֻׅ׷ ˖ 4FSWJDFNFTIך㼪Ⰵ ˖ 黝ⴖזٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ ˖ ꥺ㹱ָ饯ֹ׋הֹח.553ָ〳腉זꣲ׶瀉ְ״ֲחׅ׷ ˖

    .POJUPSJOH 0CTFSWBCJMJUZך然⥂זו ˖ $IBPT&OHJOFFSJOHפך《׶穈׫ ˖ "-JGFPG$IBPT&OHJOFFSJOH4UBSUJOHXJUI3FTJMJFODF IUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  24. 4FSWJDFNFTIך《׶穈׫ ˖ ؟٦ؽأ꟦鸐⥋ךקרⰋגד&OWPZָ欽ְ׵׸גְ׷ ˖ ،فٔ؛٦ءّٝכ➭ך؟٦ؽأח鸐⥋ׅ׷儗&OWPZ׾鸐ׅ״ֲחׅ׷ ˖ 黝ⴖחٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ָ涪⹛ׅ׷✲דِ٦ؠ䕦갟ך֮׷♧儗涸זؒٓ٦ָ慧幾 ˖ ؟٦ؽأ꟦鸐⥋ך➭ח鸐䌢ך5$11SPYZה׃ג׮⢪׏גְ׷ ˖

    &OWPZך$POUSPM1MBOFך؝ٝه٦طٝزה׃גDPPLQBEJUBDIP DPPLQBETET׾欽ְגְ׷ ˖ ٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ך鏣㹀זו׾♧⯋ד盖椚׃גְ׷ ˖ 鏣㹀ך鎸岀חכ+TPOOFU׾ⵃ欽׃גְ׷ IUUQTHJUIVCDPNDPPLQBEJUBDIP IUUQTHJUIVCDPNDPPLQBETET IUUQTUFDIMJGFDPPLQBEDPNFOUSZ 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  25. 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 

  26. .POJUPSJOH 0CTFSWBCJMJUZפך《׶穈׫ ˖ 䖞勻ך؎ٝأةٝأך湊鋔כ׆׏ה֮׷;BCCJY׮⢪׻׸גְ׷ ˖ 㢩䕎湊鋔ה׃גך4UBUVT$BLFװ鎉铂׀הך鑫稢ז"1.ה׃גך/FX3FMJDזו׮֮׷ ˖ ⟃♴ך״ֲזًزؙٔأכ1SPNFUIFVTח꧊׭ג鑫稢ח鋅׸׷״ֲח׃גְ׷ ˖ ؎ٝأةٝأ٥ىسٕؐؑ،ך㛇劤涸זًزؙٔأ

    node_exporter ぐىسٕؐؑ،׀הךFYQPSUFS ˖ ؝ٝذش橆㞮ך㛇劤涸זًزؙٔأ cAdvisor ٥&OWPZַ׵《׸׷؟٦ؽأ꟦鸐⥋ךًزؙٔأ ˖ 1SPNFUIFVTח꧊׭׋ًزؙٔأַ׵،ٓ٦زׅ׷➬穈׫ ˖ +TPOOFU׾欽ְג3VMFך㹀纏"MFSUNBOBHFSך鏣㹀׾剅ֽ׷״ֲח׃גְ׷ ˖ "8493BZװ7J[DFSBMזו׾欽ְ׋%JTUSJCVUFE5SBDJOH 7JTVBMJ[BUJPOך➬穈׫ IUUQTHJUIVCDPNSSSFFFZZZEPDLFSQSPNFUIFVTNJYJOUSFFNBTUFSFYBNQMF 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  27. 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 

  28. { prometheusAlerts+:: { groups+: [ { name: 'example-alerts', rules: [

    { alert: 'ExampleCriticalAlertPrometheusDownFor1min', // PromQL expr: ||| up{job="prometheus"} == 0 ||| % $._config, 'for': '1m', labels: { pager: 'sre', severity: 'critical', }, annotations: { title: '{{$labels.job}} is unreachable', description: '{{$labels.job}} at {{$labels.instance}} could not be scraped for over 1 minutes.', runbook: 'https://wiki.example.com/pages/runbook/ExampleCriticalAlertPrometheusDownFor1min' }, }, ], }, ], }, } 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  29. { alertmanagerConfig+:: { route+: { routes+: [{ match_re: { pager:

    '^(sre)$' }, receiver: 'sre-slack', routes: [{ match: { severity: 'critical' }, receiver: 'sre-pagerduty', }], }], }, }, } 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  30. { alertmanagerConfig+:: { receivers+: [ { name: 'sre-slack', slack_configs: [

    { send_resolved: true, channel: '#sre', }, ], }, ], }, } 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  31. { alertmanagerConfig+:: { receivers+: [ { name: 'sre-pagerduty', pagerduty_configs: [

    { service_key: '{{ template "secrets.pagerduty.sre.service_key" }}' }, ], }, ], }, } 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  32. 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 

  33. 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 

  34. 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 

  35. ת׌ת׌דֹ׷ֿה٥װ׶׋ְֿהָ֮׷ ˖ 刿ז׷إٕؿ؟٦ؽأ⻉ ˖ ぐ؟٦ؽأפךث٦ي׾䋆ֹ鴥׿׌4-*4-0ך㼪Ⰵ ˖ 刿ז׷؟٦ؽأًحءُפך《׶穈׫ ˖ 刿ז׷؝ٝذشⰟ鸐㛇湍ך何㊣פך《׶穈׫ ˖

    H31$זוך31$'SBNFXPSLפך《׶穈׫ ˖ 0CTFSWBCJMJUZ%JTUSJCVUFE5SBDJOHפך《׶穈׫ ˖ $IBPT&OHJOFFSJOHפך《׶穈׫ ˖ 婍׏גְ׷ٖؖء٦橆㞮ךٌتٝ⻉ ˖  43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ 
  36. 8FBSFIJSJOH ˖ IUUQTDPPLQBEKPCT ˖ IUUQTDPPLQBEXPSLBCMFDPN 43&-PVOHF  ]:PTIJLBXB3ZPUB !SSSFFFZZZ