SRE-Lounge-8-Cookpad-Microservice-Architecture-Overview

SRE Lounge #8 でお話させていただきました

Transcript

1. $PPLQBE.JDSPTFSWJDF
   "SDIJUFDUVSF0WFSWJFX
   ؙحؙػحس吳䒭⠓爡䪮遭鿇43&ؚٕ٦ف
   し䊛畆㣕 !SSSFFFZZZ
   
   43&-PVOHF
   ]:PTIJLBXB3ZPUB !SSSFFFZZZ
   
   

   View Slide

2. !SSSFFFZZZ
   !SSSFFFZZZ
   IUUQTSSSFFFZZZDPN
   :PTIJLBXB3ZPUB
   .F
   ˖ :PTIJLBXB3ZPUB !SSSFFFZZZ[reɪ]
   
   ˖ ؙحؙػحس吳䒭⠓爡 ։
   
   ˖ 䪮遭鿇43&ؚٕ٦ف
   ˖ 莆㄂걄㚖
   ˖ 0CTFSWBCJMJUZ 5JNFTFSJFTEBUBCBTF
   ˖ 馯㄂
   ˖ QVZP FPS[FB MFBHVFPGMFHFOET HCG
   ˖ TFJZV SFTUBHF XVH
   43&-PVOHF
   ]:PTIJLBXB3ZPUB !SSSFFFZZZ
   
   

   View Slide

3. $PPLQBE43&T
   ˖ 秈せ +1
   
   ˖ 㛇劤涸ח傈劤㕂ⰻד⹶⹡
   ˖ 秈せ (MPCBM
   
   ˖ ؎ؘٔأ׮׃ֻכ傈劤㕂ⰻד⹶⹡
   43&-PVOHF
   ]:PTIJLBXB3ZPUB !SSSFFFZZZ
   
   

   View Slide

4. $PPLQBE +1(MPCBM
   
   ˖ +BQBO
   ˖ IUUQTDPPLQBEDPN
   ˖ (MPCBM $PVOUSJFT-BOHVBHFT
   
   ˖ IUUQTDPPLQBEDPNVT
   ˖ IUUQTDPPLQBEDPNVL
   ˖ IUUQTDPPLQBEDPNJE
   ˖
   43&-PVOHF
   ]:PTIJLBXB3ZPUB !SSSFFFZZZ
   
   

   View Slide

5. $PPLQBE +1(MPCBM
   
   ˖ +BQBO
   ˖ IUUQTDPPLQBEDPN
   ˖ (MPCBM
   ˖ ֿ׍׵ך《׶穈׫כGPPUOPUFTך⚛ןח׾⿫撑׃גֻ׌ְׁ
   IUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSFOETFBTPO
   IUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSF
   43&-PVOHF
   ]:PTIJLBXB3ZPUB !SSSFFFZZZ
   
   

   View Slide

6. $PPLQBE43&Tך顑⟣眔㔲
   43&-PVOHF
   ]:PTIJLBXB3ZPUB !SSSFFFZZZ
   
   

   View Slide

7. ؎ً٦آ
   43&-PVOHF
   ]:PTIJLBXB3ZPUB !SSSFFFZZZ
   
   

   View Slide

8. 43&-PVOHF
   ]:PTIJLBXB3ZPUB !SSSFFFZZZ
   
   

   View Slide

9. 植㹋
   43&-PVOHF
   ]:PTIJLBXB3ZPUB !SSSFFFZZZ
   
   

   View Slide

10. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

11. .JDSPTFSWJDFTBOE$PPLQBE
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

12. .JDSPTFSWJDFTBOE$PPLQBE
    
    ˖ 䎃׀׹ַ׵و؎ؙٗ؟٦ؽأ⻉ח《׶穈׿דֹ׋
    ˖ ؝ىُص؛٦ءّٝ٥顑⟣眔㔲٥Ꟛ涪⸬桦זו㼎ׅ׷㉏겗䠐陎
    ˖ 㣐ֹז⽃♧ך؝٦سك٦أ٥؟٦ؽأ㔿剣ך㉏겗ָ׋ֻׁ׿֮׷
    ˖ و؎ؙٗ؟٦ؽأ⻉ָ鹌׬ֿהח״׷㣐ֹזًٔحزך❦「
    ˖ Ꟛ涪鸞䏝ךぢ♳װ僇然ז顑⟣眔㔲٥ٔٔ٦أ鸞䏝ךぢ♳זו
    ˖ 㼭ְׁ؝٦سك٦أ٥؟٦ؽأז׵דכך兛鸐ךꟚ涪ָ遤ִ׷״ֲח
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

13. .JDSPTFSWJDFTBOE$PPLQBE
    
    ˖ ♧倯דծو؎ؙٗ؟٦ؽأ⻉׾鹌׭׷ֿהד饯ֹ׷㉏겗׮֮׷
    ˖ 43&ָⰋגך؟٦ؽأ׾鋅׷הأ؛٦ٕ׃זְ
    ˖ ؟٦ؽأ꟦鸐⥋ך㉏겗٥鸬ꓲ涸זꥺ㹱٥〳錁庠䚍ך㉏겗
    ˖ 43&ך⚺噟⹡ך♧אכְֲֿ׏׋㉏겗ח㼎ׅ׷،فٗ٦ث
    ˖ ֿ׸תדוך״ֲח׃ג،فٗ٦ث׃גֹ׋ַ׾稱➜
    ˖ 4DBMBCJMJUZ "WBJMBCJMJUZך錁挿ַ׵
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

14. 4DBMBCJMJUZ
    ˖ أ؛٦ٓؽٔذ؍ծה鎉׏ג׮圫ղז⽃⡘ָ֮׷
    ˖ ➂꟦
    ˖ ؟٦غ
    ˖ ؝أز
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

15. 4DBMBCJMJUZ
    ˖ ➂꟦
    ˖ إٕؿ؟٦ؽأ⻉׾鹌׭׷
    ˖ ؟٦غ
    ˖ "VUP4DBMJOHך崞欽
    ˖ ؝أز
    ˖ Ⱏ鸐㛇湍ך4QPU⻉
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

16. إٕؿ؟٦ؽأ⻉
    ˖ ؙحؙػحسדכ"NB[PO&$4׾ⵃ欽׃ג؝ٝذشךⰟ鸐㛇湍׾⡲׏גְ׷
    ˖ ،فٔ؛٦ءّٝך㹀纏װرفٗ؎חכIBLP׾ⵃ欽׃גְ׷
    ˖ 㛇劤涸ח؟٦ؽأךꟚ涪罏ָ㹀纏׾剅ֹرفٗ؎דֹ׷״ֲחז׏גְ׷
    ˖ 㹀纏ؿ؋؎ٕך剅䒭חכ+TPOOFU׾ⵃ欽׃גְ׷
    ˖ "84ך؟٦ؽأ׮DPEFOJ[FUPPMTװ5FSSBGPSN׾⢪׏ג؝٦س⻉׃גְ׷
    ˖ Ꟛ涪罏כ䗳銲ד֮׸ל荈歋ח13׾⳿ֿׅהָדֹ׷״ֲחז׏גְ׷
    IUUQTDPEFOJ[FUPPMT
    IUUQTHJUIVCDPNFBHMFUNUIBLP
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

17. ؝ٝذشⰟ鸐㛇湍
    ˖ &$4؎ٝأةٝأכ4QPU'MFFU"VUP4DBMJOH(SPVQד盖椚ׁ׸גְ׷
    ˖ &$4؎ٝأةٝأךקה׿וכ4QPUד⹛ְגְ׷
    ˖ ؝أز꬗٥鎘皾项彁꬗ד㣐ֹזًٔحزָ欰ת׸גְ׷
    ˖ ؎ٝأةٝأךٔا٦أ٥&$4؟٦ؽأך朐䡾זו׾然钠׃גأ؛٦ٕ،ؐز׃גְ׷
    ˖ ⰻ醡ד؟٦ؽأך朐䡾׾然钠דֹ׷8FC؝ٝا٦ٕ׾⡲׏גְ׷
    ˖ ؟٦ؽأ荈⡤ךًزؙٔأװٗ٦سغٓٝ؟ךؙٔؒأز٥ٖأهٝأךًزؙٔأזו
    ˖ ،فٔ؛٦ءّٝךؚٗךꠘ鋮٥嗚稊זו
    IUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF
    IUUQTTQFBLFSEFDLDPNFBHMFUNUCVJMEJOHBTUFBEZFDTJOGSBTUSVDUVSF
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

18. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

19. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

20. {
    scheduler: {
    type: 'ecs',
    cluster: 'eagletmt',
    desired_count: 2,
    role: 'ecsServiceRole',
    elb_v2: {
    vpc_id: 'vpc-WWWWWWWW',
    health_check_path: '/site/sha',
    listeners: [
    {
    port: 443,
    protocol: 'HTTPS',
    certificate_arn: 'arn:aws:iam::012345678901:server-certificate/hello-lb-v2.example.com',
    },
    ],
    subnets: ['subnet-XXXXXXXX', 'subnet-YYYYYYYY'],
    security_groups: ['sg-ZZZZZZZZ'],
    :
    },
    },
    app: {
    image: 'ryotarai/hello-sinatra',
    :
    secrets: [{
    name: 'MESSAGE',
    value_from: 'arn:aws:ssm:ap-northeast-1:012345678901:parameter/hako/hello-lb-v2/secret-message',
    }],
    },
    sidecars: {
    front: {
    image_tag: 'hako-nginx',
    :
    },
    },
    :
    }
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

21. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

22. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

23. "WBJMBCJMJUZ
    ˖ ؟٦ؽأ꟦ך鸐⥋חֶֽ׷醱꧟䚍׾〳腉זꣲ׶㼰זֻׅ׷
    ˖ 4FSWJDFNFTIך㼪Ⰵ
    ˖ 黝ⴖזٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦
    ˖ ꥺ㹱ָ饯ֹ׋הֹח.553ָ〳腉זꣲ׶瀉ְ״ֲחׅ׷
    ˖ .POJUPSJOH 0CTFSWBCJMJUZך然⥂זו
    ˖ $IBPT&OHJOFFSJOHפך《׶穈׫
    ˖ "-JGFPG$IBPT&OHJOFFSJOH4UBSUJOHXJUI3FTJMJFODF
    IUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

24. 4FSWJDFNFTIך《׶穈׫
    ˖ ؟٦ؽأ꟦鸐⥋ךקרⰋגד&OWPZָ欽ְ׵׸גְ׷
    ˖ ،فٔ؛٦ءّٝכ➭ך؟٦ؽأח鸐⥋ׅ׷儗&OWPZ׾鸐ׅ״ֲחׅ׷
    ˖ 黝ⴖחٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ָ涪⹛ׅ׷✲דِ٦ؠ䕦갟ך֮׷♧儗涸זؒٓ٦ָ慧幾
    ˖ ؟٦ؽأ꟦鸐⥋ך➭ח鸐䌢ך5$11SPYZה׃ג׮⢪׏גְ׷
    ˖ &OWPZך$POUSPM1MBOFך؝ٝه٦طٝزה׃גDPPLQBEJUBDIP DPPLQBETET׾欽ְגְ׷
    ˖ ٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ך鏣㹀זו׾♧⯋ד盖椚׃גְ׷
    ˖ 鏣㹀ך鎸岀חכ+TPOOFU׾ⵃ欽׃גְ׷
    IUUQTHJUIVCDPNDPPLQBEJUBDIP IUUQTHJUIVCDPNDPPLQBETET
    IUUQTUFDIMJGFDPPLQBEDPNFOUSZ
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

25. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

26. .POJUPSJOH 0CTFSWBCJMJUZפך《׶穈׫
    ˖ 䖞勻ך؎ٝأةٝأך湊鋔כ׆׏ה֮׷;BCCJY׮⢪׻׸גְ׷
    ˖ 㢩䕎湊鋔ה׃גך4UBUVT$BLFװ鎉铂׀הך鑫稢ז"1.ה׃גך/FX3FMJDזו׮֮׷
    ˖ ⟃♴ך״ֲזًزؙٔأכ1SPNFUIFVTח꧊׭ג鑫稢ח鋅׸׷״ֲח׃גְ׷
    ˖ ؎ٝأةٝأ٥ىسٕؐؑ،ך㛇劤涸זًزؙٔأ node_exporter ぐىسٕؐؑ،׀הךFYQPSUFS
    
    ˖ ؝ٝذش橆㞮ך㛇劤涸זًزؙٔأ cAdvisor
    ٥&OWPZַ׵《׸׷؟٦ؽأ꟦鸐⥋ךًزؙٔأ
    ˖ 1SPNFUIFVTח꧊׭׋ًزؙٔأַ׵،ٓ٦زׅ׷➬穈׫
    ˖ +TPOOFU׾欽ְג3VMFך㹀纏"MFSUNBOBHFSך鏣㹀׾剅ֽ׷״ֲח׃גְ׷
    ˖ "8493BZװ7J[DFSBMזו׾欽ְ׋%JTUSJCVUFE5SBDJOH 7JTVBMJ[BUJPOך➬穈׫
    IUUQTHJUIVCDPNSSSFFFZZZEPDLFSQSPNFUIFVTNJYJOUSFFNBTUFSFYBNQMF
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

27. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

28. {
    prometheusAlerts+:: {
    groups+: [
    {
    name: 'example-alerts',
    rules: [
    {
    alert: 'ExampleCriticalAlertPrometheusDownFor1min',
    // PromQL
    expr: |||
    up{job="prometheus"} == 0
    ||| % $._config,
    'for': '1m',
    labels: {
    pager: 'sre',
    severity: 'critical',
    },
    annotations: {
    title: '{{$labels.job}} is unreachable',
    description: '{{$labels.job}} at {{$labels.instance}} could not be scraped for over 1 minutes.',
    runbook: 'https://wiki.example.com/pages/runbook/ExampleCriticalAlertPrometheusDownFor1min'
    },
    },
    ],
    },
    ],
    },
    }
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

29. {
    alertmanagerConfig+:: {
    route+: {
    routes+: [{
    match_re: { pager: '^(sre)$' },
    receiver: 'sre-slack',
    routes: [{
    match: { severity: 'critical' },
    receiver: 'sre-pagerduty',
    }],
    }],
    },
    },
    }
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

30. {
    alertmanagerConfig+:: {
    receivers+: [
    {
    name: 'sre-slack',
    slack_configs: [
    {
    send_resolved: true,
    channel: '#sre',
    },
    ],
    },
    ],
    },
    }
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

31. {
    alertmanagerConfig+:: {
    receivers+: [
    {
    name: 'sre-pagerduty',
    pagerduty_configs: [
    {
    service_key: '{{ template "secrets.pagerduty.sre.service_key" }}'
    },
    ],
    },
    ],
    },
    }
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

32. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

33. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

34. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

35. ת׌ת׌דֹ׷ֿה٥װ׶׋ְֿהָ֮׷
    ˖ 刿ז׷إٕؿ؟٦ؽأ⻉
    ˖ ぐ؟٦ؽأפךث٦ي׾䋆ֹ鴥׿׌4-*4-0ך㼪Ⰵ
    ˖ 刿ז׷؟٦ؽأًحءُפך《׶穈׫
    ˖ 刿ז׷؝ٝذشⰟ鸐㛇湍ך何㊣פך《׶穈׫
    ˖ H31$זוך31$'SBNFXPSLפך《׶穈׫
    ˖ 0CTFSWBCJMJUZ%JTUSJCVUFE5SBDJOHפך《׶穈׫
    ˖ $IBPT&OHJOFFSJOHפך《׶穈׫
    ˖ 婍׏גְ׷ٖؖء٦橆㞮ךٌتٝ⻉
    ˖
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide

36. 8FBSFIJSJOH
    ˖ IUUQTDPPLQBEKPCT
    ˖ IUUQTDPPLQBEXPSLBCMFDPN
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ
    
    

    View Slide