Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SRE-Lounge-8-Cookpad-Microservice-Architecture-Overview

rrreeeyyy
March 13, 2019

 SRE-Lounge-8-Cookpad-Microservice-Architecture-Overview

SRE Lounge #8 でお話させていただきました

rrreeeyyy

March 13, 2019
Tweet

More Decks by rrreeeyyy

Other Decks in Technology

Transcript

  1. $PPLQBE.JDSPTFSWJDF
    "SDIJUFDUVSF0WFSWJFX
    ؙحؙػحس吳䒭⠓爡䪮遭鿇43&ؚٕ٦ف
    し䊛畆㣕 !SSSFFFZZZ

    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  2. !SSSFFFZZZ
    !SSSFFFZZZ
    IUUQTSSSFFFZZZDPN
    :PTIJLBXB3ZPUB
    .F
    ˖ :PTIJLBXB3ZPUB !SSSFFFZZZ[reɪ]

    ˖ ؙحؙػحس吳䒭⠓爡 ։

    ˖ 䪮遭鿇43&ؚٕ٦ف
    ˖ 莆㄂걄㚖
    ˖ 0CTFSWBCJMJUZ 5JNFTFSJFTEBUBCBTF
    ˖ 馯㄂
    ˖ QVZP FPS[FB MFBHVFPGMFHFOET HCG
    ˖ TFJZV SFTUBHF XVH
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  3. $PPLQBE43&T
    ˖ 秈せ +1

    ˖ 㛇劤涸ח傈劤㕂ⰻד⹶⹡
    ˖ 秈せ (MPCBM

    ˖ ؎ؘٔأ׮׃ֻכ傈劤㕂ⰻד⹶⹡
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  4. $PPLQBE +1(MPCBM

    ˖ +BQBO
    ˖ IUUQTDPPLQBEDPN
    ˖ (MPCBM $PVOUSJFT-BOHVBHFT

    ˖ IUUQTDPPLQBEDPNVT
    ˖ IUUQTDPPLQBEDPNVL
    ˖ IUUQTDPPLQBEDPNJE
    ˖
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  5. $PPLQBE +1(MPCBM

    ˖ +BQBO
    ˖ IUUQTDPPLQBEDPN
    ˖ (MPCBM
    ˖ ֿ׍׵ך《׶穈׫כGPPUOPUFTך⚛ןח׾⿫撑׃גֻ׌ְׁ
    IUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSFOETFBTPO
    IUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSF
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  6. $PPLQBE43&Tך顑⟣眔㔲
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  7. ؎ً٦آ
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  8. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  9. 植㹋
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  10. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  11. .JDSPTFSWJDFTBOE$PPLQBE
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  12. .JDSPTFSWJDFTBOE$PPLQBE

    ˖ 䎃׀׹ַ׵و؎ؙٗ؟٦ؽأ⻉ח《׶穈׿דֹ׋
    ˖ ؝ىُص؛٦ءّٝ٥顑⟣眔㔲٥Ꟛ涪⸬桦זו㼎ׅ׷㉏겗䠐陎
    ˖ 㣐ֹז⽃♧ך؝٦سك٦أ٥؟٦ؽأ㔿剣ך㉏겗ָ׋ֻׁ׿֮׷
    ˖ و؎ؙٗ؟٦ؽأ⻉ָ鹌׬ֿהח״׷㣐ֹזًٔحزך❦「
    ˖ Ꟛ涪鸞䏝ךぢ♳װ僇然ז顑⟣眔㔲٥ٔٔ٦أ鸞䏝ךぢ♳זו
    ˖ 㼭ְׁ؝٦سك٦أ٥؟٦ؽأז׵דכך兛鸐ךꟚ涪ָ遤ִ׷״ֲח
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  13. .JDSPTFSWJDFTBOE$PPLQBE

    ˖ ♧倯דծو؎ؙٗ؟٦ؽأ⻉׾鹌׭׷ֿהד饯ֹ׷㉏겗׮֮׷
    ˖ 43&ָⰋגך؟٦ؽأ׾鋅׷הأ؛٦ٕ׃זְ
    ˖ ؟٦ؽأ꟦鸐⥋ך㉏겗٥鸬ꓲ涸זꥺ㹱٥〳錁庠䚍ך㉏겗
    ˖ 43&ך⚺噟⹡ך♧אכְֲֿ׏׋㉏겗ח㼎ׅ׷،فٗ٦ث
    ˖ ֿ׸תדוך״ֲח׃ג،فٗ٦ث׃גֹ׋ַ׾稱➜
    ˖ 4DBMBCJMJUZ "WBJMBCJMJUZך錁挿ַ׵
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  14. 4DBMBCJMJUZ
    ˖ أ؛٦ٓؽٔذ؍ծה鎉׏ג׮圫ղז⽃⡘ָ֮׷
    ˖ ➂꟦
    ˖ ؟٦غ
    ˖ ؝أز
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  15. 4DBMBCJMJUZ
    ˖ ➂꟦
    ˖ إٕؿ؟٦ؽأ⻉׾鹌׭׷
    ˖ ؟٦غ
    ˖ "VUP4DBMJOHך崞欽
    ˖ ؝أز
    ˖ Ⱏ鸐㛇湍ך4QPU⻉
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  16. إٕؿ؟٦ؽأ⻉
    ˖ ؙحؙػحسדכ"NB[PO&$4׾ⵃ欽׃ג؝ٝذشךⰟ鸐㛇湍׾⡲׏גְ׷
    ˖ ،فٔ؛٦ءّٝך㹀纏װرفٗ؎חכIBLP׾ⵃ欽׃גְ׷
    ˖ 㛇劤涸ח؟٦ؽأךꟚ涪罏ָ㹀纏׾剅ֹرفٗ؎דֹ׷״ֲחז׏גְ׷
    ˖ 㹀纏ؿ؋؎ٕך剅䒭חכ+TPOOFU׾ⵃ欽׃גְ׷
    ˖ "84ך؟٦ؽأ׮DPEFOJ[FUPPMTװ5FSSBGPSN׾⢪׏ג؝٦س⻉׃גְ׷
    ˖ Ꟛ涪罏כ䗳銲ד֮׸ל荈歋ח13׾⳿ֿׅהָדֹ׷״ֲחז׏גְ׷
    IUUQTDPEFOJ[FUPPMT
    IUUQTHJUIVCDPNFBHMFUNUIBLP
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  17. ؝ٝذشⰟ鸐㛇湍
    ˖ &$4؎ٝأةٝأכ4QPU'MFFU"VUP4DBMJOH(SPVQד盖椚ׁ׸גְ׷
    ˖ &$4؎ٝأةٝأךקה׿וכ4QPUד⹛ְגְ׷
    ˖ ؝أز꬗٥鎘皾项彁꬗ד㣐ֹזًٔحزָ欰ת׸גְ׷
    ˖ ؎ٝأةٝأךٔا٦أ٥&$4؟٦ؽأך朐䡾זו׾然钠׃גأ؛٦ٕ،ؐز׃גְ׷
    ˖ ⰻ醡ד؟٦ؽأך朐䡾׾然钠דֹ׷8FC؝ٝا٦ٕ׾⡲׏גְ׷
    ˖ ؟٦ؽأ荈⡤ךًزؙٔأװٗ٦سغٓٝ؟ךؙٔؒأز٥ٖأهٝأךًزؙٔأזו
    ˖ ،فٔ؛٦ءّٝךؚٗךꠘ鋮٥嗚稊זו
    IUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF
    IUUQTTQFBLFSEFDLDPNFBHMFUNUCVJMEJOHBTUFBEZFDTJOGSBTUSVDUVSF
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  18. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  19. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  20. {
    scheduler: {
    type: 'ecs',
    cluster: 'eagletmt',
    desired_count: 2,
    role: 'ecsServiceRole',
    elb_v2: {
    vpc_id: 'vpc-WWWWWWWW',
    health_check_path: '/site/sha',
    listeners: [
    {
    port: 443,
    protocol: 'HTTPS',
    certificate_arn: 'arn:aws:iam::012345678901:server-certificate/hello-lb-v2.example.com',
    },
    ],
    subnets: ['subnet-XXXXXXXX', 'subnet-YYYYYYYY'],
    security_groups: ['sg-ZZZZZZZZ'],
    :
    },
    },
    app: {
    image: 'ryotarai/hello-sinatra',
    :
    secrets: [{
    name: 'MESSAGE',
    value_from: 'arn:aws:ssm:ap-northeast-1:012345678901:parameter/hako/hello-lb-v2/secret-message',
    }],
    },
    sidecars: {
    front: {
    image_tag: 'hako-nginx',
    :
    },
    },
    :
    }
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  21. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  22. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  23. "WBJMBCJMJUZ
    ˖ ؟٦ؽأ꟦ך鸐⥋חֶֽ׷醱꧟䚍׾〳腉זꣲ׶㼰זֻׅ׷
    ˖ 4FSWJDFNFTIך㼪Ⰵ
    ˖ 黝ⴖזٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦
    ˖ ꥺ㹱ָ饯ֹ׋הֹח.553ָ〳腉זꣲ׶瀉ְ״ֲחׅ׷
    ˖ .POJUPSJOH 0CTFSWBCJMJUZך然⥂זו
    ˖ $IBPT&OHJOFFSJOHפך《׶穈׫
    ˖ "-JGFPG$IBPT&OHJOFFSJOH4UBSUJOHXJUI3FTJMJFODF
    IUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  24. 4FSWJDFNFTIך《׶穈׫
    ˖ ؟٦ؽأ꟦鸐⥋ךקרⰋגד&OWPZָ欽ְ׵׸גְ׷
    ˖ ،فٔ؛٦ءّٝכ➭ך؟٦ؽأח鸐⥋ׅ׷儗&OWPZ׾鸐ׅ״ֲחׅ׷
    ˖ 黝ⴖחٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ָ涪⹛ׅ׷✲דِ٦ؠ䕦갟ך֮׷♧儗涸זؒٓ٦ָ慧幾
    ˖ ؟٦ؽأ꟦鸐⥋ך➭ח鸐䌢ך5$11SPYZה׃ג׮⢪׏גְ׷
    ˖ &OWPZך$POUSPM1MBOFך؝ٝه٦طٝزה׃גDPPLQBEJUBDIP DPPLQBETET׾欽ְגְ׷
    ˖ ٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ך鏣㹀זו׾♧⯋ד盖椚׃גְ׷
    ˖ 鏣㹀ך鎸岀חכ+TPOOFU׾ⵃ欽׃גְ׷
    IUUQTHJUIVCDPNDPPLQBEJUBDIP IUUQTHJUIVCDPNDPPLQBETET
    IUUQTUFDIMJGFDPPLQBEDPNFOUSZ
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  25. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  26. .POJUPSJOH 0CTFSWBCJMJUZפך《׶穈׫
    ˖ 䖞勻ך؎ٝأةٝأך湊鋔כ׆׏ה֮׷;BCCJY׮⢪׻׸גְ׷
    ˖ 㢩䕎湊鋔ה׃גך4UBUVT$BLFװ鎉铂׀הך鑫稢ז"1.ה׃גך/FX3FMJDזו׮֮׷
    ˖ ⟃♴ך״ֲזًزؙٔأכ1SPNFUIFVTח꧊׭ג鑫稢ח鋅׸׷״ֲח׃גְ׷
    ˖ ؎ٝأةٝأ٥ىسٕؐؑ،ך㛇劤涸זًزؙٔأ node_exporter ぐىسٕؐؑ،׀הךFYQPSUFS

    ˖ ؝ٝذش橆㞮ך㛇劤涸זًزؙٔأ cAdvisor
    ٥&OWPZַ׵《׸׷؟٦ؽأ꟦鸐⥋ךًزؙٔأ
    ˖ 1SPNFUIFVTח꧊׭׋ًزؙٔأַ׵،ٓ٦زׅ׷➬穈׫
    ˖ +TPOOFU׾欽ְג3VMFך㹀纏"MFSUNBOBHFSך鏣㹀׾剅ֽ׷״ֲח׃גְ׷
    ˖ "8493BZװ7J[DFSBMזו׾欽ְ׋%JTUSJCVUFE5SBDJOH 7JTVBMJ[BUJPOך➬穈׫
    IUUQTHJUIVCDPNSSSFFFZZZEPDLFSQSPNFUIFVTNJYJOUSFFNBTUFSFYBNQMF
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  27. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  28. {
    prometheusAlerts+:: {
    groups+: [
    {
    name: 'example-alerts',
    rules: [
    {
    alert: 'ExampleCriticalAlertPrometheusDownFor1min',
    // PromQL
    expr: |||
    up{job="prometheus"} == 0
    ||| % $._config,
    'for': '1m',
    labels: {
    pager: 'sre',
    severity: 'critical',
    },
    annotations: {
    title: '{{$labels.job}} is unreachable',
    description: '{{$labels.job}} at {{$labels.instance}} could not be scraped for over 1 minutes.',
    runbook: 'https://wiki.example.com/pages/runbook/ExampleCriticalAlertPrometheusDownFor1min'
    },
    },
    ],
    },
    ],
    },
    }
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  29. {
    alertmanagerConfig+:: {
    route+: {
    routes+: [{
    match_re: { pager: '^(sre)$' },
    receiver: 'sre-slack',
    routes: [{
    match: { severity: 'critical' },
    receiver: 'sre-pagerduty',
    }],
    }],
    },
    },
    }
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  30. {
    alertmanagerConfig+:: {
    receivers+: [
    {
    name: 'sre-slack',
    slack_configs: [
    {
    send_resolved: true,
    channel: '#sre',
    },
    ],
    },
    ],
    },
    }
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  31. {
    alertmanagerConfig+:: {
    receivers+: [
    {
    name: 'sre-pagerduty',
    pagerduty_configs: [
    {
    service_key: '{{ template "secrets.pagerduty.sre.service_key" }}'
    },
    ],
    },
    ],
    },
    }
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  32. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  33. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  34. 43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  35. ת׌ת׌דֹ׷ֿה٥װ׶׋ְֿהָ֮׷
    ˖ 刿ז׷إٕؿ؟٦ؽأ⻉
    ˖ ぐ؟٦ؽأפךث٦ي׾䋆ֹ鴥׿׌4-*4-0ך㼪Ⰵ
    ˖ 刿ז׷؟٦ؽأًحءُפך《׶穈׫
    ˖ 刿ז׷؝ٝذشⰟ鸐㛇湍ך何㊣פך《׶穈׫
    ˖ H31$זוך31$'SBNFXPSLפך《׶穈׫
    ˖ 0CTFSWBCJMJUZ%JTUSJCVUFE5SBDJOHפך《׶穈׫
    ˖ $IBPT&OHJOFFSJOHפך《׶穈׫
    ˖ 婍׏גְ׷ٖؖء٦橆㞮ךٌتٝ⻉
    ˖
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide

  36. 8FBSFIJSJOH
    ˖ IUUQTDPPLQBEKPCT
    ˖ IUUQTDPPLQBEXPSLBCMFDPN
    43&-PVOHF
    ]:PTIJLBXB3ZPUB !SSSFFFZZZ

    View Slide