උ͑͋Ε͹ױ͍ͳ͠ ~ޮ཰తͳΠϯγσϯτରԠΛ໨ࢦ͢SREͷऔΓ૊Έd SRE NEXT 2023 2023-09-29 גࣜձࣾϕΨίʔϙϨʔγϣϯ খݪҰਅ @kazumax55

ࣗݾ঺հ খݪҰਅ(Kazuma Kohara) @kazumax55 γεςϜ౷ׅ෦ SRE෦ ෦௕ (SRE෦ = SRE group + Corporate IT group) झຯ 🏕Ωϟϯϓ 🎣௼Γ 🧖α΢φ ԶͨͪͷSREͱNew Relic -ॻ੶ฤ- NRUG SREࢧ෦ ӡӦ https://nrug-sre.connpass.com/

LOWYAʢϩ΢Ϡʣͱ͸ʁ 2004೥7݄ʹ෱ԬͰ૑ۀͨ͠Ո۩ɾΠϯςϦΞͷD2Cϒϥϯυ ঎඼։ൃ͔Βൢച·ͰΛҰؾ௨؏Ͱखֻ͚Δ͜ͱͰɺ τϨϯυੑͷߴ͍঎඼Λ͓ٻΊ΍͍͢Ձ֨Ͱ͓٬༷ʹఏڙ͍ͯ͠·͢ɻ 2023೥ ෱Ԭʹॳͷ௚ӦళΛOPENʂ2߸ళ͸େࡕͳΜ͹ύʔΫεʹग़ళʂ

LOWYAͷಛ௃ɿD2Cத৺ͷϏδωεϞσϧͰίετϝϦοτΛސ٬ؐݩ LOWYA=ϩʔϓϥΠεͳ͓ళ ௕͍όϦϡʔνΣʔϯΛҰؾ௨؏Ͱ୲͍ɺτϨϯυੑͷߴ͍঎඼Λ͓ٻΊ΍͍͢Ձ֨ͰͷఏڙΛ࣮ݱ

ࠓ೔͓࿩͢Δࣄ ޮ཰తʹΠϯγσϯτରԠΛߦ͏ͨΊͷ औΓ૊Έʹ͍ͭͯ

LOWYAࣄۀͷΠϯγσϯτରԠϑϩʔ උ͑͋Ε͹ױ͍ͳ͠ʂ ࠓ೔͸ϙΠϯτͱͳΔ੺ࣈͷ෦෼ʹ͍ͭͯ औΓ૊Έͷ಺༰Λ͝঺հ͠·͢ Πϯγσϯτ͕ऩଋͨ͠Βʁ ඇ೉ͷͳ͍ৼΓฦΓͱͯ͠PostmortemΛॻ͘ Πϯγσϯτ͕ൃੜͨ͠Βʁ ·ͣSeverity LevelsΛ൑ఆ͢Δ SEVϨϕϧผEscalation FlowʹԊͬͯ৘ใΛΤεΧϨʔγϣϯ͢Δ ΠϯγσϯτରԠΛ࣮ࢪ͢Δ ͞Βʹɺ๨Εͨࠒʹൃੜ͢ΔΠϯγσϯτʹඋ͑ͯ ఆظతʹIncident Fire DrillΛ࣮ࢪ͠ɺ܇࿅ͱطଘϑϩʔͷ఺ݕΛߦ͏

Severity Levels

Severity Levels Severity Levelsͱ͸ʁ Πϯγσϯτͷॏେ౓/ਂࠁ౓Λද͢Ϩϕϧ Өڹͷେ͖͞ΛϨϕϧผʹݴޠԽͨ͠΋ͷ Sevelity LevelsΛఆٛ͢Δ໨త ॏେ౓ͷೝࣝΛἧ͑ɺਝ଎͔ͭద੾ͳରԠΛߦ͏ ॏେ౓Λૉૣ͘൑அ͠ɺαʔϏε෮چʹΑΓଟ͘ͷ࣌ؒΛׂ͘

Severity Levels : ྫ Severity Description Typical Response SEV-1 n෼Ҏ্ͷαʔϏεఀࢭ͔ͭ෮چͷ໨ॲཱ͕ͨͳ͍ঢ়گ αΠόʔ߈ܸΛड͚͔ͭಥഁ͞ΕΔ ػີ৘ใͷ࿙Ӯ SEV-2ʹՃ͑ɺ ձࣾͱͯ͠αʔϏεঢ়گͷ֎෦։ࣔΛݕ౼͢Δ SEV-2 αʔϏεར༻͕Ͱ͖ͳ͍ϨϕϧͷύϑΥʔϚϯε௿Լ ػೳ͕ಈ࡞͠ͳ͍ SLOͷܧଓతͳ௿Լ ෮چʹඞཁͳମ੍Λ੔උ͠ɺ࠷༏ઌͰରԠ αʔϏεར༻ऀʹ޲͚ͨΞφ΢ϯεΛݕ౼͢Δ ͜ΕΑΓ্ͷϨϕϧ͸ॏେΠϯγσϯτͯ͠ѻ͏ SEV-3 ରԠ͕஗ΕΔͱக໋తʹͳΔΠϯγσϯτ ϢʔβʔʹӨڹͷ͋ΔΤϥʔɺδϣϒͷࣦഊ SLOͷҰ࣌తͳ௿Լ ௨ৗλεΫΑΓ༏ઌͯ͠ௐࠪɾରԠ SEV-4 ϢʔβʔʹӨڹͷ͋ΔܰඍͳΠϯγσϯτ ॲཧ஗Ԇ౳ ௨ৗλεΫΑΓ༏ઌ౓ߴΊͰௐࠪ SEV-5 Ϣʔβʔʹ௚઀Өڹ͠ͳ͍ܰඍͳΠϯγσϯτ IssueΛཱͯͯରԠ ͍͟ͱ͍͏ͱ͖ରԠʹ໎Θͳ͍Α͏ɺͰ͖Δ͚ͩγϯϓϧʹఆٛ͢Δ

Severity Levels : ఆٛͷίπ ୭Ͱ΋༰қʹ൑ఆͰ͖ΔΑ͏ɺᐆດͳදݱΛආ͚ͯγϯϓϧʹ͢Δ ஗ԆͳΒSEV-4, ΤϥʔͳΒSEV-3 ౳ ᐆດͳදݱ͸ආ͚Δ ॏཁͳػೳͷ… ͱ͔ɺ௕࣌ؒͷఀࢭ… Έ͍ͨͳදݱ ΍ΔͳΒ Կ෼Ҏ্ ͱ͔ɺॏେͳػೳ = AͱBͱC ͷΑ͏ʹܾΊΔ ໎ͬͨͱ͖͸ɺ্Ґͷॏେ౓ͱͯ͠ѻ͏ ૣ͘Ϩϕϧ൑ఆͰ͖ΔΑ͏ʹ͢ΔͨΊͷϧʔϧ

Severity Levels : ࢀߟυΩϡϝϯτ PagerDutyɿIncident Respons > Before an Incident > Severity Levels https://response.pagerduty.com/before/severity_levels/ AtlassianɿΠϯγσϯτͷॏେ౓Ϩϕϧͷ೺Ѳ https://www.atlassian.com/ja/incident-management/kpis/severity-levels ͱͯ΋ࢀߟʹͳΓ·͕͢ɺᐆດͳදݱ΋ଟ͘൑அʹ໎͏ࣄ͕͋ͬͨҝ ݟ௚͍ͯͬͨ݁͠ՌɺઌఔͷΑ͏ͳγϯϓϧͳϧʔϧʹམͪண͍ͨܦҢ͕͋Γ·͢ ༨ஊɿSlackελϯϓΛ࡞͓ͬͯ͘ͱศར SEV-1, SEV-2, SEV-3, SEV-4, SEV-5

Escalation Flow

Escalation Flow ΤεΧϨʔγϣϯϑϩʔͱ͸ʁ ༗ࣄͷࡍͷ৘ใڞ༗ઌΛSEVϨϕϧผʹ༧Ίఆٛͨ͠΋ͷ ΤεΧϨʔγϣϯϑϩʔΛఆٛ͢Δ໨త ༗ࣄͷࡍʹ໎ΘͣεϜʔζͳ৘ใڞ༗Λߦ͏ͨΊ SEVϨϕϧผʹʮ୭ʹใࠂ͢Δ͔ʁʯʮνʔϜͰͲͷΑ͏ʹ໾ׂ෼୲ͯ͠ಈ͔͘?ʯΛఆ Ί͓ͯ͘ ؔ܎ऀ୭͚ͩͬʁ ࿈བྷઌͲ͚ͩͬ͜ʁ

Escalation Flow : ྫ SEVϨϕϧຖͷΤεΧϨʔγϣϯઌͷఆٛ Πϯγσϯτใࠂ༻Slack channelΛ࡞Γؔ܎ऀΛট଴͠ɺॏཁͳ৘ใ͸ͦ͜ʹpinཹΊ͓ͯ͘͠ PinཹΊਪ঑ɿΠϯγσϯτରԠ࣌ʹࢀর͢Δ֤छυΩϡϝϯτ, VIDEOձٞURL౳ SEVϨϕϧผͷΤεΧϨʔγϣϯઌΛఆ͓ٛͯ͘͠ όΠωʔϜͰ͸ͳ͘ɺ໾৬Ͱهࡌ͢ΔʢϝϯςφϯείετΛݮΒͨ͢Ί ۀ຿࣌ؒ֎ͷۓٸ࿈བྷઌϦετ Slackʹ൓Ԡ͕ͳ͍࣌ɺۓٸ࿈བྷઌѼʹ࿈བྷ͢Δͷ͔౳΋༧ΊܾΊͯ߹ҙΛऔΔ ༨ஊɿखಈPagerൃՐखஈ ૝ఆ֎ͷΠϯγσϯτൃੜ࣌ʹɺΦϯίʔϧ౰൪ΛSlack͔Βݺͼग़ͤΔΑ͏ͳ΋ͷ SlackͷWork fl owͰ࡞͓ͬͯ͘ͱศར

Escalation Flow : ໾ׂ෼୲ ͞·͟·ͳ໾ׂ(PagerDuty͔ΒҾ༻)

Escalation Flow : ໾ׂ෼୲ͷࢀߟυΩϡϝϯτ PagerDutyɿIncident Respons > Before an Incident > Different Roles https://response.pagerduty.com/before/different_roles/ PagerDutyɿIncident Respons > Training > Incident Commander https://response.pagerduty.com/training/incident_commander/ New RelicɿNew RelicྲྀɺΦϯίʔϧͱΠϯγσϯτରԠͷ੒ޭ΁ͷಓ https://newrelic.com/jp/blog/how-to-relic/on-call-and-incident-response-new-relic-best-practices-2 αΠτͷن໛ʹΑͬͯ͸ɺ໾ׂ෼୲ʹ͍ͭͯ΋ݚڀͯ͠ӡ༻ʹऔΓೖΕΒΕΔΑ͏τϨʔχϯά ͓ͯ͘͠ͱྑ͍Ͱ͠ΐ͏

Postmortem

Postmortem ϙετϞʔςϜͱ͸ʁ ΠϯγσϯτͷࣄޙରԠͱ࣮ͯ͠ࢪ͢Δඇ೉ͷͳ͍ৼΓฦΓ ϙετϞʔςϜͷ໨త ࣦഊ͔Βֶͼɺಉ͡աͪΛ܁Γฦ͞ͳ͍ ࠜຊݪҼͷ֬ೝͱޮՌతͳ࠶ൃ๷ࢭࡦΛݕ౼͢Δ

Postmortem : ॏཁͳϙΠϯτ ඇ೉͠ͳ͍ɾ੹೚௥ٴͷ৔ʹ͠ͳ͍ ϛεͨ͠ਓΛɺԜ·ͤͯ΋Կ΋͍͍͜ͱ͸ͳ͍ લ޲͖ʹվળʹ͍ͭͯ࿩͠߹͏৔ʹ͢Δ ݪҼಛఆ΍࠶ൃ๷ࢭ͸࢓૊ΈͰߟ͑Δ ࠶ݱੑͷ͋ΔखॱΛಋ͖ग़͢ҝɺݸਓͷ஫ҙ΍ೳྗʹཔΒͳ͍ ۩ମతͳνΣοΫ߲໨Λߟ͑ͳ͍ͱɺԿਓͰ֬ೝͯ͠΋ҙຯ͕ͳ͍ NGϫʔυɿ͔ͬ͠Γɺ஫ҙਂ͘ ಉ͡աͪΛ๷͙ʹ͸ɺԿ͕Ͳ͏͍͏ঢ়ଶͰ͋Δ͜ͱΛ֬ೝ͢΂͖͔ ͞ΒʹͦΕΛ࢓૊Έͱཱͯ֬͠Ͱ͖ͳ͍͔ߟ͑Δ͜ͱ͕ॏཁ

Postmortem : ޻෉͍ͯ͠Δ఺ ೋ౓ݟ͢Δ ि࣍։࠵͢ΔΤϯδχΞఆྫͰશମʹڞ༗ ͞ΒʹSRE+ϓϩμΫτ։ൃνʔϜؒͰ݄࣍։࠵͢ΔSLO ReviewͰ࠶֬ೝ ࣌ؒΛ͓͍ͯ࠶֬ೝ͢Δͱɺ৽ͨͳٙ໰΍վળ఺͕ු͔ΜͰ͘Δࣄ͕͋Δ ςϯϓϨʔτʹΠϯγσϯτରԠϑϩʔͷৼΓฦΓ߲໨Λ௥Ճ SEV൑ఆ͍͖ͤͣͳΓमਖ਼Λ࢝Ίͯ͠·͏νʔϜʹɺϑϩʔΛ࠶ೝࣝͯ͠΋Β͏ 1. ॳಈͰSEV൑ఆͰ͖·͔ͨ͠ʁ 2. ࣄલʹఆΊͨϑϩʔʹԊͬͯΤεΧϨʔγϣϯ͕ग़དྷ·͔ͨ͠ʁ ࠓޙ΍Γ͍ͨ͜ͱɿϙετϞʔςϜͷ෼ੳ ҰఆͷपظͰϥϯΩϯάʢτϦΨʔ, ࠜຊݪҼ, MTTR ౳ʣΛग़ͯ͠τϨϯυ֬ೝ΍஫ҙשىΛ͢Δ

Postmortem : ࢀߟυΩϡϝϯτ ʮαΠτϦϥΠΞϏϦςΟϫʔΫϒοΫʯ10ষ ϙετϞʔςϜͷจԽɿࣦഊ͔Βͷֶͼ ྑ͘ͳ͍ϙετϞʔςϜ, ྑ͍ϙετϞʔςϜ ʮSRE αΠτϦϥΠΞΠϦςΟΤϯδχΞϦϯάʯͷςϯϓϨʔτվగ൛ https://bit.ly/2kIhFNC GoogleɿPostmortem Checklist https://g.co/SiteReliabilityWorkbookMaterials PagerDutyɿIncident Respons > After an Incident > Postmortem Template https://response.pagerduty.com/after/post_mortem_template/

Incident Fire Drill

Incident Fire Drill ΠϯγσϯτFire Drillͱ͸ʁ FireDrillʹආ೉܇࿅/ফ๷܇࿅ Πϯγσϯτͷൃੜʹඋ͑ͯߦ͏໛ٖ܇࿅ ΠϯγσϯτFire Drillͷ໨త ܦݧΛ { ੵΜͰ, ࢥ͍ग़ͯ͠ } ΋Β͏ ༗ࣄͷࡍʹରԠʹ໎Θͳ͍Α͏ʹ͢Δ ΠϯγσϯτରԠͷҰ࿈ͷϑϩʔΛ఺ݕ͠վળ͢Δ

Incident Fire Drill : ૝ఆγφϦΦ ૝ఆγφϦΦΛߟ͑ΔʢAWSدΓʣ 2. ௕࣌ؒ෮چͷݟࠐΈཱ͕ͨͳ͍AZো֐͕ൃੜͨ͠ IaCͰյͯ͠΋ྑ͍؀ڥΛ࡞Γ (※)AWS FIS ౳Λ࢖ͬͯAZো֐Λىͯ͜͠ΈΔ (※)AWS Fault Injection Simulator ૝ఆ௨Γͷಈ࡞Λ͠ͳ͔ͬͨ৔߹ɺϚϧνAZରԠͷϥϯχϯάίετɾ޻਺͕ݟੵ΋ΕΔ AWS Resilience HubΛ࢖͏ͱɺܭըͨ͠RTO/RPOʹԊ͍ͬͯΔ͔ࣄલʹධՁͰ͖Δ 1. ͱ͋Δमਖ਼ͷσϓϩΠޙɺػೳ͕ಈ࡞͠ͳ͘ͳͬͨ ϩά/ΦϒβʔόϏϦςΟπʔϧ౳Λ࢖ͬͨτϥϒϧγϡʔςΟϯάͷ܇࿅ɺ੾Γ໭͠ͷ܇࿅ 3. ৘ใ࿙Ӯ/ηΩϡϦςΟܥΠϯγσϯτ ηΩϡϦςΟΠϯγσϯτൃੜ࣌Λ૝ఆͨ͠܇࿅ → AWS GameDay / ײ૝هࣄ SIEM on AOSͷར׆༻ɺൃݟత౷੍ɾ༧๷త౷੍ͷ؍఺Ͱඞཁͳ४උ͕Ͱ͖͍ͯΔ͔Λݟ௚͢ AWSηΩϡϦςΟΠϯγσϯτٙࣅମݧ GameDay ʹSREνʔϜͰࢀՃ͠ɺͱͯ΋ྑ͍ܦݧΛಘΒΕͨ

Incident Fire Drill : ࣮ࢪํ๏ ࣮ࢪํ๏ͷྫ “ͱ͋Δमਖ਼ΛσϓϩΠޙɺػೳ͕ಈ࡞͠ͳ͘ͳͬͨ” ͱԾఆ SRE͕ओ࠵ɺ֤DEVνʔϜ͕ࢀՃऀ τʔλϧ60෼ɿ໨త/֓ཁ/࣭ٙԠ౴ 20෼, ࣮ࢪ30෼, ৼΓฦΓ10෼ ΠϯγσϯτରԠϑϩʔͷҰ࿈ͷྲྀΕΛ௨ͯ͠΋Β͏ ·ͣSEV൑ఆɺΤεΧϨʔγϣϯɺௐࠪ/ରԠɺϙετϞʔςϜ ࣮ࢪͷίπ VIDEOձٞͷϒϨΠΫΞ΢τϧʔϜΛ࢖ͬͯɺ֤νʔϜ͝ͱʹ࣮ࢪ SRE͸࢓੾Δͱ܇࿅ʹͳΒͳ͍ͷͰɺαϙʔτʹప͢Δ ࣭໰͕͋ͬͨΒճ౴͢Δఔ౓ͰOK

Incident Fire Drill : ఺ݕ಺༰ ఺ݕ಺༰ Ұ࿈ͷΠϯγσϯτରԠϑϩʔ͕௨ͤΔ͜ͱ ಺༰ʹෆඋ΍վળ఺͕ͳ͍͔ ୭Ͱ΋SEV൑ఆ͕Ͱ͖Δ͔ɺ൑அʹ໎͏Օॴ͕ͳ͍͔ υΩϡϝϯτ΍ΤεΧϨʔγϣϯઌΛ໎Θͣʹ୳ͤΔ͔ ݪҼͷௐࠪ/෮چ͕Ͱ͖Δ͜ͱ ϩά, ΦϒβʔόϏϦςΟπʔϧ౳͔ΒΤϥʔ಺༰͕֬ೝͰ͖Δ͔ ௐࠪ/෮چͷखॱ͕ڞ༗͞Ε͍ͯΔ͔ ઀ଓ৘ใ΍ϊ΢ϋ΢΍ଐਓԽ͍ͯ͠ͳ͍͔ ඞཁʹԠͯ͡ɺΠϯγσϯτରԠͷڞ༗ΛΦϯϘʔσΟϯάʹՃ͑ͯ΋Β͏

Incident Fire Drill : ৼΓฦΓ ৼΓฦΓ ֤νʔϜͷ୅දऀʹ ֶͼ/վળ఺ ʹ͍ͭͯҰݴͣͭίϝϯτΛ΋Β͏ ଞνʔϜͷؾ෇͖͕ɺ શһʹΞϯέʔτΛ͓ئ͍͠ɺ࣍ճͷࢀߟʹ͢Δ Ξϯέʔτ݁ՌɿࢀՃऀͷ੠ʢൈਮʣ νʔϜ಺ͰΠϯγσϯτͷରԠखॱΛڞ༗ɾ֬ೝͰ͖ͯΑ͔ͬͨ υΩϡϝϯτ΍ϑϩʔͷվળ఺͕ݟ͔ͭͬͨ γφϦΦΛม͑ͯఆظతʹ࣮ࢪ͍ͨ͠ ͕࣌ؒ଍Γͳ͔ͬͨ(આ໌20෼, ࣮ࢪ30෼, ৼΓฦΓ10෼ ӡӦଆ΋վળ͕ඞཁ) ࢀՃશνʔϜʹ ԿΒ͔ͷվળ఺͕ݟ͔ͭΓ·ͨ͠

·ͱΊ

·ͱΊ ޮ཰తͳΠϯγσϯτରԠΛ͢Δʹ͸ʁ ͍͟ͱ͍͏ͱ͖ରԠʹ໎Θ͵Α͏ɺ υΩϡϝϯτ΍ϑϩʔͷࣄલ੔උ͕ॏཁ ͦͯ͠ɺ๨Εͨࠒʹ΍ͬͯ͘ΔΠϯγσϯτʹඋ͑ͯɺ ఆظతʹτϨʔχϯά΍ݟ௚͠Λ͠·͠ΐ͏ උ͑͋Ε͹ױ͍ͳ͠ʂ ຊηογϣϯ͕ɺօ༷ʹͱͬͯ ޮ཰తͳΠϯγσϯτରԠΛਐΊΔ͖͔͚ͬʹͳΕ͹޾͍Ͱ͢

No content