Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
インシデントキーメトリクスによるインシデント対応の改善 / Improving Inciden...
Search
Narimichi Takamura
January 26, 2025
Technology
14k
1
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
インシデントキーメトリクスによるインシデント対応の改善 / Improving Incident Response using Incident Key Metrics
SRE Kaigi 2025の発表資料です。TTXメトリクスがメイントピックです。
https://2025.srekaigi.net/
Narimichi Takamura
January 26, 2025
Transcript
56 slides
None
2
גࣜձࣾTopotalʢͱΆͨΔʣ • h#ps:/ /topotal.com • SREΛओ࣠ʹͨ͠ελʔτΞοϓ • 2ࣄۀΛӡӦ • SRE
as a Service • SaaS for SREʢWaroomʣ • ຊΠϕϯτͷ Pla;num εϙϯαʔ 3
SRE as a Service • topotal.com/services/sre-as-a-service • SREʹಛԽٕͨ͠ज़ࢧԉαʔϏε • ࢧԉͷྫ
• SLI/SLOͷಋೖɾӡ༻վળ • CI/CDͷߏஙɾվળ • ΠϯγσϯτϚωδϝϯτͷվળ 4
WaroomʢΘΔʔΉʣ • waroom.com • ৫తʹΠϯγσϯτରԠΛߦ͏ͨΊ ͷSaaS • Slack ϕʔεͷରԠʹ߹ΘͤͯࣗಈԽɾ লྗԽ͕Ͱ͖Δ
5
6
վળͷϑΟʔυόοΫΛߏங͢Δ 7
8
ΞδΣϯμ 1. MTTRͷ 2. ࣮ફతͳ TTX ϝτϦΫεͷఆٛ 3. TTX ϝτϦΫεͷ׆༻ྫ
4. ൃలతͳϝτϦΫε 9
1. MTTRͷ 10
MTTRʢฏۉ෮چ࣌ؒʣ ͱ • ো͕ൃੜ͔ͯ͠Βम෮·ͨ෮چ͢Δ ·Ͱͷฏۉ࣌ؒͷ͜ͱ • Mean Time To Recovery(Repair,
Resolve, Restore)ͷུ • ࢉग़ํ๏1 • MTTR = ૯मཧ࣌ؒ / ૯ނোճ • Four Keys ͷࢦඪͷҰͭͰ͋Δ 1 MTTRʢฏۉ෮چ࣌ؒʣͱʁܭࢉํ๏ͱMTBFͱͷނোɾՔಇʹ ͓͚Δؔ 11
12
SREs should move away from defaul/ng to the assump/on that
MTTX can be useful. 13
MTTRͷ༗ޮੑͷݕূ • Ծઆ • MTTR͕༗ޮͳࢦඪͳͷͰ͋ΕɺTTRΛվળʢॖʣ͢ΔͱMTTRվ ળ͞ΕΔͣ • ݕূ֓ཁ • σʔληοτΛ1:1Ͱׂ͠ɺยํTTRΛ10%վળɺ͏ยํͳʹ
͠ͳ͍ͰMTTRΛࢉग़ɾൺֱ͢Δ • MTTR͕10%վળ͞ΕΔ͔Ͳ͏͔Λ֬ೝ͢Δ 14
MTTRͷ༗ޮੑͷݕূ 1. Πϯγσϯτͷσʔληοτ2ΛϥϯμϜʹ2ׂ͢Δ 2. ยํͷσʔληοτͷम෮࣌ؒ(TTR)Λ10%ݮΒ͢ 3. ֤σʔληοτͷMTTR(ฏۉम෮࣌ؒ)Λܭࢉ͢Δ 4. σʔληοτؒͷMTTRͷࠩΛऔΔ •
diff = MTTR(unmodified) - MTTR(modified) • diff > 0 => MTTRվળ • diff < 0 => MTTRѱԽ 5. 1ʙ4Λ10ສճ܁Γฦ͢ 2 σʔληοτɺ༗໊ͳΠϯλʔ ωοτاۀ3ࣾͷΠϯγσϯτες ʔλεμογϡϘʔυ͔Βऔಘ 15
Πϯγσϯτσʔλͷಛ3 • େ͔ͳΓૣ͘ऩଋ͢Δ • Ұ෦൵ࢂͳΠϯγσϯτʢϒϥοΫ εϫϯΠϕϯτʣʹͳΔ • → ແ࡞ҝʹσʔληοτΛׂ͢Δ ͱɺ൵ࢂͳΠϯγσϯτͷภΓ͕
MTTRͷࢉग़ʹେ͖ͳӨڹΛٴ΅͢ 3 The VOID Report 16
ࢀߟ: ϒϥοΫεϫϯΠϕϯτ • ༧ظͰ͖ͳ͍ɺյ໓తͳ݁ՌΛҾ͖ى ͜͢ࣄ • ϤʔϩούͰനௗന͍ௗ͚ͩͱࢥ ΘΕ͍ͯͨ • "༧ظ͞Εͳ͍େ͖ͳग़དྷࣄ"
Λ “ϒ ϥοΫεϫϯ” ͱݺͿΑ͏ʹͳͬͨ • 2007ʹൃץ͞ΕͨʮThe Black Swanʯ͕͖͔͚ͬ 17
γϛϡϨʔγϣϯ݁Ռ ֤Πϯγσϯτͷम෮࣌ؒΛ10%ͨ͘͠ʹ͔͔ΘΒͣɺMTTR͕10%Ҏ্͘ͳΔέʔε49%ɺ50%ɺ64%ͷΈ → ͘Β͍ɺम෮࣌ؒͷॖ͕MTTRʹө͞Εͳ͍ 18
ࢀߟ: म෮࣌ؒΛมߋͤͣʹγϛϡϨʔγϣϯͨ݁͠Ռ → վળ׆ಈͷ༗ແʹ͔͔ΘΒͣɺMTTRσʔληοτ࣍ୈͰվળ or ѱԽ͢Δ 19
Incident Metrics in SRE ͷओு • γϛϡϨʔγϣϯ͔ΒΘ͔ͬͨ͜ͱ • ΠϯγσϯτނোظؒͷΒ͖͕ͭେ͖͍ͨΊɺվળ݁Ռ͕ MTTR
ʹө͞ΕͮΒ͍ • վળͯ͠ѱԽ͢Δέʔεͦͦ͋͜͜Δ • ݁ • MTTR վળͷධՁࢦඪͱͯ͠ʹཱͨͳ͍ 20
ͳʹ͕ͩͬͨͷʁ • Πϯγσϯτظؒͷมಈੑ͕ߴ͍͜ͱ • MTTRΛͳΜΒ͔ͷࢦඪʹ͢Δ͜ͱ • ࢦඪΛͱʹվળͷՌΛ֬ೝ͢Δ͜ͱ ֤ཁૉͳ͍ → తͱࢦඪ͕טΈ߹͍ͬͯͳ͍͜ͱ͕
21
σʔλੳʢԾઆݕূܕʣͷྲྀΕ 22
MTTRΛࢦඪʹ͢Δͱ͖ͷࢥߟͷྲྀΕ 23
ى͖͍ͯͨ͜ͱ: ԾઆݕূϩδοΫͷෆ߹ 24
ղܾࡦ: վળՕॴΛ໌Β͔ʹ͠ɺมಈੑΛ͑Δ 25
ղܾࡦ: վળՕॴΛ໌Β͔ʹ͠ɺมಈੑΛ͑Δ 26
ิ: TTRͷ͍ಓ ฏۉ(MTTR)େࡶ͗͢Δ → ͷൺֱ՝ൃݟͷࢳޱʹͳΔ • ex. ଈ࣌෮چͷো͕ݮগ • →
ܰඍͳোͷࣗಈ෮چͷՌʁ • → োݕͷΈʹෆ۩߹ʁ • ex. ϒϥοΫεϫϯΠϕϯτ͕૿Ճ • → ίʔυΠϯϑϥͷ࣭Լʁ 27
͜͜·Ͱͷ·ͱΊ • MTTR(෮چ࣌ؒ)σʔλมಈੑ͕ߴ͍ͨΊվળࢦඪʹෆద • վળՕॴΛ໌֬Խ͠ɺΑΓࡉ͔͍ TTX ϝτϦΫεΛར༻͢Δ͜ ͱͰɺมಈੑΛ͑Δ͜ͱ͕Մೳ → TTRΑΓࡉ͔͍ϝτϦΫεͷधཁ͕ग़ͯ͘Δ
28
2. ࣮ફతͳ TTX ϝτϦΫε 29
Waroom͕ߟ͑Δ࣮ફతͳϝτϦΫεͱ • ཏతͰ͋Δ͜ͱ • ཻ͕ࡉ͔͍͜ͱ • ऩू͕ݱ࣮తͰ͋Δ͜ͱ 30
ͲΜͳTTXϝτϦΫεΛ ऩू͢ΔͱΑ͍ͩΖ͏͔ 31
32
TTXϝτϦΫεͷ՝ײ • ੈͷதʹࣄྫ͍͔ͭ͋͘Δ͕ɺఆٛ౷Ұ͞Ε͍ͯͳ͍ • ࣄྫಉ࢜ΛΈ߹ΘͤΑ͏ͱͯ͠ɺॏෳෆ͕ੜ͡Δ • → ஶ໊ͳจݙΛϕʔεʹɺࡉ͔͘ɺཏతͳఆٛΛࢦ͢ 33
TTXϝτϦΫεఆٛͷྲྀΕ 1. ϕετϓϥΫςΟεΛֶͿ 2. ΠϯγσϯτεςʔλεΛఆٛ͢Δ 3. ΠϯγσϯτϚΠϧετʔϯ(εςʔλεͷڥ)Λఆٛ͢Δ 4. TTXϝτϦΫεΛఆٛ͢Δ 34
ϕετϓϥΫςΟεΛֶͿ 35
େ·͔ʹεςʔλεΛఆٛ͢Δ 36
37
38
ϚΠϧετʔϯΛͱʹ TTXʹམͱ͠ࠐΉ 39
40
ίϥϜ: ϝτϦΫεऩू͍ͨΜ • ࡉ͔ͳϝτϦΫεΛఆٛ͢ΔͱɺϚΠϧετʔϯΛ͑Δ͝ͱ ʹλΠϜελϯϓΛه͢Δඞཁ͕͋Δ • ରԠதʹ͍͍ͪͪਓ͕ؒଧࠁ͢Δͷඇݱ࣮త • → WaroomͰࣗಈऩू͍ͯ͠·͢
41
ରԠதͷΠϕϯτΛτϦΨʔʹࣗಈऩू͢Δྫ ϚΠϧετʔϯ ରԠதͷΠϕϯτ Detectedʢݕʣ Ξϥʔτൃੜ௨ Acknowledgedʢೝʣ νϟϯωϧ࡞ɺΠϯγσϯτىථ Iden.fiedʢղܾࡦͷಛఆʣ RunbookͷϑΣʔζ͚ʢPrecheck ͱResolu.onʣ
Recoveredʢ෮چʣ SlackͷΓͱΓ͔ΒAI͕அ͢Δ 42
3. TTXϝτϦΫεͷ׆༻ 43
ϝτϦΫεΛޮՌతʹ͏ͨΊʹ ੳͷతͱϝτϦΫεͷಛΛ߹ͤ͞Δ 44
45
ϝτϦΫεͱվળࢪࡦͷྫ TTX ՝ վળࢪࡦ TTDetectʢݕʣ ൃੜ͔ͯ͠Βݕ·Ͱʹ࣌ ͕͔͔ؒΔ ϞχλϦϯάͷվળ TTEngageʢνʔϜߏʣ ରԠνʔϜΛߏஙʹ͕࣌ؒ
͔͔Δ γϑτׂͷ໌֬ԽɺΦ ϯίʔϧ੍ͷಋೖ TTInves-gateʢௐࠪʣ োΓ͚ʹ͕͔͔࣌ؒ Δ RunbookͷμογϡϘʔυͷ උ TTFixʢम෮ʣ োͷम෮ʹ͕͔͔࣌ؒΔ ϩʔϧόοΫͷߴԽ 46
47
യવͱͨ͠ԾઆΛͱʹɺ͔Β՝Λݟ͚ͭΔ Ծઆ ৽ͨʹൃݟͨ͠՝ͷྫ ࣾͰੜ͡ΔΠϯγσϯτͰ͋ ΕTTXͷҰఆͷͣ αʔϏενʔϜʹΑͬͯύϑ ΥʔϚϯε͕ҟͳΔ ֤TTXఆʹ͍ۙͣ ʢex. TTAͳΒ10Ҏ͘Β
͍ʣ ʢ࣮ʣணख͕શମతʹ͍ɺ ղܾࡦͷಛఆ͕શମతʹ͍ 48
49
50
4. ൃలతͳϝτϦΫε 51
αʔϏε෮چҎ֎ʹॏཁͳ͜ͱ • ͜Ε·ͰΈ͖ͯͨTTXϝτϦΫεγεςϜ෮چʹয͕͋ͨͬ ͍ͯΔ • ࣮ࡍͷΠϯγσϯτରԠ γεςϜ͚ͩͰͳ͘ɺਓʹྀ͢ Δඞཁ͕͋Δ • ސ٬ରԠࣄۀӡӦ؍ͷϝτϦΫεΛ׆༻͢Δ͜ͱͰɺΤ
ϯδχΞҎ֎ͷϝϯόʔؚΊͨ৫తͳରԠͷ࣮ݱ͕ۙͮ ͘ 52
ൃలͳϝτϦΫεͷྫ ސ٬ରԠࠜຊରࡦʹযΛͯɺ͞·͟·ͳϩʔϧΛר͖ࠐΈɺ৫తͳΠϯγσϯτରԠΛՃͤ͞ Δ ϝτϦΫε໊ λʔήοτϩʔϧ త Incident Response Metrics Engineer
७ਮͳ෮چରԠͷ՝ಛఆɾվળ ࢦඪ Customer Reliability Metrics Sales, CRE ސ٬ରԠͷ՝ಛఆɾվળࢦඪ Learning Metrics Maneger, Engineer ৫ֶ͕ͼΛಘΔ·Ͱͷ׆ಈͷτ ϥοΩϯά Improvement Metrics Maneger, Engineer ࠜຊରࡦͷ࣮ࢪঢ়گͷੳ 53
·ͱΊ ҎԼͷ5Λ͓͑͠·ͨ͠ɻෆ໌͕͋Γ·ͨ͠ΒɺAsk the Speaker͓ӽ͍ͩ͘͠͞ʂ 1. MTTRվળࢦඪͱཱͯͨ͠ͳ͍ • ཧ༝: Πϯγσϯτσʔλͷมಈੑ͕ߴ͍͔Β 2.
ϝτϦΫε׆༻ɺతʙσʔλੳʹࢸΔ·Ͱͷ߹ੑ͕ॏཁ 3. มಈੑΛ͑ΔͨΊʹɺ͍ͷ۩ମԽͱϝτϦΫεͷࡉԽ͕ॏཁ 4. Waroomʹ͓͚ΔTTXϝτϦΫεͷఆٛաఔͱ׆༻ํ๏ 5. αʔϏε෮چҎ֎ʹॏཁͳϝτϦΫε 54
͍͞͝ʹ • ϝτϦΫεͷࣗಈऩूͷ͔͚͠Λ࡞Δ ͷ͍ͨΜ • ͞ΒʹɺՄࢹԽج൫ͷߏங͍ͨΜ • ͞ΒʹɺݪҼΧςΰϦҙϥϕϧΛ ͱʹ෦நग़͢Δͷ͍ͨΜ •
→ ͥͻ Waroom Λ͝׆༻͍ͩ͘͞ • ڵຯ͕༙͍ͨํ Topotal ͷϒʔε ͥͻ͓ӽ͍ͩ͘͠͞ 55
͋Γ͕ͱ͏͍͟͝·ͨ͠
More Decks by Narimichi Takamura
See All by Narimichi Takamura
SRE本の知られざる名シーン / The Hidden Gems of Google SRE Book
nari_ex
2
810
AIをフル活用してオンコール機能のプロトタイプを2日で作った話 / Building an AI-Powered On-Call Prototype in Just Two Days
nari_ex
1
340
小さくはじめるSLI/SLO ~育てながら組織に定着させる実践知~ / Starting Small with SLI/SLOs: Building Adoption Through Continuous Growth
nari_ex
9
2.8k
Observability — Extending Into Incident Response
nari_ex
2
1.5k
組織的なインシデント対応を目指して〜成熟度評価と改善のステップ〜 / Towards an Organized Incident Response - Maturity Assessment and Improvement Steps -
nari_ex
7
11k
Waroomの開発モチベーションと今後のロードマップ / Waroom development motivation and roadmap
nari_ex
1
2k
Engineering with Business Impact
nari_ex
2
400
How We Foster Reliability in Diversity
nari_ex
14
13k
SRE Practices in Organizations
nari_ex
16
12k
Other Decks in Technology
See All in Technology
閉域網でのAPI公開の選択肢全部比較!
takano0131
4
560
アラートを見直していたら営業組織への理解にちょっとだけ繋がった話
z63d
1
210
農家は秘伝のタレを知らない - NOBODY KNOWS THE HIDDEN SOURCE - Kaggriculture 3rd Place Solution
masspeaks
2
530
Agent Routerで作るマルチテナント 推論基盤 #CNDW2026 / Building Multi-Tenant Inference Platform with Agent Router #CNDW2026
everpeace
3
270
MCPでネットワーク障害調査を自動化する前に直すべき3つのこと
yama3133
1
620
速くなるほど、深く考える AI時代のQAに学ぶ
kobayashiyorimitsu
0
260
異夢同船〜異なる夢でも同じ船でなんとかする〜
fortegp05
1
520
目を覚ませ!僕らの認証情報が何者かに侵略されてるぞ
tkito
2
460
3文字略語に騙されない! 世界一わかりやすい『EDR』超入門
kamadamakoto
1
270
2026-11-16 JAWS-UG コンテナ支部 #31 ゆるLT大会 ECR のライフサイクルポリシーを いい感じに設定したい / ECR Lifecycle policy
masasuzu
2
520
自動計装なしでECSサービスをApplication Signalsに載せるまで
ota1022
2
400
Sonnetだけで業務はまわる?
kintotechdev
1
150
Featured
See All Featured
Being A Developer After 40
akosma
PRO
92
590k
Lightning talk: Run Django tests with GitHub Actions
sabderemane
0
320
Kristin Tynski - Automating Marketing Tasks With AI
techseoconnect
PRO
0
590
Rebuilding a faster, lazier Slack
samanthasiow
85
9.8k
The Art of Programming - Codeland 2020
erikaheidi
57
15k
New Earth Scene 8
popppiees
5
2.9k
Reflections from 52 weeks, 52 projects
jeffersonlam
356
22k
Art, The Web, and Tiny UX
lynnandtonic
304
22k
Building AI with AI
inesmontani
PRO
1
1.4k
Unsuck your backbone
ammeep
672
58k
How to train your dragon (web standard)
notwaldorf
96
6.9k
Build your cross-platform service in a week with App Engine
jlugia
234
19k