#45 “How to Fight Production Incidents? An Empirical Study on a Large-scale Cloud Service”

How to Fight Production Incidents?
 An Empirical Study on a Large-scale Cloud Service

SoCC '22 (Symposium on Cloud Computing, SIGMOD: Management of Data)
https://dl.acm.org/doi/10.1145/3542929.3563482
https://www.microsoft.com/en-us/research/publication/how-to-fight-production-incidents-an-empirical-study-on-a-large-scale-cloud-service/

Transcript

1. Research Paper Introduction #45 “How to Fight Production Incidents?
 An Empirical Study on a Large-scale Cloud Service” ௨ࢉ#115 @cafenero_777 2022/04/20 1
2. Agenda •ର৅࿦จ •֓ཁͱಡ΋͏ͱͨ͠ཧ༝ 1. INTRODUCTION 2. METHODOLOGY 3. WHAT CAUSES INCIDENTS AND HOW WERE THEY MITIGATED? 4. WHAT CAUSES DELAY IN RESPONSE? 5. LESSONS LEARNT FOR RESILIENCY 6. MULTI-DEMENSIONAL INCIDENT ANALYSIS 7. RELATED WORK 8. CONCLUSION 2
3. ର৅࿦จ •How to Fight Production Incidents? An Empirical Study on a Large-scale Cloud Service • Supriyo Ghosh, et al • Microsoft • SoCC '22 (Symposium on Cloud Computing, SIGMOD: Management of Data) • https://dl.acm.org/doi/10.1145/3542929.3563482 • https://www.microsoft.com/en-us/research/publication/how-to- fi ght-production- incidents-an-empirical-study-on-a-large-scale-cloud-service/ 3
4. ֓ཁͱಡ΋͏ͱͨ͠ཧ༝ •֓ཁ • ڊେαʔϏεͷΠϯγσϯτ͸ඃ֐ਙେ͕ͩɺطଘٕज़Ͱ͸ରࡦ͕ࠔ೉ • Microsoft-TeamsͷΠϯγσϯτͱpostmortemΛௐࠪ • ͳͥൃੜʁରࡦ஗Ε͸ͳͥʁ༗ޮͳࣗಈԽͱ͸ʁ • ݕग़ɺࠜຊݪҼɺ؇࿨ͷ૬ؔͱଟ࣍ݩղੳɺࣗಈԽɾςετͷࢦඪΛڞ༗ •ಡ΋͏ͱͨ͠ཧ༝ • Ops-meetupͰ࿩୊ʹͳ͍ͬͯͨͷͰɻ • ӡ༻ܥͷ΋ͷ͕ಡΈ͍ͨؾ෼ 4
5. 1. INTRODUCTION 1/2 •େن໛αʔϏε͸ো֐͕ආ͚ΒΕͳ͍ • ࠜຊݪҼͷཧղɾ໰୊मਖ਼ɺӨڹ࣌ؒͷ୹ॖ͸΍͍ͬͯΔɺ͕ɻɻɻ •Microsoft-Teams: ௚ۙ1೥ɺ152݅ΠϯγσϯτΛௐࠪ • Azure IaaS্ͷweb scaleͳ෼ࢄαʔϏε • ֤regionʹ·͕ͨΔNW, ετϨʔδɺೝূΛ࢖͏ɿϝοηɺ௨࿩ɺձٞɺϦΞϧλΠϜ௨৴ͷͨΊ৴པੑॏཁ • ݕ஌ɺࠜຊݪҼɺ؇࿨: ൒෼͸ݕ஌Ͱ͖ͨʢ؆୯ͳվળͰݕ஌཰޲্ʣ • ࠜຊݪҼ: ໿൒෼͸ιʔείʔυόάͰ͸ͳ͍ • ༰ྔ໰୊ɺखಈσϓϩΠΤϥʔɺূ໌ॻ੾Ε • ؇࿨: 9ׂ͸όάमਖ਼Ҏ֎ʢϩʔϧόοΫɺߏ੒ɾઃఆมߋɺpod࠶ىಈͳͲʣͰ௚Δ • ؇࿨ରࡦͷ஗Εɿਓ͕བྷΉ͜ͱ͕ओͳݪҼʢσϓϩΠಋೖڐՄ͕߱Γͳ͍ɺจॻɾཧղෆ଍ɺਓؒͷख࡞ۀʣ 5
6. 1. INTRODUCTION 2/2 •ຊ࿦จͷ෼ੳ݁Ռ • όά΍֎෦ґଘΠϯγσϯτ͸؂ࢹෆे෼ɺݕ஌࣌ؒ௕͍ • ؂ࢹ͚ͩͰ͸ͳ͘in-situௐࠪ΋ॏཁʢo11y͕ॏཁʣ • ϩʔϧόοΫ΍ઃఆमਖ਼Ͱ͙͢ʹ௚Δ΋ͷ΋ଟ͍ɻ͜ͷ൑அΛπʔϧͰ ਝ଎ʹߦ͏ͱྑ͍ • ຊ࣭తʹ؂ࢹࠔ೉ͳ΋ͷ΋͋ΔɻࣄલͷςετΛ΋ͬͱ΍Δ΂͖ 6
7. 2. METHODOLOGY •ΠϯγσϯτDB͔Βબ୒ • 2021/05/15 - 2022/05/15ʹى͖ͨTeamsͷॏେΠϯγσϯτ152݅ • ෳ਺ςφϯτ΍ސ٬ʹӨڹ͠ʢͨՄೳੑ͕͋Γʣɺ͢Ͱʹղܾɾ؇࿨͠ɺͦͷํ๏͕׬શʹΘ͔͍ͬͯΔ ΋ͷɻશମΠϯγσϯτͷ2% •ཁҼΧςΰϦ෼͚ • ݕ஌ɾݪҼ෼ੳɾ؇࿨ͷ࣮ϓϩηεΛߟྀ͠ɺ6ͭͷཁҼͰղੳ • Teamsݻ༗ͷಛੑʹ஫ҙ • MSݻ༗ͷࣗಈπʔϧ͕Πϯγσϯτ؇࿨͍ͯ͠Δ • ࠶ݱੑෆ໌ͳΠϯγσϯτΛআ֎͍ͯ͠Δ 7
8. Part1 •֤छཁҼͷΧςΰϦ෼͚ • RCA෼ྨฤ • ؇࿨ରࡦฤ • ෮چ࣌ؒͰൺֱ • ݕ஌஗Εͷཧ༝ • ؇࿨஗Εͷཧ༝ • ࣗಈԽʹΑΔରࡦఏҊ/ֶΜͩ͜ͱ 8
9. 3. WHAT CAUSES INCIDENTS AND HOW WERE THEY MITIGATED? RCA෼ྨฤ •7ͭʹ෼ྨɿ4ׂ͸Code/con fi gόά͕ͩɺ6ׂ͸σϓϩΠ΍Πϯϑϥ͕ݪҼʂ • όάɿfeature fl agґଘ΍ಛఆػೳ͚ͩະαϙʔτͷͨΊʹى͜Δόά 25%ɺϑϥά΍ఆ਺ʢ͖͍͠஋΋ؚΉʣ25%ɺίʔυґଘ 20%ɺtype/validation/exception 17%, ޿ใޓ׵ੑ 15% • ΠϯϑϥɿCPUΩϟύ 33%, NWΩϟύ 40%, scaling໰୊ʢclusterͷҰ෦ར༻ʣ16%, ϝϯς࣌ͷΩϟογϡ࡟আىҼ 8% • σϓϩΠɿূ໌ॻ؅ཧ 55%, ޡͬͨύον 25%, खΦϖϛε 20% • ίϯϑΟάɿϛε 47%,ʢ৻ॏͳ෼ੳ͕ඞཁͳʣมߋ 42%, ߏ੒ಉظ 10% • ґଘؔ܎ɿόʔδϣϯඇޓ׵ 25%, ґଘαʔϏεͷ݈શੑ 20%, ֎෦ίʔυͷมߋ 28%ɺػೳґଘ 28% • DB/NW: ஗Ԇ 25%, Մ༻ੑɾ઀ଓੑ 31%, DBϑΝΠϧૢ࡞ 25%, ϦΫΤετෆՄ 19% • ೝূɿݖݶෆ଍ 42%, ূ໌ॻϩʔςʔγϣϯ 28%, ϙϦγʔมߋىҼͷೝূΤϥʔ 28% 9
10. 3. WHAT CAUSES INCIDENTS AND HOW WERE THEY MITIGATED? ؇࿨ରࡦฤ •؇࿨ࡦ: 40%͸ίʔυ΍ߏ੒όά͕ͩɺ80%͸ͦΕΒͷमਖ਼ແ͠ͰҰ࣍ରԠ • ϩʔϧόοΫɿίʔυ 35%, ߏ੒ 24%, Ϗϧυ 41% • Πϯϑϥมߋ: ผϊʔυʹಀ͕͢ 16%, ผΫϥελ΁ 9%, ผrigion΁ 19%, ੑೳ্͛Δ 31% ෆཁϊʔυ࡟আ 10%, ࠶ىಈ 15% • ֎෦मਖ਼: ϩʔϧόοΫ 29%, ίʔυ΍ઃఆͷमਖ਼ 17%, ͦͷଞશͯ 54% • ઃఆमਖ਼ɿઃఆमਖ਼ 20%, ઃఆ໭͠ 25%, 25% (৽ػೳແޮԽɺػೳ໭͠ɺػೳϑΣΠϧΦʔόʔʣ • ίʔυमਖ਼: σόοά 42%, ϚδοΫφϯόʔ 17%, ྫ֎ॲཧ 25%, Ϟδϡʔϧमਖ਼ 17% • ΞυϗοΫରԠ: ΄΅खಈରԠʹ͍ۙ΋ͷɻূ໌ॻ΍伴ͷߋ৽ͳͲ • Transient: ʢNWͳͲͷʣࣗಈ෮چɺΞοϓσʔτ, ΞϓϦଆͷҰ࣌తͳෛՙͳͲ 10 40%͸ϩʔϧόοΫ΍ϑΣΠϧ Φʔόʔɺ࠶ىಈʢʂʣ ֎෦γεςϜͷमਖ਼಺༁: ίʔυ΍ઃఆͷमਖ਼͸17%͔͠ͳ͍
11. 4. WHAT CAUSES DELAY IN RESPONSE? ෮چ࣌ؒͰൺֱ •Πϯγσϯτൃੜ͔Βऴྃ·Ͱͷ࣌ؒ = Time To Detect + Time To Mitigate 11 ґଘ΍όά: TDD, TTM௕͍ɺͭ·Γ؂ࢹݕ஌͸೉͍͠ Auth, DB/NW͸TDD୹͍͕, TTM௕Ί Deploy: TDD > TTM ݪҼʹ஫໨ͨ͠৔߹ ؇࿨ํ๏ʹ஫໨ͨ͠৔߹ खಈमਖ਼ΑΓϩʔϧόοΫ͕߹ཧతʂ Ad-hoc/Code fi x: खಈͷͨΊTTM௕͍ ϩʔϧόοΫ:TTM୹͍
12. 4. WHAT CAUSES DELAY IN RESPONSE? ݕ஌஗Εͷཧ༝ •55%͸watchdogʢlatency, CPU, memory usageᮢ஋௒͑ʣͰࣗಈݕ஌ •45%͸֎෦ใࠂ: 30%͸ސ٬, 10%͸MS਎಺, ࢒Γ͸ࣗνʔϜ •watchdogͰݕ஌Ͱ͖ͳ͔ͬͨཧ༝ • Monitor Bug: ͖͍͠஋͕ߴ͗͢ 25%ɺseverityೝࣝϛε 25%ɺઃఆϛε΍ݕ஌ෆՄ 25% • Telemetry Coverage: Ϋϥ΢υ؀ڥґଘͳσʔλෆ଍ 31%, ಛఆγφϦΦ(HTTPίʔυͳͲ)ͷෆ଍ 31% • External E ff ect: ಛఆ֎෦αʔϏεͷ؂ࢹෆ଍ • No Monitor: ؂ࢹ࿙Ε 12 ؂ࢹͰ͖͍ͯͳ͍΋ͷTTD௕͍ʢؾ͖ͮʹ͍͘ʣ 17%͸؂ࢹ࿙Εɺσʔλ࿙Ε
13. 4. WHAT CAUSES DELAY IN RESPONSE? ؇࿨஗Εͷཧ༝ •Doc-Procedures: ஌ࣝෆ଍ 20%, Doc඼࣭ 50%, ಛఆ؀ڥิ଍ෆඋ 30% •Deployment Delay: खಈঝೝ 25%, ΞΫηεݖݶෆ଍ 25%, ൓өΏͬ͘Γ 50% •Manual E ff ort/External Dependency: ରԠ࿈ܞෆ଍ 14%, ޡ਍ 30%, •Complex Root Cause: ൃੜස౓௿͍΋ͷ 18%, ϝτϦΫεෆ଍ 27%, σόοά࡞ۀ 55% 13 ʢ࣌ؒͷ͔͔ΔσόοάͳͲͰʣݪҼ͕෼͔ͬͯ΋ɺ
 ʢखॱॻෆ଍΍σϓϩΠ൓ө଴ͪͰʣҰ࣍ରԠ͕஗ΕΔɻ
 ͜Ε͕30%΋͋Δɻ
14. 5. LESSONS LEARNT FOR RESILIENCY ࣗಈԽʹΑΔରࡦఏҊ •ࣗಈԽʹΑΔରࡦఏҊ • Manual/Con fi g Test: perf-test, γφϦΦςετɺvalidation/integration/unit test • Auto Alert/Triage: ͖͍͠஋ͷࣗಈԽɺࣗಈΤεΧϨʔγϣϯ • Auto Deploy: ϑΣΠϧΦʔόʔ΍ϦϦʔεͷࣗಈԽ • Unclear/None: طʹࣗಈԽࡁΈɻࣗಈͰ͸ରॲͰ͖ͳ͍ɻ 14
15. 5. LESSONS LEARNT FOR RESILIENCY ֶΜͩ͜ͱ •Improve Monitoring/Testing: ςϨϝτϦ௥Ճɻޓ׵ੑςετ௥Ճ •Behavioral Change: σϓϩΠ͸஫ҙਂ͘΍ͬͯʂϑϥάon/o f ͸γφϦΦςετ΍ͬͯʂࣄલͷݖݶ֬ೝ΍ͬͯʂ •Doc/Training: υΩϡϝϯτվળɺࣄޙৼΓฦΓɺAPIυΩϡϝϯτ֬ೝͱվળ •Auto Mitigation: ূ໌ॻߋ৽ɺ؂ࠪɺϑΣΠϧΦʔόʔɺϊʔυεέʔϧΞ΢τ •External Coordination: ࣄલʹ࿈བྷνϟωϧ࡞͓ͬͯ͘ɺΤεΧϨʔγϣϯʢ࿈བྷʣͷઃܭ 15 20%͸ؒ઀తͳ͜ͱɻ ʢυΩϡϝϯτɾτϨʔχϯάɾ֤छ࣮ફʣ
16. Part2 •ଟ࣍ݩݕఆ • ؂ࢹࣦഊͱRC • RCͱ؇࿨ࡦ • ؇࿨ࡦͱֶΜͩ͜ͱ 16
17. 6. MULTI-DIMENSIONAL INCIDENT ANALYSIS ؂ࢹࣦഊͱRC •୯७ͳ෼ੳͰ͸ࣄ৅ͷҰ෦͔͠ݟ͍͑ͯͳ͍ʁ • ྫ: ύϥϝλͷޓ׵ੑόάɻΞϥʔτ͸ى͖ͣɺϩʔϧόοΫ Ͱ͸ͳ͘hot fi xͯ͠͠·ͬͨɻ • ૬ؔݕఆͰΠϯγσϯτཧղΛࢼΈΔ • ؂ࢹ͕ແ͠/ΧόϨοδ௿: ίʔυόά͕େ෦෼ • "͔ͬ͠Γͱ"ςετ͢Δ͜ͱ͕ෆՄܽ • ݕ஌ෆՄ: ґଘ֎෦αʔϏεͷো֐ 17 Not failedͱͷൺֱΛݟͯΈΔͱ໘ന͍ɻ όάΛ؂ࢹͰݟ͚ͭΔͷ͸΄΅ແཧͩͱࢥͬͯྑ͍ɻ
18. 6. MULTI-DIMENSIONAL INCIDENT ANALYSIS RCͱ؇࿨ࡦ •Rollback͸Ұൠతͳ؇࿨ࡦ • Con fi g BugͰ͢ΒFix (21%)ΑΓ࣌୹ͰRollback (47%) • con fi gͷߏ੒ςετͰճආͰ͖ΔՄೳੑ •Deployment Errorͷେ൒͸ূ໌ॻΤϥʔʢCon fi g FixͰରԠʣ • ༗ޮظݶνΣοΫΛపఈ͢Δ •DB/NWো֐ • ʢ͍͍ҙຯͰʣ֎෦ཁҼɻ৑௕ԽΛਪਐͯ͠΋Β͏ 18
19. 6. MULTI-DIMENSIONAL INCIDENT ANALYSIS ؇࿨ࡦͱֶΜͩ͜ͱ •21%͸υΩϡϝϯτɾରԠτϨʔχϯάͰվળ • ඼࣭ʢಡΈ΍͢͞ɺTTM, ࠷৽৘ใʣ • ࣗಈԽ: ϫʔΫϑϩʔ࡞Γ •खॱॻͰରࡦ: ؂ࢹʢΧόϨοδʣෆ଍ͷิڧ •ख࡞ۀʹΑΔTTM௕ظԽ • ࣗಈԽͰ࣌୹ʢূ໌ॻߋ৽ɺfailover, auto-scalingʣ • ϞχλϦϯάͱϫʔΫϑϩʔͷηοτԽ 19
20. 8. CONCLUSION •େن໛αʔϏε (Microsoft Teams)ͷแׅతͳো֐෼ੳ • 152݅Πϯγσϯτͷ෼ੳɺ؇࿨ࡦɺࠜຊରࡦͰͷΧςΰϦ෼͚ • ଟ࣍ݩ෼ੳ݁ՌʹΑΔରࡦͷఏҊ 20
21. EoP 21