Research Paper Introduction #65 "Chaos engineering: Building confidence in system behavior through experiments"

Blog
https://www.oreilly.com/content/chaos-engineering

Ebook
https://learning.oreilly.com/library/view/chaos-engineering/9781491988459/

Transcript

1. Research Paper Introduction #18 “Chaos engineering: Building con fi dence in system behavior through experiments” ௨ࢉ#65 @cafenero_777 2021/01/28
2. Agenda • ର৅࿦จ • ֓ཁͱಡ΋͏ͱͨ͠ཧ༝ 1. Part I. IntroductionɿͳΜͰChaos EngineeringΛ΍ΔΜ͚ͩͬʁ 2. Part II. The Principles of ChaosɿChaos EngineeringͬͯͲ͏͍͏΋ͷʁ 3. Part III. Chaos In Practiceɿ࣮ࡍʹChaos EngineeringͬͯͲ͏΍Δͷʁ
3. $ which • Chaos engineering: Building con fi dence in system behavior through experiments • Casey Rosenthal, Lorin Hochstein, Aaron Blohowiak, Nora Jones, and Ali Basiri • Net fl ix, Inc. 2017 • Report? book? • https://learning.oreilly.com/library/view/chaos-engineering/9781491988459/ • Blog • https://www.oreilly.com/content/chaos-engineering
4. ֓ཁͱಡ΋͏ͱͨ͠ཧ༝ • ֓ཁ • γεςϜ͸ඞͣ”མͪΔ”͕ɺෳࡶͳγεςϜΛ੩తʹݪҼಛఆ͢Δͷ͸೉͍͠ • ຊ൪؀ڥͰ͏·࣮͘ݧΛͯ͠ʢChaos EngineeringΛͯ͠ʣɺݪҼಛఆɾγεςϜվળʹܨ͛Δ • Net fl ixͰͷ10೥͙Β͍ͷCEܦݧଇΛڞ༗ • ಡ΋͏ͱͨ͠ཧ༝ • ࣄނΛݮΒ͍ͨ͠ • Chaos Engineering͕ྑͦ͞͏ɻ໘നͦ͏ͩ͠ӡ༻஌ݟ΋ར༻Ͱ͖ͦ͏ • ҎલಡΜͩ΋ͷ͸Ԡ༻ฤʢϑϨʔϜϫʔΫدΓͷ࿩ͩͬͨʣͩͬͨɻجૅฤΛ஌Γ͍ͨ • લճɿAutomating chaos experiments in production https://arxiv.org/abs/1905.04648
5. PRINCIPLES OF CHAOS ENGINEERING https://principlesofchaos.org/
6. Net fl ixͷʢࢲͷཧղ͍ͯ͠ΔʣCEྺ࢙ • 2008ʹAWSશ໘ҠߦʢCDNҎ֎ʣ • Chaos Monkey: VMࢭΊΔ • 2012೥ΫϦεϚεΠϒʹ୯ҰϦʔδϣϯͰେো֐ • Chaos Kong: RegionࢭΊΔ • յ͢͜ͱ͕CEͱ͍͏ޡղ͕ൃੜɻ৭ʑߟ͑Δ • 2016: ChAP (ChaosAutomation Platform), ϚΠΫϩαʔϏεʹಋೖ • 2017: PRINCIPLES OF CHAOS ENGINEERINGΛެ։ • 2019: Automating chaos experiments in production
7. Net fl ixͷʢࢲͷཧղ͍ͯ͠ΔʣCEྺ࢙ • 2008ʹAWSશ໘ҠߦʢCDNҎ֎ʣ • Chaos Monkey: VMࢭΊΔ • 2012೥ΫϦεϚεΠϒʹ୯ҰϦʔδϣϯͰେো֐ • Chaos Kong: RegionࢭΊΔ • յ͢͜ͱ͕CEͱ͍͏ޡղ͕ൃੜɻ৭ʑߟ͑Δ • 2016: ChAP (ChaosAutomation Platform), ϚΠΫϩαʔϏεʹಋೖ • 2017: PRINCIPLES OF CHAOS ENGINEERINGΛެ։ • 2019: Automating chaos experiments in production ࠓ೔ͷ࿩͸͜ͷลΓ
8. Part I. Introduction • ͳΜͰChaos EngineeringΛ΍ΔΜ͚ͩͬʁ
9. Why Do Chaos Engineering? (1/2) • ෼ࢄγεςϜͰ͸༧ظ͠ʹ͍͘Πϕϯτ͕ൃੜ • ίϯϙʔωϯτଟ͍ɺ૬ޓ࡞༻͢Δ • disk/NW failure, ΞΫηεٸ૿ͰαʔϏεϨϕϧྼԽ • ى͖Δ”લ”ʹγεςϜͷଟ͘ͷऑ఺Λಛఆ͍ͨ͠ • ͜ͷํ๏Λମܥతʹ·ͱΊͨͷ͕ΧΦεΤϯδχΞϦϯά • @Net fl ix • 2008-, AWSҠߦҎ߱։࢝ɺVM/Regionো֐ɺΫϦεϚεো֐ɻ • Chaos Monkey/Chaos Kong, FIT (Failure Injection Test)ɺSpinnakerͱ౷߹ɻPrinciple of Chaos Engineering • Chaos Moneky:ʢաڈ5೥ͰαʔϏεӨڹ͸̍౓͚ͩʣ • Chaos Kong: ݄ʹҰ౓ൃಈɺසൟʹ໰୊ൃੜɺɺɺ2೥໨Ͱ҆ఆ ྫ: Clos NW, ෼ࢄLB, ෼ࢄStorage, IaaS/PaaS/CaaS on k8s
10. Why Do Chaos Engineering? (2/2) • ܦݧతௐࠪͷ෼໺ʹద༻ɺγεςϜΛཧղ͢ΔͨΊʹ࣮ݧ͢Δ • γεςϜఀࢭͷՄೳੑ͕͋Δऑ఺ΛֶͿ • ࠶ൃ๷ࢭࡦ(post motem)ͱͯ͠ࣄޙʹ΍ΔͷͰ͸ͳ͘ɺࣄલʹ΍Δ(proactive) • طଘͷςετͱඃΔࣄ͸ଟ͍ɿݒ೦ࣄ߲ɾπʔϧ౳ • ςετ: ط஌ͷಛఆ৚݅ΛΫϦΞ͢Δ͔൱͔ɻyes or no ʢྫɿ֤Ξαʔγϣϯʣ • CE: ৽͍͠৘ใΛੜΈग़͢ɻྫɿNW஗ԆɾϥϯμϜʹྫ֎ൃੜɾλΠϜτϥϕϧɾI/O΍CPUߴෛՙ • લఏ৚݅ • ໌Β͔ʹऑ఺͕෼͔͍ͬͯΔ৔߹͸·ͣ͸௚͢ɻͦͷޙʹCEͯ͠ؾ͍͍ͮͯͳ͔ͬͨऑ఺͕໌Β͔ʹ͢Δ • ؂ࢹγεςϜɿఆৗঢ়ଶͱ൑அͰ͖ΔΑ͏ʹՄࢹԽ
11. Managing Complexity • ύϑΥʔϚϯεɾՄ༻ੑɾϑΥʔϧττϨϥϯε + ։ൃ଎౓͕ඞཁ • ϚΠΫϩαʔϏεΛ࠾༻ͯ͠ɺ֤νʔϜಠཱʹ։ൃɾӡ༻ • ਓؒͷཧղΛ٘ਜ਼ʹͯ͠଎౓ɾॊೈੑΛ୲อɻ • ࣄલͷςετͰ͸ݟ͚ͭͮΒ͍ • ط஌ͷΞαʔγϣϯ͚ͩͰ͸Χόʔ͖͠Εͳ͍ • ྫɿΩϡʔɺΦʔτεέʔϧɺϩʔΧϧΩϟογϡɾεέʔϧμ΢ϯɻ
 ɹɹෛՙͷฏۉ஋͚ͩͰे෼ʁʁ • ϒϧ΢ΟοϓޮՌ • ݸʑͷϚΠΫϩαʔϏε͸߹ཧత͕ͩɺ࠷ऴӨڹ͕༧ظͰ͖ͳ͍৔߹͕͋Δ get meta-data get meta-data map/reduce Ϩεϙϯε ϦΫΤετ ิ଍ɿ࠷ۙྲྀߦΓͷΞϨ΋ಉ͡? ౴: DeepLearning
12. Part II. The Principles of Chaos • Chaos EngineeringͬͯͲ͏͍͏΋ͷʁ
13. Part II. The Principles of Chaos • ΧΦεΤϯδχΞϦϯά • ϥϯμϜʢແܭըʣͰ͸ͳ͍ɻ࢓ࣄͷΧΦεΛ༠ൃ͢Δ͜ͱ΋ͳ͍ɻ • ίϯϙʔωϯτ૬ޓ࡞༻ͷ݁ՌɺγεςϜ͕ΧΦεʹͳΔՄೳੑΛཧղ͢Δ͜ͱ • গ͚ͩ͠γεςϜʹΧΦεΛೖΕͯɺγεςϜ͕Ͳ͏ػೳ͢Δ͔ʁ • ఆৗঢ়ଶͷԾઆΛཱͯΔ • ࣮ੈքͷΠϕϯτΛมߋ͢Δ • prodͰ࣮ݧ͢Δ • ࣮ݧΛࣗಈԽͯ͠ܧଓతʹߦ͏ • Bast RadiusʢӨڹൣғʣΛ࠷খԽ͓ͯ͘͠
14. Hypothesize about Steady State (1/2) • ఆৗঢ়ଶͱ͸ʁʹϏδωεϩδοΫͷظ଴஋΍SLA • ֤छKPI (޿ࠂදࣔճ਺ɺΧʔτ௥Ճɺ໰͍߹Θͤྔ) • ϝτϦΫεɾΞϥʔτ؂ࢹ • CPU/Memory/NW, I/O, RPS, duration • εϧʔϓοτ, Τϥʔ཰, 99%ile • ࠶ੜϘλϯΛԡ͢଎౓: SPS • ΫϥΠΞϯτଆͱҰॹʹݕূͰ͖ΔͱΑΓྑ͍ɻ • पظతʹϝτϦΫεΛݟΔඞཁ͕͋Δ৔߹΋ɻ
15. Hypothesize about Steady State (2/2) • ԾઆΛཱͯΔඞཁ͋Γ • Ծઆ͕ແ͍ͱɺͲͷϝτϦΫεʹ஫໨͢΂͖͔͑͞෼͔Βͳ͍ɻ • ʮ͋ΔCE࣮ݧΛͯ͠΋҆ఆঢ়ଶͷ··Ͱ͢ɻʯ͕Ծઆɻ • ྫɿpersonalized listΛࣦഊ-> ௨ৗͷlistΛදࣔͤ͞ɺSPS (start/sec)ʹӨڹΛ༩͑ͳ͍ • ྫɿτϥϑΟοΫΛผϦʔδϣϯʹϦμΠϨΫτͤͯ͞΋SPS҆ఆ • ͖͍͠஋ͷઃఆ͸ʁ • Ծઆ͕ࣝผͰ͖Δ஋ɻʢଥ౰ͩͱࢥ͏ൣғɻओ؍తͳภࠩྔʣ
16. Vary Real-World Events (1/2) • ൃੜΠϕϯτͷස౓ͱӨڹɺରॲίετɾෳࡶ͞Λݟੵ΋Δ • VMఀࢭɿසൃɾ୯७ɾίετ௿ • ϦʔδϣϯఀࢭɿෳࡶͰίετߴ • จԽతཁҼ΋͋ΔछͷίετͱΈͳ͢ • DC͸ݎ࿚ੑɾ҆ఆੑ >>> | ӽ͑ΒΕͳ͍น | >>> ढ़හੑ • ҰํͰΫϥ΢υར༻ʢHW੹೚ͷ֎෦ԽʣͰHWো֐͸౰વʹͳ͖ͬͯͨ Πϕϯτྫɿ ϋʔυ΢ΣΞো֐ ػೳతͳόά ঢ়ଶෆҰகʹΑΔૹ৴Τϥʔ ωοτϫʔΫͷ஗ԆͱύʔςΟγϣϯ ೖྗͷେ͖ͳมಈɺ͓ΑͼϦτϥΠετʔϜ Ϧιʔεͷރׇ Ϗβϯνϯো֐ɾڝ߹ঢ়ଶ
17. Vary Real-World Events (2/2) • Πϕϯτ • ߴෛՙɺ௨৴஗ԆɺNWύʔςΟγϣϯɺແޮͳূ໌ॻɺσʔλංେԽ • όάࠞೖίʔυͷσϓϩΠ • ΧφϦΞϦϦʔεͩͱখن໛͕ͩӨڹ͸ग़ͯ͠·͏ • αʔϏεݺͼग़࣌͠ʹΤϥʔΛฦͯ͠ֆϛϡʔϨͱ͢Δͱ҆શʢྫɿಛఆΫϦΞϯτ͔ΒͷAPIͷΈࣦഊʣ • ো֐υϝΠϯʢӨڹͱൣғʣΛڱ͘͢ΔΑ͏ʹCE͢Δɻ • ผͷݪҼ͕ىͨ͜͠ࣄ৅ͩͬͨͱͯ͠΋੾Γ෼͚͠΍͍͢ɻ • ϦιʔεʢαʔϏεͷάϧʔϓʣ͸ڞ༗͞Ε͍ͯΔͨΊɺશͯΛো֐υϝΠϯͱͯ͠ઢҾ͖͸Ͱ͖ͳ͍఺ʹ஫ҙɻ • ॲཧͰ͖Δͱ༧૝͞ΕΔΠϕϯτͷΈࢼ͢ʢͦͯ͠௚͢ʣɻແҙຯʹյ͢ͷ͸ແҙຯɻ
18. Run Experiments in Production • ίʔυͷਖ਼͠͞ʹՃ͑ͯγεςϜશମͷਖ਼͠͞Λ୲อ͍ͨ͠ • prodͱಉ͡ঢ়ଶΛΤϛϡʔϨʔτͯ͠devͰࢼ͢ʁ • ೖྗ(input)΍֎෦γεςϜΛΤϛϡϨʔτ͢Δ͸ݶք͕͋Δ • CIతʹιʔείʔυɾઃఆϑΝΠϧ͸౎౓มΘΔɻ • Α͋͘Δݴ͍༁ • ਓͷ໋͕௚઀͔͔ͬͯΔγεςϜ͸͠ΐ͏͕ͳ͍ɻ͕ɺ͘͝Ұ෦ͷϋζɻ • γεςϜʹճ෮ྗ͕ͳ͍ʢͬͱ͍͏ࣄΛӅ͍ͨ͠ʁʣ • ͡Ό͋Ͳ͏͢Δ͔ʁ • ͙͢ʹ࣮ݧఀࢭͰ͖ΔΑ͏ʹ͓ͯ͘͠ • Blast RadiusʢӨڹൣғʣΛ࠷খʹ͓ͯ͘͠ • ʢͲ͏ͯ͠΋prodͰͰ͖ͳ͚Ε͹ʣຊ൪؀ڥʹ͍ۙ؀ڥͰ࣮ݧ͢Δ
19. Automate Experiments to Run Continuously • ςετࣗಈ࣮ߦ • ຊ൪؀ڥ͸CI/CD͞ΕΔ -> CE΋ࣗಈԽͯ͠ܧଓతʹνΣοΫ͢Δ • ཧ૝తʹ͸ίʔυมߋͷ౓ʹ࣮ߦʢΧΦεΧφϦΞʁʣ • ࢀߟɿਓͰͳ͘γεςϜ͕ҟৗΛ൑அɾ࣮ݧఀࢭɻӨڹςϯϓϨԽɻSpinnaker͕ຊ൪ʹ࣮ݧϊʔυىಈɻ • ςετࣗಈੜ੒ • ·ͩى͖ͨ͜ͱͷͳ͍ɺى͖ͦ͏ͳΠϕϯτ΍ো֐Λಛఆͯ͠ਓ͕ؒ࡞Δɻʢཁ͸ηϯεʁʂʣ • Incident Trackerʹه࿥͞Ε͍ͯΔط஌ͷࣄ৅~ʹ࠷௿ݶ΍Δ΂͖ςετɻ ࢀߟɿLDFI: lineage-driven fault injection (ܥ౷ۦಈܕϑΥʔϧτΠϯδΣΫγϣϯ) Automating Failure Testing Research at Internet Scale, https://dl.acm.org/doi/10.1145/2987550.2987555
20. Minimize Blast Radius • νΣϧϊϒΠϦݪൃࣄނͷݪҼ͸ϝϯςதͷ”ࢼݧ”ͩͬͨɾɾɾ • ྫɿ • গ਺ͷϢʔβʢclientʣʹ࢓ֻ͚Δ • গ਺ͷϢʔβʢclientҎ֎ʣʹ࢓ֻ͚Δ • શϢʔβɺͨͩ͠ಛఆՕॴʢαʔϏεɾϩδοΫʣ΁ϧʔςΟϯά • શϢʔβɺશαʔϏε • ػೳͷfallback͸ඞਢɺۓٸఀࢭػೳ͸ඞਢʢࣗಈऴ͕͓ྃ͢͢Ίʣ • ͙͢ʹରԠͰ͖ΔΑ͏ʹ೔தଳʢϔϧϓ͕ಘΒΕΔ࣌ؒଳʣʹ࣮ݧ͢Δɻ
21. Part III. Chaos In Practice • ࣮ࡍʹChaos EngineeringͬͯͲ͏΍Δͷʁ • ΍Δ͔Ͳ͏͔໎͍ͬͯΔਓ΁ • ͲͪΒ͕ྑ͍͔ɻ • ҙਤͨ͠λΠϛϯάͰখ͞ͳো֐Λىͯͦ͜͠ΕΛ௚͢ • ҙਤ͠ͳ͍λΠϛϯάͰେ͖ͳো֐͕ى͖͔ͯΒ௚͢ • ۜߦܥɺۚ༥ܥͰ͑͞΋CEΛ࢝Ί͍ͯΔ
22. Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ
23. Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ
24. Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ Ծઆྫɿ RedisͰtimeoutͨ͠৔߹ Act/Stb failoverͨ͠৔߹
25. Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ Ծઆྫɿ RedisͰtimeoutͨ͠৔߹ Act/Stb failoverͨ͠৔߹ • BRΛ࠷খʹͯ͠ɺprod؀ڥͰߦ͏ • dev؀ڥͰͷdry-run͢Δ
26. Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ Ծઆྫɿ RedisͰtimeoutͨ͠৔߹ Act/Stb failoverͨ͠৔߹ • BRΛ࠷খʹͯ͠ɺprod؀ڥͰߦ͏ • dev؀ڥͰͷdry-run͢Δ • ”ਖ਼ৗ”ͷఆٛͱɺͦΕͱͷࠩʢҟৗʣͷࢦඪܾΊ • ۓٸఀࢭ༻ͷ͖͍͠஋ܾΊ
27. Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ Ծઆྫɿ RedisͰtimeoutͨ͠৔߹ Act/Stb failoverͨ͠৔߹ • BRΛ࠷খʹͯ͠ɺprod؀ڥͰߦ͏ • dev؀ڥͰͷdry-run͢Δ • ”ਖ਼ৗ”ͷఆٛͱɺͦΕͱͷࠩʢҟৗʣͷࢦඪܾΊ • ۓٸఀࢭ༻ͷ͖͍͠஋ܾΊ • CE࣮ߦதͰ͋Δࣄɾཧ༝ɾ಺༰ • ಛʹ࠷ॳ͸प஌ॏཁ
28. Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ Ծઆྫɿ RedisͰtimeoutͨ͠৔߹ Act/Stb failoverͨ͠৔߹ • BRΛ࠷খʹͯ͠ɺprod؀ڥͰߦ͏ • dev؀ڥͰͷdry-run͢Δ • ”ਖ਼ৗ”ͷఆٛͱɺͦΕͱͷࠩʢҟৗʣͷࢦඪܾΊ • ۓٸఀࢭ༻ͷ͖͍͠஋ܾΊ • CE࣮ߦதͰ͋Δࣄɾཧ༝ɾ಺༰ • ಛʹ࠷ॳ͸प஌ॏཁ • ϝτϦΫεΛؾʹ͠ͳ͕Β΍Δ • ֎෦γεςϜʹ΋ؾΛ഑Δ
29. Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ Ծઆྫɿ RedisͰtimeoutͨ͠৔߹ Act/Stb failoverͨ͠৔߹ • BRΛ࠷খʹͯ͠ɺprod؀ڥͰߦ͏ • dev؀ڥͰͷdry-run͢Δ • ”ਖ਼ৗ”ͷఆٛͱɺͦΕͱͷࠩʢҟৗʣͷࢦඪܾΊ • ۓٸఀࢭ༻ͷ͖͍͠஋ܾΊ • CE࣮ߦதͰ͋Δࣄɾཧ༝ɾ಺༰ • ಛʹ࠷ॳ͸प஌ॏཁ • ϝτϦΫεΛؾʹ͠ͳ͕Β΍Δ • ֎෦γεςϜʹ΋ؾΛ഑Δ • 1.ͷԾઆ͸ਖ਼͔͔ͬͨ͠ʁ • ؔ࿈νʔϜ΁ͷϑΟʔυόοΫ
30. Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ Ծઆྫɿ RedisͰtimeoutͨ͠৔߹ Act/Stb failoverͨ͠৔߹ • BRΛ࠷খʹͯ͠ɺprod؀ڥͰߦ͏ • dev؀ڥͰͷdry-run͢Δ • ”ਖ਼ৗ”ͷఆٛͱɺͦΕͱͷࠩʢҟৗʣͷࢦඪܾΊ • ۓٸఀࢭ༻ͷ͖͍͠஋ܾΊ • CE࣮ߦதͰ͋Δࣄɾཧ༝ɾ಺༰ • ಛʹ࠷ॳ͸प஌ॏཁ • ϝτϦΫεΛؾʹ͠ͳ͕Β΍Δ • ֎෦γεςϜʹ΋ؾΛ഑Δ • 1.ͷԾઆ͸ਖ਼͔͔ͬͨ͠ʁ • ؔ࿈νʔϜ΁ͷϑΟʔυόοΫ • ෼ੳ݁Ռʹࣗ৴͕࣋ͯͨΒൣғΛ޿͛Δ • ৽ͨͳӨڹ͕ग़ͯ͘ΔՄೳੑʹ஫ҙ
31. Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ Ծઆྫɿ RedisͰtimeoutͨ͠৔߹ Act/Stb failoverͨ͠৔߹ • BRΛ࠷খʹͯ͠ɺprod؀ڥͰߦ͏ • dev؀ڥͰͷdry-run͢Δ • ”ਖ਼ৗ”ͷఆٛͱɺͦΕͱͷࠩʢҟৗʣͷࢦඪܾΊ • ۓٸఀࢭ༻ͷ͖͍͠஋ܾΊ • CE࣮ߦதͰ͋Δࣄɾཧ༝ɾ಺༰ • ಛʹ࠷ॳ͸प஌ॏཁ • ϝτϦΫεΛؾʹ͠ͳ͕Β΍Δ • ֎෦γεςϜʹ΋ؾΛ഑Δ • 1.ͷԾઆ͸ਖ਼͔͔ͬͨ͠ʁ • ؔ࿈νʔϜ΁ͷϑΟʔυόοΫ • ෼ੳ݁Ռʹࣗ৴͕࣋ͯͨΒൣғΛ޿͛Δ • ৽ͨͳӨڹ͕ग़ͯ͘ΔՄೳੑʹ஫ҙ • खಈ࣮ߦ͔Βఆظࣗಈ࣮ߦ΁
32. Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • C͕յΕͯ΋D͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ
33. Chaos Maturity Model (CMM) Sophistication Adoption • devͰखಈ࣮ߦͯ͠system metricsΛ؂ࢹ • prodΛ໛ͨ͠؀ڥɺࣗಈ࣮ߦɾ؂ࢹɺbusiness metrics؂ࢹ • ݁Ռ͸खಈूܭɻ࣮ݧ͸੩తఆٛ • prodͰࣗಈ࣮ߦʢ؂ࢹɾ݁Ռूܭʣ • ࣮ݧ͸CDͱ౷߹ɻରর࣮ݧՄೳʢbusiness metricsൺֱʣ • શͯͷ؀ڥͰࣗಈ࣮ߦ • A/BςετɺٴͼಈతͳBRมߋɻଛӹ༧ଌ΍Ωϟύϓϥ༧ଌ ΄΅࠾༻͞Ε͍ͯͳ͍ɻCEೝ஌͕௿͍ o ffi cialʹ࣮ߦɺؔ৺Λ࣋ͭνʔϜ΋͋Δɻͨ·ʹCE࣮ߦ νʔϜ͕CE࣮ફʹઐ೦ɻλʔήοτΛߜͬͯఆظ࣮ߦɻIRͱGamedayͰFeedback සൟͳCE࣮ߦɻجຊతʹ͸CEΛߦ͏ɻجຊεΩϧԽɻCE as a default. Adoption Sophistication
34. Conclusion • ෼ࢄγεςϜͷߏஙɾӡ༻ && ։ൃεϐʔυͷҡ࣋ɾ޲্͕ٻΊΒΕΔɻ • γεςϜͷresiliencyΛ޲্͢ΔͨΊʹChaos Engineering͕ޮՌతͰ͋Δɻ • CEࣗମ͸ඇৗʹए͍෼໺ɻπʔϧɾ஌ݟɾਪਐʹ͝ڠྗ͍ͩ͘͞ɻ
35. ਺ֶతͳΧΦεͱ͸ʁʢ;Θͬͱͨ͠ղઆʣ • ܾఆ࿦తͳಈ͖Λ͢Δʢඍ෼ํఔࣜͰܾ·Δʣ • ϥϯμϜͰ͸ͳ͍ʢ֬཰աఔ͸ͳ͍ʣ • ಛ௃ • ॳظ஋ʹහײ • ඇઢܗͳಈ͖ • पظੑ͸ʢ͋ΔΑ͏ʹݟ͑Δ͕ʣແ͍ • ൃࢄ͸͠ͳ͍ • ྫɿؾ৅༧ใ
 ɹɹೋॏৼΓࢠ (https://youtu.be/zdW6nTNWbkc) • ༧ଌ͠೉͍͕ɺ࠶ݱੑ͸ʢݶఆత͕ͩʣ༗Δɻ͜ΕΛར༻ɻ ϩδεςΟοΫࣸ૾ https://ja.wikipedia.org/wiki/%E3%83%AD%E3%82%B8%E3%82%B9%E3%83%86%E3%82%A3%E3%83%83%E3%82%AF%E5%86%99%E5%83%8F a=1.8
36. EoP