Tweet
Tweet

Slide 1

Slide 1 text

Research Paper Introduction #18 “Chaos engineering: Building con fi dence in system behavior through experiments” ௨ࢉ#65 @cafenero_777 2021/01/28

Slide 2

Slide 2 text

Agenda • ର৅࿦จ • ֓ཁͱಡ΋͏ͱͨ͠ཧ༝ 1. Part I. IntroductionɿͳΜͰChaos EngineeringΛ΍ΔΜ͚ͩͬʁ 2. Part II. The Principles of ChaosɿChaos EngineeringͬͯͲ͏͍͏΋ͷʁ 3. Part III. Chaos In Practiceɿ࣮ࡍʹChaos EngineeringͬͯͲ͏΍Δͷʁ

Slide 3

Slide 3 text

$ which • Chaos engineering: Building con fi dence in system behavior through experiments • Casey Rosenthal, Lorin Hochstein, Aaron Blohowiak, Nora Jones, and Ali Basiri • Net fl ix, Inc. 2017 • Report? book? • https://learning.oreilly.com/library/view/chaos-engineering/9781491988459/ • Blog • https://www.oreilly.com/content/chaos-engineering

Slide 4

Slide 4 text

֓ཁͱಡ΋͏ͱͨ͠ཧ༝ • ֓ཁ • γεςϜ͸ඞͣ”མͪΔ”͕ɺෳࡶͳγεςϜΛ੩తʹݪҼಛఆ͢Δͷ͸೉͍͠ • ຊ൪؀ڥͰ͏·࣮͘ݧΛͯ͠ʢChaos EngineeringΛͯ͠ʣɺݪҼಛఆɾγεςϜվળʹܨ͛Δ • Net fl ixͰͷ10೥͙Β͍ͷCEܦݧଇΛڞ༗ • ಡ΋͏ͱͨ͠ཧ༝ • ࣄނΛݮΒ͍ͨ͠ • Chaos Engineering͕ྑͦ͞͏ɻ໘നͦ͏ͩ͠ӡ༻஌ݟ΋ར༻Ͱ͖ͦ͏ • ҎલಡΜͩ΋ͷ͸Ԡ༻ฤʢϑϨʔϜϫʔΫدΓͷ࿩ͩͬͨʣͩͬͨɻجૅฤΛ஌Γ͍ͨ • લճɿAutomating chaos experiments in production https://arxiv.org/abs/1905.04648

Slide 5

Slide 5 text

PRINCIPLES OF CHAOS ENGINEERING https://principlesofchaos.org/

Slide 6

Slide 6 text

Net fl ixͷʢࢲͷཧղ͍ͯ͠ΔʣCEྺ࢙ • 2008ʹAWSશ໘ҠߦʢCDNҎ֎ʣ • Chaos Monkey: VMࢭΊΔ • 2012೥ΫϦεϚεΠϒʹ୯ҰϦʔδϣϯͰେো֐ • Chaos Kong: RegionࢭΊΔ • յ͢͜ͱ͕CEͱ͍͏ޡղ͕ൃੜɻ৭ʑߟ͑Δ • 2016: ChAP (ChaosAutomation Platform), ϚΠΫϩαʔϏεʹಋೖ • 2017: PRINCIPLES OF CHAOS ENGINEERINGΛެ։ • 2019: Automating chaos experiments in production

Slide 7

Slide 7 text

Net fl ixͷʢࢲͷཧղ͍ͯ͠ΔʣCEྺ࢙ • 2008ʹAWSશ໘ҠߦʢCDNҎ֎ʣ • Chaos Monkey: VMࢭΊΔ • 2012೥ΫϦεϚεΠϒʹ୯ҰϦʔδϣϯͰେো֐ • Chaos Kong: RegionࢭΊΔ • յ͢͜ͱ͕CEͱ͍͏ޡղ͕ൃੜɻ৭ʑߟ͑Δ • 2016: ChAP (ChaosAutomation Platform), ϚΠΫϩαʔϏεʹಋೖ • 2017: PRINCIPLES OF CHAOS ENGINEERINGΛެ։ • 2019: Automating chaos experiments in production ࠓ೔ͷ࿩͸͜ͷลΓ

Slide 8

Slide 8 text

Part I. Introduction • ͳΜͰChaos EngineeringΛ΍ΔΜ͚ͩͬʁ

Slide 9

Slide 9 text

Why Do Chaos Engineering? (1/2) • ෼ࢄγεςϜͰ͸༧ظ͠ʹ͍͘Πϕϯτ͕ൃੜ • ίϯϙʔωϯτଟ͍ɺ૬ޓ࡞༻͢Δ • disk/NW failure, ΞΫηεٸ૿ͰαʔϏεϨϕϧྼԽ • ى͖Δ”લ”ʹγεςϜͷଟ͘ͷऑ఺Λಛఆ͍ͨ͠ • ͜ͷํ๏Λମܥతʹ·ͱΊͨͷ͕ΧΦεΤϯδχΞϦϯά • @Net fl ix • 2008-, AWSҠߦҎ߱։࢝ɺVM/Regionো֐ɺΫϦεϚεো֐ɻ • Chaos Monkey/Chaos Kong, FIT (Failure Injection Test)ɺSpinnakerͱ౷߹ɻPrinciple of Chaos Engineering • Chaos Moneky:ʢաڈ5೥ͰαʔϏεӨڹ͸̍౓͚ͩʣ • Chaos Kong: ݄ʹҰ౓ൃಈɺසൟʹ໰୊ൃੜɺɺɺ2೥໨Ͱ҆ఆ ྫ: Clos NW, ෼ࢄLB, ෼ࢄStorage, IaaS/PaaS/CaaS on k8s

Slide 10

Slide 10 text

Why Do Chaos Engineering? (2/2) • ܦݧతௐࠪͷ෼໺ʹద༻ɺγεςϜΛཧղ͢ΔͨΊʹ࣮ݧ͢Δ • γεςϜఀࢭͷՄೳੑ͕͋Δऑ఺ΛֶͿ • ࠶ൃ๷ࢭࡦ(post motem)ͱͯ͠ࣄޙʹ΍ΔͷͰ͸ͳ͘ɺࣄલʹ΍Δ(proactive) • طଘͷςετͱඃΔࣄ͸ଟ͍ɿݒ೦ࣄ߲ɾπʔϧ౳ • ςετ: ط஌ͷಛఆ৚݅ΛΫϦΞ͢Δ͔൱͔ɻyes or no ʢྫɿ֤Ξαʔγϣϯʣ • CE: ৽͍͠৘ใΛੜΈग़͢ɻྫɿNW஗ԆɾϥϯμϜʹྫ֎ൃੜɾλΠϜτϥϕϧɾI/O΍CPUߴෛՙ • લఏ৚݅ • ໌Β͔ʹऑ఺͕෼͔͍ͬͯΔ৔߹͸·ͣ͸௚͢ɻͦͷޙʹCEͯ͠ؾ͍͍ͮͯͳ͔ͬͨऑ఺͕໌Β͔ʹ͢Δ • ؂ࢹγεςϜɿఆৗঢ়ଶͱ൑அͰ͖ΔΑ͏ʹՄࢹԽ

Slide 11

Slide 11 text

Managing Complexity • ύϑΥʔϚϯεɾՄ༻ੑɾϑΥʔϧττϨϥϯε + ։ൃ଎౓͕ඞཁ • ϚΠΫϩαʔϏεΛ࠾༻ͯ͠ɺ֤νʔϜಠཱʹ։ൃɾӡ༻ • ਓؒͷཧղΛ٘ਜ਼ʹͯ͠଎౓ɾॊೈੑΛ୲อɻ • ࣄલͷςετͰ͸ݟ͚ͭͮΒ͍ • ط஌ͷΞαʔγϣϯ͚ͩͰ͸Χόʔ͖͠Εͳ͍ • ྫɿΩϡʔɺΦʔτεέʔϧɺϩʔΧϧΩϟογϡɾεέʔϧμ΢ϯɻ
 ɹɹෛՙͷฏۉ஋͚ͩͰे෼ʁʁ • ϒϧ΢ΟοϓޮՌ • ݸʑͷϚΠΫϩαʔϏε͸߹ཧత͕ͩɺ࠷ऴӨڹ͕༧ظͰ͖ͳ͍৔߹͕͋Δ get meta-data get meta-data map/reduce Ϩεϙϯε ϦΫΤετ ิ଍ɿ࠷ۙྲྀߦΓͷΞϨ΋ಉ͡? ౴: DeepLearning

Slide 12

Slide 12 text

Part II. The Principles of Chaos • Chaos EngineeringͬͯͲ͏͍͏΋ͷʁ

Slide 13

Slide 13 text

Part II. The Principles of Chaos • ΧΦεΤϯδχΞϦϯά • ϥϯμϜʢແܭըʣͰ͸ͳ͍ɻ࢓ࣄͷΧΦεΛ༠ൃ͢Δ͜ͱ΋ͳ͍ɻ • ίϯϙʔωϯτ૬ޓ࡞༻ͷ݁ՌɺγεςϜ͕ΧΦεʹͳΔՄೳੑΛཧղ͢Δ͜ͱ • গ͚ͩ͠γεςϜʹΧΦεΛೖΕͯɺγεςϜ͕Ͳ͏ػೳ͢Δ͔ʁ • ఆৗঢ়ଶͷԾઆΛཱͯΔ • ࣮ੈքͷΠϕϯτΛมߋ͢Δ • prodͰ࣮ݧ͢Δ • ࣮ݧΛࣗಈԽͯ͠ܧଓతʹߦ͏ • Bast RadiusʢӨڹൣғʣΛ࠷খԽ͓ͯ͘͠

Slide 14

Slide 14 text

Hypothesize about Steady State (1/2) • ఆৗঢ়ଶͱ͸ʁʹϏδωεϩδοΫͷظ଴஋΍SLA • ֤छKPI (޿ࠂදࣔճ਺ɺΧʔτ௥Ճɺ໰͍߹Θͤྔ) • ϝτϦΫεɾΞϥʔτ؂ࢹ • CPU/Memory/NW, I/O, RPS, duration • εϧʔϓοτ, Τϥʔ཰, 99%ile • ࠶ੜϘλϯΛԡ͢଎౓: SPS • ΫϥΠΞϯτଆͱҰॹʹݕূͰ͖ΔͱΑΓྑ͍ɻ • पظతʹϝτϦΫεΛݟΔඞཁ͕͋Δ৔߹΋ɻ

Slide 15

Slide 15 text

Hypothesize about Steady State (2/2) • ԾઆΛཱͯΔඞཁ͋Γ • Ծઆ͕ແ͍ͱɺͲͷϝτϦΫεʹ஫໨͢΂͖͔͑͞෼͔Βͳ͍ɻ • ʮ͋ΔCE࣮ݧΛͯ͠΋҆ఆঢ়ଶͷ··Ͱ͢ɻʯ͕Ծઆɻ • ྫɿpersonalized listΛࣦഊ-> ௨ৗͷlistΛදࣔͤ͞ɺSPS (start/sec)ʹӨڹΛ༩͑ͳ͍ • ྫɿτϥϑΟοΫΛผϦʔδϣϯʹϦμΠϨΫτͤͯ͞΋SPS҆ఆ • ͖͍͠஋ͷઃఆ͸ʁ • Ծઆ͕ࣝผͰ͖Δ஋ɻʢଥ౰ͩͱࢥ͏ൣғɻओ؍తͳภࠩྔʣ

Slide 16

Slide 16 text

Vary Real-World Events (1/2) • ൃੜΠϕϯτͷස౓ͱӨڹɺରॲίετɾෳࡶ͞Λݟੵ΋Δ • VMఀࢭɿසൃɾ୯७ɾίετ௿ • ϦʔδϣϯఀࢭɿෳࡶͰίετߴ • จԽతཁҼ΋͋ΔछͷίετͱΈͳ͢ • DC͸ݎ࿚ੑɾ҆ఆੑ >>> | ӽ͑ΒΕͳ͍น | >>> ढ़හੑ • ҰํͰΫϥ΢υར༻ʢHW੹೚ͷ֎෦ԽʣͰHWো֐͸౰વʹͳ͖ͬͯͨ Πϕϯτྫɿ ϋʔυ΢ΣΞো֐ ػೳతͳόά ঢ়ଶෆҰகʹΑΔૹ৴Τϥʔ ωοτϫʔΫͷ஗ԆͱύʔςΟγϣϯ ೖྗͷେ͖ͳมಈɺ͓ΑͼϦτϥΠετʔϜ Ϧιʔεͷރׇ Ϗβϯνϯো֐ɾڝ߹ঢ়ଶ

Slide 17

Slide 17 text

Vary Real-World Events (2/2) • Πϕϯτ • ߴෛՙɺ௨৴஗ԆɺNWύʔςΟγϣϯɺແޮͳূ໌ॻɺσʔλංେԽ • όάࠞೖίʔυͷσϓϩΠ • ΧφϦΞϦϦʔεͩͱখن໛͕ͩӨڹ͸ग़ͯ͠·͏ • αʔϏεݺͼग़࣌͠ʹΤϥʔΛฦͯ͠ֆϛϡʔϨͱ͢Δͱ҆શʢྫɿಛఆΫϦΞϯτ͔ΒͷAPIͷΈࣦഊʣ • ো֐υϝΠϯʢӨڹͱൣғʣΛڱ͘͢ΔΑ͏ʹCE͢Δɻ • ผͷݪҼ͕ىͨ͜͠ࣄ৅ͩͬͨͱͯ͠΋੾Γ෼͚͠΍͍͢ɻ • ϦιʔεʢαʔϏεͷάϧʔϓʣ͸ڞ༗͞Ε͍ͯΔͨΊɺશͯΛো֐υϝΠϯͱͯ͠ઢҾ͖͸Ͱ͖ͳ͍఺ʹ஫ҙɻ • ॲཧͰ͖Δͱ༧૝͞ΕΔΠϕϯτͷΈࢼ͢ʢͦͯ͠௚͢ʣɻແҙຯʹյ͢ͷ͸ແҙຯɻ

Slide 18

Slide 18 text

Run Experiments in Production • ίʔυͷਖ਼͠͞ʹՃ͑ͯγεςϜશମͷਖ਼͠͞Λ୲อ͍ͨ͠ • prodͱಉ͡ঢ়ଶΛΤϛϡʔϨʔτͯ͠devͰࢼ͢ʁ • ೖྗ(input)΍֎෦γεςϜΛΤϛϡϨʔτ͢Δ͸ݶք͕͋Δ • CIతʹιʔείʔυɾઃఆϑΝΠϧ͸౎౓มΘΔɻ • Α͋͘Δݴ͍༁ • ਓͷ໋͕௚઀͔͔ͬͯΔγεςϜ͸͠ΐ͏͕ͳ͍ɻ͕ɺ͘͝Ұ෦ͷϋζɻ • γεςϜʹճ෮ྗ͕ͳ͍ʢͬͱ͍͏ࣄΛӅ͍ͨ͠ʁʣ • ͡Ό͋Ͳ͏͢Δ͔ʁ • ͙͢ʹ࣮ݧఀࢭͰ͖ΔΑ͏ʹ͓ͯ͘͠ • Blast RadiusʢӨڹൣғʣΛ࠷খʹ͓ͯ͘͠ • ʢͲ͏ͯ͠΋prodͰͰ͖ͳ͚Ε͹ʣຊ൪؀ڥʹ͍ۙ؀ڥͰ࣮ݧ͢Δ

Slide 19

Slide 19 text

Automate Experiments to Run Continuously • ςετࣗಈ࣮ߦ • ຊ൪؀ڥ͸CI/CD͞ΕΔ -> CE΋ࣗಈԽͯ͠ܧଓతʹνΣοΫ͢Δ • ཧ૝తʹ͸ίʔυมߋͷ౓ʹ࣮ߦʢΧΦεΧφϦΞʁʣ • ࢀߟɿਓͰͳ͘γεςϜ͕ҟৗΛ൑அɾ࣮ݧఀࢭɻӨڹςϯϓϨԽɻSpinnaker͕ຊ൪ʹ࣮ݧϊʔυىಈɻ • ςετࣗಈੜ੒ • ·ͩى͖ͨ͜ͱͷͳ͍ɺى͖ͦ͏ͳΠϕϯτ΍ো֐Λಛఆͯ͠ਓ͕ؒ࡞Δɻʢཁ͸ηϯεʁʂʣ • Incident Trackerʹه࿥͞Ε͍ͯΔط஌ͷࣄ৅~ʹ࠷௿ݶ΍Δ΂͖ςετɻ ࢀߟɿLDFI: lineage-driven fault injection (ܥ౷ۦಈܕϑΥʔϧτΠϯδΣΫγϣϯ) Automating Failure Testing Research at Internet Scale, https://dl.acm.org/doi/10.1145/2987550.2987555

Slide 20

Slide 20 text

Minimize Blast Radius • νΣϧϊϒΠϦݪൃࣄނͷݪҼ͸ϝϯςதͷ”ࢼݧ”ͩͬͨɾɾɾ • ྫɿ • গ਺ͷϢʔβʢclientʣʹ࢓ֻ͚Δ • গ਺ͷϢʔβʢclientҎ֎ʣʹ࢓ֻ͚Δ • શϢʔβɺͨͩ͠ಛఆՕॴʢαʔϏεɾϩδοΫʣ΁ϧʔςΟϯά • શϢʔβɺશαʔϏε • ػೳͷfallback͸ඞਢɺۓٸఀࢭػೳ͸ඞਢʢࣗಈऴ͕͓ྃ͢͢Ίʣ • ͙͢ʹରԠͰ͖ΔΑ͏ʹ೔தଳʢϔϧϓ͕ಘΒΕΔ࣌ؒଳʣʹ࣮ݧ͢Δɻ

Slide 21

Slide 21 text

Part III. Chaos In Practice • ࣮ࡍʹChaos EngineeringͬͯͲ͏΍Δͷʁ • ΍Δ͔Ͳ͏͔໎͍ͬͯΔਓ΁ • ͲͪΒ͕ྑ͍͔ɻ • ҙਤͨ͠λΠϛϯάͰখ͞ͳো֐Λىͯͦ͜͠ΕΛ௚͢ • ҙਤ͠ͳ͍λΠϛϯάͰେ͖ͳো֐͕ى͖͔ͯΒ௚͢ • ۜߦܥɺۚ༥ܥͰ͑͞΋CEΛ࢝Ί͍ͯΔ

Slide 22

Slide 22 text

Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ

Slide 23

Slide 23 text

Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ

Slide 24

Slide 24 text

Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ Ծઆྫɿ RedisͰtimeoutͨ͠৔߹ Act/Stb failoverͨ͠৔߹

Slide 25

Slide 25 text

Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ Ծઆྫɿ RedisͰtimeoutͨ͠৔߹ Act/Stb failoverͨ͠৔߹ • BRΛ࠷খʹͯ͠ɺprod؀ڥͰߦ͏ • dev؀ڥͰͷdry-run͢Δ

Slide 26

Slide 26 text

Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ Ծઆྫɿ RedisͰtimeoutͨ͠৔߹ Act/Stb failoverͨ͠৔߹ • BRΛ࠷খʹͯ͠ɺprod؀ڥͰߦ͏ • dev؀ڥͰͷdry-run͢Δ • ”ਖ਼ৗ”ͷఆٛͱɺͦΕͱͷࠩʢҟৗʣͷࢦඪܾΊ • ۓٸఀࢭ༻ͷ͖͍͠஋ܾΊ

Slide 27

Slide 27 text

Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ Ծઆྫɿ RedisͰtimeoutͨ͠৔߹ Act/Stb failoverͨ͠৔߹ • BRΛ࠷খʹͯ͠ɺprod؀ڥͰߦ͏ • dev؀ڥͰͷdry-run͢Δ • ”ਖ਼ৗ”ͷఆٛͱɺͦΕͱͷࠩʢҟৗʣͷࢦඪܾΊ • ۓٸఀࢭ༻ͷ͖͍͠஋ܾΊ • CE࣮ߦதͰ͋Δࣄɾཧ༝ɾ಺༰ • ಛʹ࠷ॳ͸प஌ॏཁ

Slide 28

Slide 28 text

Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ Ծઆྫɿ RedisͰtimeoutͨ͠৔߹ Act/Stb failoverͨ͠৔߹ • BRΛ࠷খʹͯ͠ɺprod؀ڥͰߦ͏ • dev؀ڥͰͷdry-run͢Δ • ”ਖ਼ৗ”ͷఆٛͱɺͦΕͱͷࠩʢҟৗʣͷࢦඪܾΊ • ۓٸఀࢭ༻ͷ͖͍͠஋ܾΊ • CE࣮ߦதͰ͋Δࣄɾཧ༝ɾ಺༰ • ಛʹ࠷ॳ͸प஌ॏཁ • ϝτϦΫεΛؾʹ͠ͳ͕Β΍Δ • ֎෦γεςϜʹ΋ؾΛ഑Δ

Slide 29

Slide 29 text

Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ Ծઆྫɿ RedisͰtimeoutͨ͠৔߹ Act/Stb failoverͨ͠৔߹ • BRΛ࠷খʹͯ͠ɺprod؀ڥͰߦ͏ • dev؀ڥͰͷdry-run͢Δ • ”ਖ਼ৗ”ͷఆٛͱɺͦΕͱͷࠩʢҟৗʣͷࢦඪܾΊ • ۓٸఀࢭ༻ͷ͖͍͠஋ܾΊ • CE࣮ߦதͰ͋Δࣄɾཧ༝ɾ಺༰ • ಛʹ࠷ॳ͸प஌ॏཁ • ϝτϦΫεΛؾʹ͠ͳ͕Β΍Δ • ֎෦γεςϜʹ΋ؾΛ഑Δ • 1.ͷԾઆ͸ਖ਼͔͔ͬͨ͠ʁ • ؔ࿈νʔϜ΁ͷϑΟʔυόοΫ

Slide 30

Slide 30 text

Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ Ծઆྫɿ RedisͰtimeoutͨ͠৔߹ Act/Stb failoverͨ͠৔߹ • BRΛ࠷খʹͯ͠ɺprod؀ڥͰߦ͏ • dev؀ڥͰͷdry-run͢Δ • ”ਖ਼ৗ”ͷఆٛͱɺͦΕͱͷࠩʢҟৗʣͷࢦඪܾΊ • ۓٸఀࢭ༻ͷ͖͍͠஋ܾΊ • CE࣮ߦதͰ͋Δࣄɾཧ༝ɾ಺༰ • ಛʹ࠷ॳ͸प஌ॏཁ • ϝτϦΫεΛؾʹ͠ͳ͕Β΍Δ • ֎෦γεςϜʹ΋ؾΛ഑Δ • 1.ͷԾઆ͸ਖ਼͔͔ͬͨ͠ʁ • ؔ࿈νʔϜ΁ͷϑΟʔυόοΫ • ෼ੳ݁Ռʹࣗ৴͕࣋ͯͨΒൣғΛ޿͛Δ • ৽ͨͳӨڹ͕ग़ͯ͘ΔՄೳੑʹ஫ҙ

Slide 31

Slide 31 text

Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • A͕յΕͯ΋B͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ Ծઆྫɿ RedisͰtimeoutͨ͠৔߹ Act/Stb failoverͨ͠৔߹ • BRΛ࠷খʹͯ͠ɺprod؀ڥͰߦ͏ • dev؀ڥͰͷdry-run͢Δ • ”ਖ਼ৗ”ͷఆٛͱɺͦΕͱͷࠩʢҟৗʣͷࢦඪܾΊ • ۓٸఀࢭ༻ͷ͖͍͠஋ܾΊ • CE࣮ߦதͰ͋Δࣄɾཧ༝ɾ಺༰ • ಛʹ࠷ॳ͸प஌ॏཁ • ϝτϦΫεΛؾʹ͠ͳ͕Β΍Δ • ֎෦γεςϜʹ΋ؾΛ഑Δ • 1.ͷԾઆ͸ਖ਼͔͔ͬͨ͠ʁ • ؔ࿈νʔϜ΁ͷϑΟʔυόοΫ • ෼ੳ݁Ռʹࣗ৴͕࣋ͯͨΒൣғΛ޿͛Δ • ৽ͨͳӨڹ͕ग़ͯ͘ΔՄೳੑʹ஫ҙ • खಈ࣮ߦ͔Βఆظࣗಈ࣮ߦ΁

Slide 32

Slide 32 text

Designing Experiments 1.ԾઆΛཱͯΔ 2. ࣮ݧൣғΛܾΊΔ 3. ϝτϦΫεΛܾΊΔ 4. प஌͢Δ 5. ࣮ݧ͢Δ 6. ෼ੳ͢Δ 7. BRΛ޿͛Δ 8. ࣗಈԽ͢Δ • C͕յΕͯ΋D͕ҙਤͨ͠ಈ࡞Λ͢Δ͔ʁ • ฏۉൃݟɾӨڹɾ෮چ࣌ؒ͸ʁਓͷಈ͖͸ʁ

Slide 33

Slide 33 text

Chaos Maturity Model (CMM) Sophistication Adoption • devͰखಈ࣮ߦͯ͠system metricsΛ؂ࢹ • prodΛ໛ͨ͠؀ڥɺࣗಈ࣮ߦɾ؂ࢹɺbusiness metrics؂ࢹ • ݁Ռ͸खಈूܭɻ࣮ݧ͸੩తఆٛ • prodͰࣗಈ࣮ߦʢ؂ࢹɾ݁Ռूܭʣ • ࣮ݧ͸CDͱ౷߹ɻରর࣮ݧՄೳʢbusiness metricsൺֱʣ • શͯͷ؀ڥͰࣗಈ࣮ߦ • A/BςετɺٴͼಈతͳBRมߋɻଛӹ༧ଌ΍Ωϟύϓϥ༧ଌ ΄΅࠾༻͞Ε͍ͯͳ͍ɻCEೝ஌͕௿͍ o ffi cialʹ࣮ߦɺؔ৺Λ࣋ͭνʔϜ΋͋Δɻͨ·ʹCE࣮ߦ νʔϜ͕CE࣮ફʹઐ೦ɻλʔήοτΛߜͬͯఆظ࣮ߦɻIRͱGamedayͰFeedback සൟͳCE࣮ߦɻجຊతʹ͸CEΛߦ͏ɻجຊεΩϧԽɻCE as a default. Adoption Sophistication

Slide 34

Slide 34 text

Conclusion • ෼ࢄγεςϜͷߏஙɾӡ༻ && ։ൃεϐʔυͷҡ࣋ɾ޲্͕ٻΊΒΕΔɻ • γεςϜͷresiliencyΛ޲্͢ΔͨΊʹChaos Engineering͕ޮՌతͰ͋Δɻ • CEࣗମ͸ඇৗʹए͍෼໺ɻπʔϧɾ஌ݟɾਪਐʹ͝ڠྗ͍ͩ͘͞ɻ

Slide 35

Slide 35 text

਺ֶతͳΧΦεͱ͸ʁʢ;Θͬͱͨ͠ղઆʣ • ܾఆ࿦తͳಈ͖Λ͢Δʢඍ෼ํఔࣜͰܾ·Δʣ • ϥϯμϜͰ͸ͳ͍ʢ֬཰աఔ͸ͳ͍ʣ • ಛ௃ • ॳظ஋ʹහײ • ඇઢܗͳಈ͖ • पظੑ͸ʢ͋ΔΑ͏ʹݟ͑Δ͕ʣແ͍ • ൃࢄ͸͠ͳ͍ • ྫɿؾ৅༧ใ
 ɹɹೋॏৼΓࢠ (https://youtu.be/zdW6nTNWbkc) • ༧ଌ͠೉͍͕ɺ࠶ݱੑ͸ʢݶఆత͕ͩʣ༗Δɻ͜ΕΛར༻ɻ ϩδεςΟοΫࣸ૾ https://ja.wikipedia.org/wiki/%E3%83%AD%E3%82%B8%E3%82%B9%E3%83%86%E3%82%A3%E3%83%83%E3%82%AF%E5%86%99%E5%83%8F a=1.8

Slide 36

Slide 36 text

EoP