カオスエンジニアリングへの招待

 カオスエンジニアリングへの招待

「AKIBA.AWS #15 ガチ編〜re:Invent振り返り その1〜」での発表資料です。
https://classmethod.connpass.com/event/159529/

142221c4c936c5c0c5a7429f40a7f3ff?s=128

KyoheiSaiki

January 15, 2020
Tweet

Transcript

  1. ΧΦεΤϯδχΞϦϯά΁ͷট଴  ࠤഢګฏ ,ZP ",*#""84ΨνฤʙSF*OWFOUৼΓฦΓͦͷʙ

  2. 8)0".*   ࠤഢګฏʢ,ZPʣ w+0*/ w"84ࣄۀຊ෦ίϯαϧ෦ wιϦϡʔγϣϯΞʔΩςΫτ झຯɿ ϥΠϑαΠΤϯε

  3. Կͷ೔Ͱ͔͢ʁ  2019/08/23

  4. "84౦ژϦʔδϣϯͷେن໛ো֐  ͪͳΈʹೖࣾޙॳͷ͓٬༷؀ڥͰͷ࡞ۀ༧ఆ೔Ͱͨ͠

  5.  "Everything fails, all the time" - Werner Vogels (CTO,

    Amazon.com)
  6. ࠓ೔ͷຊ୊  ࣮ݧΛ௨ͯ͠γεςϜͷʢো֐࣌ͷʣৼΔ෣͍ʹ ࣗ৴Λ͚ͭΔ

  7.  ΧΦεΤϯδχΞϦϯά is Կ

  8. ఆٛʢจݙΑΓʣ  wΧΦεΤϯδχΞϦϯά͸ɺຊ൪؀ڥͷߥΕڰͬͨঢ়ଶʹ଱͑ΔγεςϜͷೳྗ ʹࣗ৴Λ࣋ͨͤΔͨΊʹɺ෼ࢄγεςϜͰ࣮ݧ͢Δֶ໰Ͱ͢ Netflix paper ɻ wΧΦεΤϯδχΞϦϯάͷ࠷ऴతͳ໨ඪ͸ɺγεςϜͷ҆ఆੑͱճ෮ྗΛڧԽ͢ Δ͜ͱͰ͢ Gremlin

    white paper ɻ resilience  ʤපؾɾෆ޾ɾࠔ೉ɾۤڥͳͲ͔Βͷʥճ෮ྗɺཱͪ ௚Δྗɺ෮׆ྗ  ʤมܗ͞Εͨ෺͕ݩͷܗʹ໭Δʥ෮ݩྗɺ஄ྗʢੑʣ ӳࣙ࿠POUIF8&#ΑΓൈਮ
  9. ఆٛʢͬ͘͟Γʣ  wෳࡶͰڍಈ͕༧૝ෆೳͳʢ෼ࢄʣγεςϜʹରͯ͠ɺ ো֐஫ೖ ࣮ݧΛߦ͏͜ͱͰͦͷऑ఺Λ͋Ϳ Γग़͢ ‎͋ͿΓग़ͨ͠ऑ఺ΛڧԽ͢Δ͜ͱͰΑΓճ෮ྗͷ͋ΔγεςϜΛߏங ‎ऑ఺Ͱͳ͔ͬͨ৔߹ɺͦ͜ʹ͸ࣗ৴͕࣋ͯΔ wো֐஫ೖ͸໨తͰ͸ͳ͘खஈ wߥͬΆ͍݈߁਍அͱ༧๷஫ࣹͷηοτͱ͍͏Πϝʔδ

  10. Կ͕خ͍͠ͷʁ  wͲΜͳγεςϜ΋μ΢ϯͯ͠͠·͏ͱՁ஋ΛఏڙͰ͖ͳ͍ ‎ճ෮ྗ্͕͕Δ͜ͱ͸ʢఔ౓ʹࠩ͸͋ΕͲʣ΄΅શͯͷγεςϜʹͱͬͯϋοϐʔ wചΓ্͛௿Լͷ๷ࢭ ‎μ΢ϯλΠϜ͕ചΓ্͛ʹ௚઀Өڹ͢Δ&$αΠτͳͲ wࣗ෼ͨͪͷϒϥϯυΠϝʔδΛकΔʢϓϥΠεϨεʣ

  11. ͲΜͳاۀ͕΍ͬͯΔͷʁ  ւ֎ ೔ຊ

  12.  ΧΦεΤϯδχΞϦϯάͷݪଇ

  13. w ௨ৗͷಈ࡞Λࣔ͢γεςϜͷଌఆՄೳͳग़ྗͱͯ͠ʮఆৗঢ়ଶʯΛఆٛ͢Δ͜ͱ͔Β࢝Ί·͢ w ͜ͷఆৗঢ়ଶ͸ɺରর܈͓Αͼ࣮ݧ܈ͷ྆ํͰܧଓ͢ΔͱԾఆ͠·͢ w αʔόʔͷΫϥογϡɺϋʔυυϥΠϒͷޡ࡞ಈɺωοτϫʔΫ઀ଓͷ੾அͳͲɺݱ࣮ੈքͷΠϕϯ τΛ൓ө͢Δม਺Λಋೖ͠·͢ w ରর܈ͱ࣮ݧ܈ͱͷؒͷఆৗঢ়ଶͷҧ͍Λௐ΂Δ͜ͱʹΑͬͯԾઆΛ൓ূ͠Α͏ͱ͠·͢ ΧΦεΤϯδχΞϦϯάͷݪଇجຊݪଇ

     https://principlesofchaos.org/?lang=ENcontent# ݕূʹ͓͚ΔΧΦε
  14. ΧΦεΤϯδχΞϦϯάͷݪଇৄࡉͳݪଇ  w ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔ w ࣮ੈքͷࣄ৅͸ଟ༷Ͱ͋Δ w ຊ൪؀ڥͰݕূΛ࣮ߦ͢Δ w ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ

    w ӨڹൣғΛ࠷খԽ͢Δ
  15. ఆৗঢ়ଶJTԿ  γεςϜ͕ద੾ʹಈ࡞ͯ͠ΔͬͯͲ͏͍͏͜ͱʁ

  16. ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔఆৗঢ়ଶͷ೺Ѳ  ϏδωεϝτϦΫεʹ஫໨ w ސ٬Λࣦ͍ͬͯΔ͔ʁ w ސ٬͸αΠτͷॏཁͳػೳΛ࣮ߦͰ͖Δ͔ʁʢ&$αΠτͰΧʔτͷνΣοΫΞ΢τ΍௥ՃͳͲʣ w ஗ԆΛཧ༝ʹސ٬͸αʔϏεΛఀࢭ͢Δ͔ʁ w

    ຖඵϏσΦͷετϦʔϛϯάΛ։࢝͢ΔϢʔβʔ਺ ‎4USFBNJOHQFSTFDPOE 414  ࢀߟ  w"NB[PO F#BZച্ w(PPHMF 'BDFCPPL޿ࠂͷදࣔճ਺ https://medium.com/netflix-techblog/sps-the-pulse-of-netflix-streaming-ae4db0e05f8a SPS
  17. ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔԾઆΛཱͯΔ  • γεςϜʹΠϕϯτΛ஫ೖ͢Δͱɺఆৗঢ়ଶͷಈ࡞͕ͲͷΑ͏ʹมԽ͢Δ͔Λߟ͑Δ • ఆৗঢ়ଶ͸தஅ͞ΕΔʁͦΕͱ΋มԽ͠ͳ͍ʁ • தஅͷ৔߹ɺγεςϜग़ྗ͕૿Ճ·ͨ͸ݮগ͢Δͱ༧૝͍ͯ͠Δʁ w ஫ೖ͢ΔΠϕϯτʹΑͬͯɺʮγεςϜͷಈ࡞͕ఆৗঢ়ଶ͔ΒมԽ͢Δ͜ͱ͸ͳ͍ʯͱ͍͏Ծઆͷ΋ͱ

    ࣮ݧΛ࣮ࢪ w ʢྫ Ϣʔβʔʹදࣔ͞ΕΔύʔιφϥΠζ͞ΕͨөըϦετΛੜ੒͢ΔαʔϏεͷࣦഊ ‎ γεςϜ͸σϑΥϧτͷʢύʔιφϥΠζ͞Ε͍ͯͳ͍ʣөըϦετΛฦ͢ w طʹऑ఺͕෼͔͍ͬͯΔ৔߹͸࣮ݧΛߦ͏લʹमਖ਼͢Δ
  18. ࣮ੈքͷࣄ৅͸ଟ༷Ͱ͋Δ  • ݱ࣮ͷੈքͰൃੜ͢ΔՄೳੑͷ͋Δೖྗ͔ΒΠϕϯτΛαϯϓϦϯάͯ͠஫ೖ • Ҏલʹൃੜͨ͠໰୊ͷ࠶ൃΛ๷͙ͨΊɺͦΕʹؔ܎͢ΔΠϕϯτΛར༻͢Δ͜ͱ͸ͱͯ΋༗༻ w Ծ૝ϚγϯΠϯελϯεΛऴྃ w αʔϏεؒͷϦΫΤετʹϨΠςϯγΛૠೖ

    w αʔϏεؒͷϦΫΤετͷࣦഊ w ಺෦αʔϏε͕ࣦഊ w ಛఆͷ"84ϦʔδϣϯΛ࢖༻ෆೳʹ Chaos Kong
  19. ຊ൪؀ڥͰݕূΛ࣮ߦ͢Δ  ΧΦεΤϯδχΞϦϯάͰ͸ຊ൪؀ڥͰͷγεςϜશମͷ;Δ·͍Λ஌Γ͍ͨɻ ։ൃ؀ڥͰ͸ҎԼΛຊ൪؀ڥͱಉҰʹ͢Δ͜ͱ͸೉͍͠ɻ • γεςϜͷ࣋ͭσʔλ΍εςʔτ • Ϣʔβ͔Βͷ༧૝֎ͷೖྗ • ઀ଓ͍ͯ͠Δ֎෦αʔϏε

    ➡ DNS, ֎෦SaaS etc. • ϋΠϖʔεͰ೔ʑߋ৽͞Εଓ͚Δίʔυ w ຊ൪؀ڥͰݕূΛ࣮ߦ
  20. ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ  • ॳΊͷҰճΛखಈͰ࣮ݧΛߦͬͯΈΔͷ͸࣮ݧ͕ਖ਼͘͠ߦ͑Δ͔Λ֬ೝ͢Δͷʹྑ͍ • ҰํͰɺݱ୅ͷγεςϜͷෳࡶ͞͸ɺͲͷมߋ͕࣮ݧͷ݁ՌΛม͑Δ͔༧ଌͰ͖ͳ͍ ➡ ͢΂ͯͷมߋ͕݁ՌʹӨڹ͢Δͱߟ͑Δ • લ߲Ͱड़΂ͨͱ͓Γɺຊ൪؀ڥ͸ৗʹมԽ

    ➡ ࣮ݧ݁Ռ͕୲อ͢Δ৴པੑ͸࣌ؒͱͱ΋ʹݮਰ • ΤϯδχΞϦϯάϦιʔε͸༗ݶ ➡ खಈ࣮ݧʹΑΔ։ൃεϐʔυͷ௿Լ͸ຊ຤స౗ w ΧΦε࣮ݧࣗಈԽͷͨΊͷπʔϧͱϓϥοτϑΥʔϜͷ࡞੒
  21. ӨڹൣғΛ࠷খԽ͢Δ  • ຊ൪؀ڥͰͷ࣮ݧ͸ɺސ٬ʹ௧ΈΛҾ͖ى͜͢Մೳੑ͕͋Δ • ࣮ݧ͔ΒͷμϝʔδΛ࠷খݶʹ཈͑Δ͜ͱ͸ΧΦεΤϯδχΞͷٛ຿ w ސ٬͕γεςϜΛར༻Ͱ͖ͳ͘ͳͬͨ৔߹͸ɺ͙͢ʹ࣮ݧΛऴྃ ‎ ۓٸఀࢭϘλϯͷ࣮૷

    ‎ ಛఆ৚݅ʹΑΔࣗಈऴྃ w ҟৗ͕ൃੜͨ͠ࡍʹ͸ૉૣ͍ରԠ ‎ Ӧۀ࣌ؒதʹͷΈ࣮ݧΛ࣮ࢪ
  22.  ো֐஫ೖπʔϧ

  23.  Tool (N)umber of attack types available Enterprise support Can

    halt attack in progress GUI Open source Software as a service (SaaS) Cloud agnostic Can attack container / pod Can attack using k8s primitives Can attack serverless Has API Bloomberg Powerful Seal 1 ✔ ✔ ✔ ✔ ✔ ChaosBlade 4 ✔ ✔ ✔ ChaosIQ Chaos Toolkit 3 Via vendor ✔ ✔ ✔ ✔ ✔ ✔ Gremlin Free 2 ✔ ✔ ✔ ✔ ✔ ✔ ✔ Gremlin ILFI 11 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Gremlin ALFI 1 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Istio 2 ✔ ✔ ✔ ✔ Netflix Chaos Monkey 1 ✔ Netflix Simian Army 2 ✔ New Relic Chaos Panda 2 ✔ Nutanix X- Ray 3 ✔ ✔ Pumba 6 ✔ ✔ ✔ https://www.gremlin.com/chaos-engineering/chaos-engineering-tools-comparison/ ো֐஫ೖπʔϧ͸݁ߏ๛෋
  24. (SFNMJOɺࣾ಺Ͱྲྀߦͬͯ·͢  • ݩNetflixͷΧΦεΤϯδχΞKolton AndrusΒʹΑΔSaaS • ʢEC2ͷ৔߹ʣ15෼ఔ౓ͰΧΦε࣮ݧ͕࢝ΊΒΕΔखܰ͞ • ؂ࢹπʔϧ΍Slackͱͷ౷߹΋؆୯ɻۓٸఀࢭϘλϯ΋࣮૷ࡁΈ

  25.  ΧΦεΤϯδχΞϦϯάͷ࢝Ίํʁ

  26. ࢝ΊΑ͏ͱ͢Δͱʜ  w ຊ൪Ͱͷ࣮ݧ͸ϦεΫߴͯ͘ා͍ɻຊ൪Ͱ͸΍Γͨ͘ͳ͍ʜ w Ͱ΋࣮ݧͷͨΊ͚ͩͷ؀ڥΛҡ࣋͢Δͷ΋େมͩ͠ɺαʔυ ύʔςΟ࿈ܞ౳ຊ൪ͱҧ͏ͱ͜Ζ΋ଟ͍ʜ ো֐ൃੜ࣌ͷϏδωεΠϯύΫτ w ػձଛࣦ

    w ചΓ্͛ w ϒϥϯυΠϝʔδ w ิঈඅ༻ ΧΦεΤϯδχΞϦϯάಋೖίετ w ࣄނൃੜͷϦεΫ w ਓతϦιʔε w ৺ཧతϋʔυϧ VS ܦݧ͕ͳ͍ͱΧΦεΤϯδχΞϦϯάͷίετͷݟੵ΋Γ͸೉͍͠ʜ ✖ ൃੜ֬཰
  27. (BNFEBZͷεεϝ  "84(BNFEBZ w ౉͞Εͨ"84؀ڥʹͲΜͲΜো֐͕ൃੜ͢ΔͷͰɺͦΕΛൃݟ͠मਖ਼ w ୹࣌ؒͰେྔͷো֐ʹରԠ͢Δ͜ͱʹͳΔͷͰຊ౰ͷҙຯͰ"84΁ͷཧղ͕ࢼ͞ΕΔ w Ϋϥϝι΋ࣾ಺(BNFEBZΛݱࡏاըதɻ΋͠धཁ͕͋Ε͹֎෦޲͚ʹ΋ʜ •

    ҆શͳ؀ڥͰͷΧΦεΤϯδχΞϦϯάͷ࿅श • είʔϓΛߜͬͯ਺࣌ؒఔ౓ɺো֐஫ೖΛ࣮ࢪ • ࣗࣾͰग़དྷΔͷ͕ϕλʔ͕ͩAWS Gameday΋͋Δ https://awsgamedaymicroservicestokyo.splashthat.com/
  28. None