カオスエンジニアリングへの招待

「AKIBA.AWS #15 ガチ編〜re:Invent振り返り その1〜」での発表資料です。
https://classmethod.connpass.com/event/159529/

Transcript

1. ΧΦεΤϯδχΞϦϯά΁ͷট଴ 
   ࠤഢګฏ
   ,ZP ",*#""84
   
   ΨνฤʙSF*OWFOUৼΓฦΓ
   ͦͷʙ
2. 8)0
   ".
   * 
   ࠤഢګฏʢ,ZPʣ
   w
   +0*/
   w"84ࣄۀຊ෦ίϯαϧ෦
   wιϦϡʔγϣϯΞʔΩςΫτ झຯɿ
   ϥΠϑαΠΤϯε
3. Կͷ೔Ͱ͔͢ʁ  2019/08/23
4. "84౦ژϦʔδϣϯͷେن໛ো֐  ͪͳΈʹೖࣾޙॳͷ͓٬༷؀ڥͰͷ࡞ۀ༧ఆ೔Ͱͨ͠
5.  "Everything fails, all the time" - Werner Vogels (CTO, Amazon.com)
6. ࠓ೔ͷຊ୊  ࣮ݧΛ௨ͯ͠γεςϜͷʢো֐࣌ͷʣৼΔ෣͍ʹ ࣗ৴Λ͚ͭΔ
7.  ΧΦεΤϯδχΞϦϯά is Կ
8. ఆٛʢจݙΑΓʣ  wΧΦεΤϯδχΞϦϯά͸ɺຊ൪؀ڥͷߥΕڰͬͨঢ়ଶʹ଱͑ΔγεςϜͷೳྗ ʹࣗ৴Λ࣋ͨͤΔͨΊʹɺ෼ࢄγεςϜͰ࣮ݧ͢Δֶ໰Ͱ͢ Netflix paper ɻ
   wΧΦεΤϯδχΞϦϯάͷ࠷ऴతͳ໨ඪ͸ɺγεςϜͷ҆ఆੑͱճ෮ྗΛڧԽ͢ Δ͜ͱͰ͢ Gremlin white paper ɻ resilience  ʤපؾɾෆ޾ɾࠔ೉ɾۤڥͳͲ͔Βͷʥճ෮ྗɺཱͪ ௚Δྗɺ෮׆ྗ
    ʤมܗ͞Εͨ෺͕ݩͷܗʹ໭Δʥ෮ݩྗɺ஄ྗʢੑʣ
   ӳࣙ࿠
   PO
   UIF
   8&#ΑΓൈਮ
9. ఆٛʢͬ͘͟Γʣ  wෳࡶͰڍಈ͕༧૝ෆೳͳʢ෼ࢄʣγεςϜʹରͯ͠ɺ ো֐஫ೖ ࣮ݧΛߦ͏͜ͱͰͦͷऑ఺Λ͋Ϳ Γग़͢
   ‎
   ͋ͿΓग़ͨ͠ऑ఺ΛڧԽ͢Δ͜ͱͰΑΓճ෮ྗͷ͋ΔγεςϜΛߏங
   ‎
   ऑ఺Ͱͳ͔ͬͨ৔߹ɺͦ͜ʹ͸ࣗ৴͕࣋ͯΔ
   wো֐஫ೖ͸໨తͰ͸ͳ͘खஈ
   wߥͬΆ͍݈߁਍அͱ༧๷஫ࣹͷηοτͱ͍͏Πϝʔδ
10. Կ͕خ͍͠ͷʁ  wͲΜͳγεςϜ΋μ΢ϯͯ͠͠·͏ͱՁ஋ΛఏڙͰ͖ͳ͍
    ‎
    ճ෮ྗ্͕͕Δ͜ͱ͸ʢఔ౓ʹࠩ͸͋ΕͲʣ΄΅શͯͷγεςϜʹͱͬͯϋοϐʔ
    wചΓ্͛௿Լͷ๷ࢭ
    ‎
    μ΢ϯλΠϜ͕ചΓ্͛ʹ௚઀Өڹ͢Δ&$αΠτͳͲ
    wࣗ෼ͨͪͷϒϥϯυΠϝʔδΛकΔʢϓϥΠεϨεʣ
11. ͲΜͳاۀ͕΍ͬͯΔͷʁ  ւ֎ ೔ຊ
12.  ΧΦεΤϯδχΞϦϯάͷݪଇ
13. w ௨ৗͷಈ࡞Λࣔ͢γεςϜͷଌఆՄೳͳग़ྗͱͯ͠ʮఆৗঢ়ଶʯΛఆٛ͢Δ͜ͱ͔Β࢝Ί·͢
    w ͜ͷఆৗঢ়ଶ͸ɺରর܈͓Αͼ࣮ݧ܈ͷ྆ํͰܧଓ͢ΔͱԾఆ͠·͢
    w αʔόʔͷΫϥογϡɺϋʔυυϥΠϒͷޡ࡞ಈɺωοτϫʔΫ઀ଓͷ੾அͳͲɺݱ࣮ੈքͷΠϕϯ τΛ൓ө͢Δม਺Λಋೖ͠·͢
    w ରর܈ͱ࣮ݧ܈ͱͷؒͷఆৗঢ়ଶͷҧ͍Λௐ΂Δ͜ͱʹΑͬͯԾઆΛ൓ূ͠Α͏ͱ͠·͢ ΧΦεΤϯδχΞϦϯάͷݪଇ
    جຊݪଇ  https://principlesofchaos.org/?lang=ENcontent# ݕূʹ͓͚ΔΧΦε
14. ΧΦεΤϯδχΞϦϯάͷݪଇ
    ৄࡉͳݪଇ  w ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔ
    w ࣮ੈքͷࣄ৅͸ଟ༷Ͱ͋Δ
    w ຊ൪؀ڥͰݕূΛ࣮ߦ͢Δ
    w ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ
    w ӨڹൣғΛ࠷খԽ͢Δ
15. ఆৗঢ়ଶ
    JT
    Կ  γεςϜ͕ద੾ʹಈ࡞ͯ͠ΔͬͯͲ͏͍͏͜ͱʁ
16. ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔ
    ఆৗঢ়ଶͷ೺Ѳ  ϏδωεϝτϦΫεʹ஫໨
    w ސ٬Λࣦ͍ͬͯΔ͔ʁ
    w ސ٬͸αΠτͷॏཁͳػೳΛ࣮ߦͰ͖Δ͔ʁʢ&$αΠτͰΧʔτͷνΣοΫΞ΢τ΍௥ՃͳͲʣ
    w ஗ԆΛཧ༝ʹސ٬͸αʔϏεΛఀࢭ͢Δ͔ʁ w ຖඵϏσΦͷετϦʔϛϯάΛ։࢝͢ΔϢʔβʔ਺
    ‎
    4USFBNJOH
    QFS
    TFDPOE
    414
    ࢀߟ
    w"NB[PO
    F#BZ
    ച্
    w(PPHMF
    'BDFCPPL
    ޿ࠂͷදࣔճ਺ https://medium.com/netflix-techblog/sps-the-pulse-of-netflix-streaming-ae4db0e05f8a SPS
17. ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔ
    ԾઆΛཱͯΔ  • γεςϜʹΠϕϯτΛ஫ೖ͢Δͱɺఆৗঢ়ଶͷಈ࡞͕ͲͷΑ͏ʹมԽ͢Δ͔Λߟ͑Δ • ఆৗঢ়ଶ͸தஅ͞ΕΔʁͦΕͱ΋มԽ͠ͳ͍ʁ • தஅͷ৔߹ɺγεςϜग़ྗ͕૿Ճ·ͨ͸ݮগ͢Δͱ༧૝͍ͯ͠Δʁ w ஫ೖ͢ΔΠϕϯτʹΑͬͯɺʮγεςϜͷಈ࡞͕ఆৗঢ়ଶ͔ΒมԽ͢Δ͜ͱ͸ͳ͍ʯͱ͍͏Ծઆͷ΋ͱ ࣮ݧΛ࣮ࢪ
    w ʢྫ Ϣʔβʔʹදࣔ͞ΕΔύʔιφϥΠζ͞ΕͨөըϦετΛੜ੒͢ΔαʔϏεͷࣦഊ
    ‎ γεςϜ͸σϑΥϧτͷʢύʔιφϥΠζ͞Ε͍ͯͳ͍ʣөըϦετΛฦ͢
    w طʹऑ఺͕෼͔͍ͬͯΔ৔߹͸࣮ݧΛߦ͏લʹमਖ਼͢Δ
18. ࣮ੈքͷࣄ৅͸ଟ༷Ͱ͋Δ  • ݱ࣮ͷੈքͰൃੜ͢ΔՄೳੑͷ͋Δೖྗ͔ΒΠϕϯτΛαϯϓϦϯάͯ͠஫ೖ • Ҏલʹൃੜͨ͠໰୊ͷ࠶ൃΛ๷͙ͨΊɺͦΕʹؔ܎͢ΔΠϕϯτΛར༻͢Δ͜ͱ͸ͱͯ΋༗༻ w Ծ૝ϚγϯΠϯελϯεΛऴྃ
    w αʔϏεؒͷϦΫΤετʹϨΠςϯγΛૠೖ
    w αʔϏεؒͷϦΫΤετͷࣦഊ
    w ಺෦αʔϏε͕ࣦഊ
    w ಛఆͷ"84ϦʔδϣϯΛ࢖༻ෆೳʹ Chaos Kong
19. ຊ൪؀ڥͰݕূΛ࣮ߦ͢Δ  ΧΦεΤϯδχΞϦϯάͰ͸ຊ൪؀ڥͰͷγεςϜશମͷ;Δ·͍Λ஌Γ͍ͨɻ ։ൃ؀ڥͰ͸ҎԼΛຊ൪؀ڥͱಉҰʹ͢Δ͜ͱ͸೉͍͠ɻ • γεςϜͷ࣋ͭσʔλ΍εςʔτ • Ϣʔβ͔Βͷ༧૝֎ͷೖྗ • ઀ଓ͍ͯ͠Δ֎෦αʔϏε ➡ DNS, ֎෦SaaS etc. • ϋΠϖʔεͰ೔ʑߋ৽͞Εଓ͚Δίʔυ w ຊ൪؀ڥͰݕূΛ࣮ߦ
20. ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ  • ॳΊͷҰճΛखಈͰ࣮ݧΛߦͬͯΈΔͷ͸࣮ݧ͕ਖ਼͘͠ߦ͑Δ͔Λ֬ೝ͢Δͷʹྑ͍ • ҰํͰɺݱ୅ͷγεςϜͷෳࡶ͞͸ɺͲͷมߋ͕࣮ݧͷ݁ՌΛม͑Δ͔༧ଌͰ͖ͳ͍ ➡ ͢΂ͯͷมߋ͕݁ՌʹӨڹ͢Δͱߟ͑Δ • લ߲Ͱड़΂ͨͱ͓Γɺຊ൪؀ڥ͸ৗʹมԽ ➡ ࣮ݧ݁Ռ͕୲อ͢Δ৴པੑ͸࣌ؒͱͱ΋ʹݮਰ • ΤϯδχΞϦϯάϦιʔε͸༗ݶ ➡ खಈ࣮ݧʹΑΔ։ൃεϐʔυͷ௿Լ͸ຊ຤స౗ w ΧΦε࣮ݧࣗಈԽͷͨΊͷπʔϧͱϓϥοτϑΥʔϜͷ࡞੒
21. ӨڹൣғΛ࠷খԽ͢Δ  • ຊ൪؀ڥͰͷ࣮ݧ͸ɺސ٬ʹ௧ΈΛҾ͖ى͜͢Մೳੑ͕͋Δ • ࣮ݧ͔ΒͷμϝʔδΛ࠷খݶʹ཈͑Δ͜ͱ͸ΧΦεΤϯδχΞͷٛ຿ w ސ٬͕γεςϜΛར༻Ͱ͖ͳ͘ͳͬͨ৔߹͸ɺ͙͢ʹ࣮ݧΛऴྃ
    ‎ ۓٸఀࢭϘλϯͷ࣮૷
    ‎ ಛఆ৚݅ʹΑΔࣗಈऴྃ
    w ҟৗ͕ൃੜͨ͠ࡍʹ͸ૉૣ͍ରԠ
    ‎ Ӧۀ࣌ؒதʹͷΈ࣮ݧΛ࣮ࢪ
22.  ো֐஫ೖπʔϧ
23.  Tool (N)umber of attack types available Enterprise support Can halt attack in progress GUI Open source Software as a service (SaaS) Cloud agnostic Can attack container / pod Can attack using k8s primitives Can attack serverless Has API Bloomberg Powerful Seal 1 ✔ ✔ ✔ ✔ ✔ ChaosBlade 4 ✔ ✔ ✔ ChaosIQ Chaos Toolkit 3 Via vendor ✔ ✔ ✔ ✔ ✔ ✔ Gremlin Free 2 ✔ ✔ ✔ ✔ ✔ ✔ ✔ Gremlin ILFI 11 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Gremlin ALFI 1 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Istio 2 ✔ ✔ ✔ ✔ Netflix Chaos Monkey 1 ✔ Netflix Simian Army 2 ✔ New Relic Chaos Panda 2 ✔ Nutanix X- Ray 3 ✔ ✔ Pumba 6 ✔ ✔ ✔ https://www.gremlin.com/chaos-engineering/chaos-engineering-tools-comparison/ ো֐஫ೖπʔϧ͸݁ߏ๛෋
24. (SFNMJOɺࣾ಺Ͱྲྀߦͬͯ·͢  • ݩNetflixͷΧΦεΤϯδχΞKolton AndrusΒʹΑΔSaaS • ʢEC2ͷ৔߹ʣ15෼ఔ౓ͰΧΦε࣮ݧ͕࢝ΊΒΕΔखܰ͞ • ؂ࢹπʔϧ΍Slackͱͷ౷߹΋؆୯ɻۓٸఀࢭϘλϯ΋࣮૷ࡁΈ
25.  ΧΦεΤϯδχΞϦϯάͷ࢝Ίํʁ
26. ࢝ΊΑ͏ͱ͢Δͱʜ  w ຊ൪Ͱͷ࣮ݧ͸ϦεΫߴͯ͘ා͍ɻຊ൪Ͱ͸΍Γͨ͘ͳ͍ʜ
    w Ͱ΋࣮ݧͷͨΊ͚ͩͷ؀ڥΛҡ࣋͢Δͷ΋େมͩ͠ɺαʔυ ύʔςΟ࿈ܞ౳ຊ൪ͱҧ͏ͱ͜Ζ΋ଟ͍ʜ ো֐ൃੜ࣌ͷϏδωεΠϯύΫτ w ػձଛࣦ
    w ചΓ্͛
    w ϒϥϯυΠϝʔδ
    w ิঈඅ༻ ΧΦεΤϯδχΞϦϯάಋೖίετ w ࣄނൃੜͷϦεΫ
    w ਓతϦιʔε
    w ৺ཧతϋʔυϧ VS ܦݧ͕ͳ͍ͱΧΦεΤϯδχΞϦϯάͷίετͷݟੵ΋Γ͸೉͍͠ʜ ✖ ൃੜ֬཰
27. (BNFEBZͷεεϝ  "84
    (BNFEBZ
    w ౉͞Εͨ"84؀ڥʹͲΜͲΜো֐͕ൃੜ͢ΔͷͰɺͦΕΛൃݟ͠मਖ਼
    w ୹࣌ؒͰେྔͷো֐ʹରԠ͢Δ͜ͱʹͳΔͷͰຊ౰ͷҙຯͰ"84΁ͷཧղ͕ࢼ͞ΕΔ
    w Ϋϥϝι΋ࣾ಺(BNFEBZΛݱࡏاըதɻ΋͠धཁ͕͋Ε͹֎෦޲͚ʹ΋ʜ • ҆શͳ؀ڥͰͷΧΦεΤϯδχΞϦϯάͷ࿅श • είʔϓΛߜͬͯ਺࣌ؒఔ౓ɺো֐஫ೖΛ࣮ࢪ • ࣗࣾͰग़དྷΔͷ͕ϕλʔ͕ͩAWS Gameday΋͋Δ https://awsgamedaymicroservicestokyo.splashthat.com/