Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
カオスエンジニアリングへの招待
KyoheiSaiki
January 15, 2020
Programming
1
970
カオスエンジニアリングへの招待
「AKIBA.AWS #15 ガチ編〜re:Invent振り返り その1〜」での発表資料です。
https://classmethod.connpass.com/event/159529/
KyoheiSaiki
January 15, 2020
Tweet
Share
More Decks by KyoheiSaiki
See All by KyoheiSaiki
カオスエンジニアリングのススメ
kyo1024
1
910
New Relic と Classmethod で実践するカオスエンジニアリング
kyo1024
0
1.3k
ANGEL_Dojo_最終発表_問題とミニブログで技術を学ぶ_エンジニア向け学習サービス_Loop_I_O.pdf
kyo1024
0
1.8k
問題とミニブログで技術を学ぶ エンジニア向け学習サービス Loop I/O
kyo1024
0
3.6k
Other Decks in Programming
See All in Programming
Jakarta EE 10 and Beyond
ivargrimstad
0
1.8k
Vite でお手軽 Vue.js の環境構築
azuki
2
170
Amazon Aurora の v1 が EOL になるので 10 クラスタアップグレードして出てきたノウハウ
dekokun
0
840
Maintaining Software Correctness
dlew
PRO
3
240
Reactアプリケーションのテスト戦略
0906koki
10
4.6k
Jetpack Compose best practices 動画紹介 @GoogleI/O LT会
takakitojo
0
270
What's new in Android development tools まとめ
mkeeda
0
280
PythonユーザによるRust入門
rmizuta3
9
3.1k
短納期でローンチした新サービスをJavaで開発した話/launched new service using Java
eichisanden
6
1.9k
Power Automateドリブンのチームマネジメント
hanaseleb
0
180
io22 extended What's new in app performance
veronikapj
0
330
Beyond Micro Frontends: Frontend Moduliths for the Enterprise @wad2022
manfredsteyer
PRO
0
130
Featured
See All Featured
GraphQLの誤解/rethinking-graphql
sonatard
27
6.5k
The Pragmatic Product Professional
lauravandoore
19
2.9k
The Cult of Friendly URLs
andyhume
68
4.8k
Raft: Consensus for Rubyists
vanstee
126
5.4k
Embracing the Ebb and Flow
colly
73
3.4k
Art Directing for the Web. Five minutes with CSS Template Areas
malarkey
196
9.4k
Designing the Hi-DPI Web
ddemaree
272
32k
Docker and Python
trallard
27
1.6k
Designing for Performance
lara
597
63k
Large-scale JavaScript Application Architecture
addyosmani
499
110k
What the flash - Photography Introduction
edds
61
10k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
15
940
Transcript
ΧΦεΤϯδχΞϦϯάͷট ࠤഢګฏ ,ZP ",*#""84ΨνฤʙSF*OWFOUৼΓฦΓͦͷʙ
8)0".* ࠤഢګฏʢ,ZPʣ w+0*/ w"84ࣄۀຊ෦ίϯαϧ෦ wιϦϡʔγϣϯΞʔΩςΫτ झຯɿ ϥΠϑαΠΤϯε
ԿͷͰ͔͢ʁ 2019/08/23
"84౦ژϦʔδϣϯͷେنো ͪͳΈʹೖࣾޙॳͷ͓٬༷ڥͰͷ࡞ۀ༧ఆͰͨ͠
"Everything fails, all the time" - Werner Vogels (CTO,
Amazon.com)
ࠓͷຊ ࣮ݧΛ௨ͯ͠γεςϜͷʢো࣌ͷʣৼΔ͍ʹ ࣗ৴Λ͚ͭΔ
ΧΦεΤϯδχΞϦϯά is Կ
ఆٛʢจݙΑΓʣ wΧΦεΤϯδχΞϦϯάɺຊ൪ڥͷߥΕڰͬͨঢ়ଶʹ͑ΔγεςϜͷೳྗ ʹࣗ৴Λ࣋ͨͤΔͨΊʹɺࢄγεςϜͰ࣮ݧ͢ΔֶͰ͢ Netflix paper ɻ wΧΦεΤϯδχΞϦϯάͷ࠷ऴతͳඪɺγεςϜͷ҆ఆੑͱճ෮ྗΛڧԽ͢ Δ͜ͱͰ͢ Gremlin
white paper ɻ resilience ʤපؾɾෆɾࠔɾۤڥͳͲ͔Βͷʥճ෮ྗɺཱͪ Δྗɺ෮׆ྗ ʤมܗ͞Ε͕ͨݩͷܗʹΔʥ෮ݩྗɺྗʢੑʣ ӳࣙPOUIF8&#ΑΓൈਮ
ఆٛʢͬ͘͟Γʣ wෳࡶͰڍಈ͕༧ෆೳͳʢࢄʣγεςϜʹରͯ͠ɺ োೖ ࣮ݧΛߦ͏͜ͱͰͦͷऑΛ͋Ϳ Γग़͢ ͋ͿΓग़ͨ͠ऑΛڧԽ͢Δ͜ͱͰΑΓճ෮ྗͷ͋ΔγεςϜΛߏங ऑͰͳ͔ͬͨ߹ɺͦ͜ʹࣗ৴͕࣋ͯΔ wোೖతͰͳ͘खஈ wߥͬΆ͍݈߁அͱ༧ࣹͷηοτͱ͍͏Πϝʔδ
Կ͕خ͍͠ͷʁ wͲΜͳγεςϜμϯͯ͠͠·͏ͱՁΛఏڙͰ͖ͳ͍ ճ෮ྗ্͕͕Δ͜ͱʢఔʹࠩ͋ΕͲʣ΄΅શͯͷγεςϜʹͱͬͯϋοϐʔ wചΓ্͛Լͷࢭ μϯλΠϜ͕ചΓ্͛ʹӨڹ͢Δ&$αΠτͳͲ wࣗͨͪͷϒϥϯυΠϝʔδΛकΔʢϓϥΠεϨεʣ
ͲΜͳاۀ͕ͬͯΔͷʁ ւ֎ ຊ
ΧΦεΤϯδχΞϦϯάͷݪଇ
w ௨ৗͷಈ࡞Λࣔ͢γεςϜͷଌఆՄೳͳग़ྗͱͯ͠ʮఆৗঢ়ଶʯΛఆٛ͢Δ͜ͱ͔Β࢝Ί·͢ w ͜ͷఆৗঢ়ଶɺରর܈͓Αͼ࣮ݧ܈ͷ྆ํͰܧଓ͢ΔͱԾఆ͠·͢ w αʔόʔͷΫϥογϡɺϋʔυυϥΠϒͷޡ࡞ಈɺωοτϫʔΫଓͷஅͳͲɺݱ࣮ੈքͷΠϕϯ τΛө͢ΔมΛಋೖ͠·͢ w ରর܈ͱ࣮ݧ܈ͱͷؒͷఆৗঢ়ଶͷҧ͍ΛௐΔ͜ͱʹΑͬͯԾઆΛূ͠Α͏ͱ͠·͢ ΧΦεΤϯδχΞϦϯάͷݪଇجຊݪଇ
https://principlesofchaos.org/?lang=ENcontent# ݕূʹ͓͚ΔΧΦε
ΧΦεΤϯδχΞϦϯάͷݪଇৄࡉͳݪଇ w ఆৗঢ়ଶʹ͓͚ΔৼΔ͍ͷԾઆΛཱͯΔ w ࣮ੈքͷࣄଟ༷Ͱ͋Δ w ຊ൪ڥͰݕূΛ࣮ߦ͢Δ w ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ
w ӨڹൣғΛ࠷খԽ͢Δ
ఆৗঢ়ଶJTԿ γεςϜ͕దʹಈ࡞ͯ͠ΔͬͯͲ͏͍͏͜ͱʁ
ఆৗঢ়ଶʹ͓͚ΔৼΔ͍ͷԾઆΛཱͯΔఆৗঢ়ଶͷѲ ϏδωεϝτϦΫεʹ w ސ٬Λࣦ͍ͬͯΔ͔ʁ w ސ٬αΠτͷॏཁͳػೳΛ࣮ߦͰ͖Δ͔ʁʢ&$αΠτͰΧʔτͷνΣοΫΞτՃͳͲʣ w ԆΛཧ༝ʹސ٬αʔϏεΛఀࢭ͢Δ͔ʁ w
ຖඵϏσΦͷετϦʔϛϯάΛ։࢝͢ΔϢʔβʔ 4USFBNJOHQFSTFDPOE 414 ࢀߟ w"NB[PO F#BZച্ w(PPHMF 'BDFCPPLࠂͷදࣔճ https://medium.com/netflix-techblog/sps-the-pulse-of-netflix-streaming-ae4db0e05f8a SPS
ఆৗঢ়ଶʹ͓͚ΔৼΔ͍ͷԾઆΛཱͯΔԾઆΛཱͯΔ • γεςϜʹΠϕϯτΛೖ͢Δͱɺఆৗঢ়ଶͷಈ࡞͕ͲͷΑ͏ʹมԽ͢Δ͔Λߟ͑Δ • ఆৗঢ়ଶதஅ͞ΕΔʁͦΕͱมԽ͠ͳ͍ʁ • தஅͷ߹ɺγεςϜग़ྗ͕૿Ճ·ͨݮগ͢Δͱ༧͍ͯ͠Δʁ w ೖ͢ΔΠϕϯτʹΑͬͯɺʮγεςϜͷಈ࡞͕ఆৗঢ়ଶ͔ΒมԽ͢Δ͜ͱͳ͍ʯͱ͍͏Ծઆͷͱ
࣮ݧΛ࣮ࢪ w ʢྫ Ϣʔβʔʹදࣔ͞ΕΔύʔιφϥΠζ͞ΕͨөըϦετΛੜ͢ΔαʔϏεͷࣦഊ γεςϜσϑΥϧτͷʢύʔιφϥΠζ͞Ε͍ͯͳ͍ʣөըϦετΛฦ͢ w طʹऑ͕͔͍ͬͯΔ߹࣮ݧΛߦ͏લʹमਖ਼͢Δ
࣮ੈքͷࣄଟ༷Ͱ͋Δ • ݱ࣮ͷੈքͰൃੜ͢ΔՄೳੑͷ͋Δೖྗ͔ΒΠϕϯτΛαϯϓϦϯάͯ͠ೖ • Ҏલʹൃੜͨ͠ͷ࠶ൃΛ͙ͨΊɺͦΕʹؔ͢ΔΠϕϯτΛར༻͢Δ͜ͱͱͯ༗༻ w ԾϚγϯΠϯελϯεΛऴྃ w αʔϏεؒͷϦΫΤετʹϨΠςϯγΛૠೖ
w αʔϏεؒͷϦΫΤετͷࣦഊ w ෦αʔϏε͕ࣦഊ w ಛఆͷ"84ϦʔδϣϯΛ༻ෆೳʹ Chaos Kong
ຊ൪ڥͰݕূΛ࣮ߦ͢Δ ΧΦεΤϯδχΞϦϯάͰຊ൪ڥͰͷγεςϜશମͷ;Δ·͍ΛΓ͍ͨɻ ։ൃڥͰҎԼΛຊ൪ڥͱಉҰʹ͢Δ͜ͱ͍͠ɻ • γεςϜͷ࣋ͭσʔλεςʔτ • Ϣʔβ͔Βͷ༧֎ͷೖྗ • ଓ͍ͯ͠Δ֎෦αʔϏε
➡ DNS, ֎෦SaaS etc. • ϋΠϖʔεͰʑߋ৽͞Εଓ͚Δίʔυ w ຊ൪ڥͰݕূΛ࣮ߦ
ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ • ॳΊͷҰճΛखಈͰ࣮ݧΛߦͬͯΈΔͷ࣮ݧ͕ਖ਼͘͠ߦ͑Δ͔Λ֬ೝ͢Δͷʹྑ͍ • ҰํͰɺݱͷγεςϜͷෳࡶ͞ɺͲͷมߋ͕࣮ݧͷ݁ՌΛม͑Δ͔༧ଌͰ͖ͳ͍ ➡ ͯ͢ͷมߋ͕݁ՌʹӨڹ͢Δͱߟ͑Δ • લ߲Ͱड़ͨͱ͓Γɺຊ൪ڥৗʹมԽ
➡ ࣮ݧ݁Ռ͕୲อ͢Δ৴པੑ࣌ؒͱͱʹݮਰ • ΤϯδχΞϦϯάϦιʔε༗ݶ ➡ खಈ࣮ݧʹΑΔ։ൃεϐʔυͷԼຊస w ΧΦε࣮ݧࣗಈԽͷͨΊͷπʔϧͱϓϥοτϑΥʔϜͷ࡞
ӨڹൣғΛ࠷খԽ͢Δ • ຊ൪ڥͰͷ࣮ݧɺސ٬ʹ௧ΈΛҾ͖ى͜͢Մೳੑ͕͋Δ • ࣮ݧ͔ΒͷμϝʔδΛ࠷খݶʹ͑Δ͜ͱΧΦεΤϯδχΞͷٛ w ސ٬͕γεςϜΛར༻Ͱ͖ͳ͘ͳͬͨ߹ɺ͙͢ʹ࣮ݧΛऴྃ ۓٸఀࢭϘλϯͷ࣮
ಛఆ݅ʹΑΔࣗಈऴྃ w ҟৗ͕ൃੜͨ͠ࡍʹૉૣ͍ରԠ Ӧۀ࣌ؒதʹͷΈ࣮ݧΛ࣮ࢪ
োೖπʔϧ
Tool (N)umber of attack types available Enterprise support Can
halt attack in progress GUI Open source Software as a service (SaaS) Cloud agnostic Can attack container / pod Can attack using k8s primitives Can attack serverless Has API Bloomberg Powerful Seal 1 ✔ ✔ ✔ ✔ ✔ ChaosBlade 4 ✔ ✔ ✔ ChaosIQ Chaos Toolkit 3 Via vendor ✔ ✔ ✔ ✔ ✔ ✔ Gremlin Free 2 ✔ ✔ ✔ ✔ ✔ ✔ ✔ Gremlin ILFI 11 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Gremlin ALFI 1 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Istio 2 ✔ ✔ ✔ ✔ Netflix Chaos Monkey 1 ✔ Netflix Simian Army 2 ✔ New Relic Chaos Panda 2 ✔ Nutanix X- Ray 3 ✔ ✔ Pumba 6 ✔ ✔ ✔ https://www.gremlin.com/chaos-engineering/chaos-engineering-tools-comparison/ োೖπʔϧ݁ߏ๛
(SFNMJOɺࣾͰྲྀߦͬͯ·͢ • ݩNetflixͷΧΦεΤϯδχΞKolton AndrusΒʹΑΔSaaS • ʢEC2ͷ߹ʣ15ఔͰΧΦε࣮ݧ͕࢝ΊΒΕΔखܰ͞ • ࢹπʔϧSlackͱͷ౷߹؆୯ɻۓٸఀࢭϘλϯ࣮ࡁΈ
ΧΦεΤϯδχΞϦϯάͷ࢝Ίํʁ
࢝ΊΑ͏ͱ͢Δͱʜ w ຊ൪Ͱͷ࣮ݧϦεΫߴͯ͘ා͍ɻຊ൪ͰΓͨ͘ͳ͍ʜ w Ͱ࣮ݧͷͨΊ͚ͩͷڥΛҡ࣋͢Δͷେมͩ͠ɺαʔυ ύʔςΟ࿈ܞຊ൪ͱҧ͏ͱ͜Ζଟ͍ʜ োൃੜ࣌ͷϏδωεΠϯύΫτ w ػձଛࣦ
w ചΓ্͛ w ϒϥϯυΠϝʔδ w ิঈඅ༻ ΧΦεΤϯδχΞϦϯάಋೖίετ w ࣄނൃੜͷϦεΫ w ਓతϦιʔε w ৺ཧతϋʔυϧ VS ܦݧ͕ͳ͍ͱΧΦεΤϯδχΞϦϯάͷίετͷݟੵΓ͍͠ʜ ✖ ൃੜ֬
(BNFEBZͷεεϝ "84(BNFEBZ w ͞Εͨ"84ڥʹͲΜͲΜো͕ൃੜ͢ΔͷͰɺͦΕΛൃݟ͠मਖ਼ w ࣌ؒͰେྔͷোʹରԠ͢Δ͜ͱʹͳΔͷͰຊͷҙຯͰ"84ͷཧղ͕ࢼ͞ΕΔ w Ϋϥϝιࣾ(BNFEBZΛݱࡏاըதɻ͠धཁ͕͋Ε֎෦͚ʹʜ •
҆શͳڥͰͷΧΦεΤϯδχΞϦϯάͷ࿅श • είʔϓΛߜͬͯ࣌ؒఔɺোೖΛ࣮ࢪ • ࣗࣾͰग़དྷΔͷ͕ϕλʔ͕ͩAWS Gameday͋Δ https://awsgamedaymicroservicestokyo.splashthat.com/
None