Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
カオスエンジニアリングへの招待
Search
KyoheiSaiki
January 15, 2020
Programming
1
1.7k
カオスエンジニアリングへの招待
「AKIBA.AWS #15 ガチ編〜re:Invent振り返り その1〜」での発表資料です。
https://classmethod.connpass.com/event/159529/
KyoheiSaiki
January 15, 2020
Tweet
Share
More Decks by KyoheiSaiki
See All by KyoheiSaiki
Grafana OnCallによる通知
kyo1024
0
170
カオスエンジニアリングのススメ
kyo1024
1
1.6k
New Relic と Classmethod で実践するカオスエンジニアリング
kyo1024
0
2.5k
ANGEL_Dojo_最終発表_問題とミニブログで技術を学ぶ_エンジニア向け学習サービス_Loop_I_O.pdf
kyo1024
0
2.6k
問題とミニブログで技術を学ぶ エンジニア向け学習サービス Loop I/O
kyo1024
0
7k
Other Decks in Programming
See All in Programming
Startups on Rails in Past, Present and Future–Irina Nazarova, RailsConf 2025
irinanazarova
0
100
Python型ヒント完全ガイド 初心者でも分かる、現代的で実践的な使い方
mickey_kubo
1
110
LT 2025-06-30: プロダクトエンジニアの役割
yamamotok
0
760
#QiitaBash MCPのセキュリティ
ryosukedtomita
1
1.3k
猫と暮らす Google Nest Cam生活🐈 / WebRTC with Google Nest Cam
yutailang0119
0
120
なぜ適用するか、移行して理解するClean Architecture 〜構造を超えて設計を継承する〜 / Why Apply, Migrate and Understand Clean Architecture - Inherit Design Beyond Structure
seike460
PRO
3
760
VS Code Update for GitHub Copilot
74th
2
640
脱Riverpod?fqueryで考える、TanStack Queryライクなアーキテクチャの可能性
ostk0069
0
120
Quand Symfony, ApiPlatform, OpenAI et LangChain s'allient pour exploiter vos PDF : de la théorie à la production…
ahmedbhs123
0
190
PHPでWebSocketサーバーを実装しよう2025
kubotak
0
280
効率的な開発手段として VRTを活用する
ishkawa
0
140
PicoRuby on Rails
makicamel
2
130
Featured
See All Featured
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
46
9.6k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
The Pragmatic Product Professional
lauravandoore
35
6.7k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
107
19k
Being A Developer After 40
akosma
90
590k
RailsConf 2023
tenderlove
30
1.1k
The Cost Of JavaScript in 2023
addyosmani
51
8.5k
Raft: Consensus for Rubyists
vanstee
140
7k
Site-Speed That Sticks
csswizardry
10
690
How to Think Like a Performance Engineer
csswizardry
25
1.7k
Code Review Best Practice
trishagee
69
18k
Transcript
ΧΦεΤϯδχΞϦϯάͷট ࠤഢګฏ ,ZP ",*#""84ΨνฤʙSF*OWFOUৼΓฦΓͦͷʙ
8)0".* ࠤഢګฏʢ,ZPʣ w+0*/ w"84ࣄۀຊ෦ίϯαϧ෦ wιϦϡʔγϣϯΞʔΩςΫτ झຯɿ ϥΠϑαΠΤϯε
ԿͷͰ͔͢ʁ 2019/08/23
"84౦ژϦʔδϣϯͷେنো ͪͳΈʹೖࣾޙॳͷ͓٬༷ڥͰͷ࡞ۀ༧ఆͰͨ͠
"Everything fails, all the time" - Werner Vogels (CTO,
Amazon.com)
ࠓͷຊ ࣮ݧΛ௨ͯ͠γεςϜͷʢো࣌ͷʣৼΔ͍ʹ ࣗ৴Λ͚ͭΔ
ΧΦεΤϯδχΞϦϯά is Կ
ఆٛʢจݙΑΓʣ wΧΦεΤϯδχΞϦϯάɺຊ൪ڥͷߥΕڰͬͨঢ়ଶʹ͑ΔγεςϜͷೳྗ ʹࣗ৴Λ࣋ͨͤΔͨΊʹɺࢄγεςϜͰ࣮ݧ͢ΔֶͰ͢ Netflix paper ɻ wΧΦεΤϯδχΞϦϯάͷ࠷ऴతͳඪɺγεςϜͷ҆ఆੑͱճ෮ྗΛڧԽ͢ Δ͜ͱͰ͢ Gremlin
white paper ɻ resilience ʤපؾɾෆɾࠔɾۤڥͳͲ͔Βͷʥճ෮ྗɺཱͪ Δྗɺ෮׆ྗ ʤมܗ͞Ε͕ͨݩͷܗʹΔʥ෮ݩྗɺྗʢੑʣ ӳࣙPOUIF8&#ΑΓൈਮ
ఆٛʢͬ͘͟Γʣ wෳࡶͰڍಈ͕༧ෆೳͳʢࢄʣγεςϜʹରͯ͠ɺ োೖ ࣮ݧΛߦ͏͜ͱͰͦͷऑΛ͋Ϳ Γग़͢ ͋ͿΓग़ͨ͠ऑΛڧԽ͢Δ͜ͱͰΑΓճ෮ྗͷ͋ΔγεςϜΛߏங ऑͰͳ͔ͬͨ߹ɺͦ͜ʹࣗ৴͕࣋ͯΔ wোೖతͰͳ͘खஈ wߥͬΆ͍݈߁அͱ༧ࣹͷηοτͱ͍͏Πϝʔδ
Կ͕خ͍͠ͷʁ wͲΜͳγεςϜμϯͯ͠͠·͏ͱՁΛఏڙͰ͖ͳ͍ ճ෮ྗ্͕͕Δ͜ͱʢఔʹࠩ͋ΕͲʣ΄΅શͯͷγεςϜʹͱͬͯϋοϐʔ wചΓ্͛Լͷࢭ μϯλΠϜ͕ചΓ্͛ʹӨڹ͢Δ&$αΠτͳͲ wࣗͨͪͷϒϥϯυΠϝʔδΛकΔʢϓϥΠεϨεʣ
ͲΜͳاۀ͕ͬͯΔͷʁ ւ֎ ຊ
ΧΦεΤϯδχΞϦϯάͷݪଇ
w ௨ৗͷಈ࡞Λࣔ͢γεςϜͷଌఆՄೳͳग़ྗͱͯ͠ʮఆৗঢ়ଶʯΛఆٛ͢Δ͜ͱ͔Β࢝Ί·͢ w ͜ͷఆৗঢ়ଶɺରর܈͓Αͼ࣮ݧ܈ͷ྆ํͰܧଓ͢ΔͱԾఆ͠·͢ w αʔόʔͷΫϥογϡɺϋʔυυϥΠϒͷޡ࡞ಈɺωοτϫʔΫଓͷஅͳͲɺݱ࣮ੈքͷΠϕϯ τΛө͢ΔมΛಋೖ͠·͢ w ରর܈ͱ࣮ݧ܈ͱͷؒͷఆৗঢ়ଶͷҧ͍ΛௐΔ͜ͱʹΑͬͯԾઆΛূ͠Α͏ͱ͠·͢ ΧΦεΤϯδχΞϦϯάͷݪଇجຊݪଇ
https://principlesofchaos.org/?lang=ENcontent# ݕূʹ͓͚ΔΧΦε
ΧΦεΤϯδχΞϦϯάͷݪଇৄࡉͳݪଇ w ఆৗঢ়ଶʹ͓͚ΔৼΔ͍ͷԾઆΛཱͯΔ w ࣮ੈքͷࣄଟ༷Ͱ͋Δ w ຊ൪ڥͰݕূΛ࣮ߦ͢Δ w ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ
w ӨڹൣғΛ࠷খԽ͢Δ
ఆৗঢ়ଶJTԿ γεςϜ͕దʹಈ࡞ͯ͠ΔͬͯͲ͏͍͏͜ͱʁ
ఆৗঢ়ଶʹ͓͚ΔৼΔ͍ͷԾઆΛཱͯΔఆৗঢ়ଶͷѲ ϏδωεϝτϦΫεʹ w ސ٬Λࣦ͍ͬͯΔ͔ʁ w ސ٬αΠτͷॏཁͳػೳΛ࣮ߦͰ͖Δ͔ʁʢ&$αΠτͰΧʔτͷνΣοΫΞτՃͳͲʣ w ԆΛཧ༝ʹސ٬αʔϏεΛఀࢭ͢Δ͔ʁ w
ຖඵϏσΦͷετϦʔϛϯάΛ։࢝͢ΔϢʔβʔ 4USFBNJOHQFSTFDPOE 414 ࢀߟ w"NB[PO F#BZച্ w(PPHMF 'BDFCPPLࠂͷදࣔճ https://medium.com/netflix-techblog/sps-the-pulse-of-netflix-streaming-ae4db0e05f8a SPS
ఆৗঢ়ଶʹ͓͚ΔৼΔ͍ͷԾઆΛཱͯΔԾઆΛཱͯΔ • γεςϜʹΠϕϯτΛೖ͢Δͱɺఆৗঢ়ଶͷಈ࡞͕ͲͷΑ͏ʹมԽ͢Δ͔Λߟ͑Δ • ఆৗঢ়ଶதஅ͞ΕΔʁͦΕͱมԽ͠ͳ͍ʁ • தஅͷ߹ɺγεςϜग़ྗ͕૿Ճ·ͨݮগ͢Δͱ༧͍ͯ͠Δʁ w ೖ͢ΔΠϕϯτʹΑͬͯɺʮγεςϜͷಈ࡞͕ఆৗঢ়ଶ͔ΒมԽ͢Δ͜ͱͳ͍ʯͱ͍͏Ծઆͷͱ
࣮ݧΛ࣮ࢪ w ʢྫ Ϣʔβʔʹදࣔ͞ΕΔύʔιφϥΠζ͞ΕͨөըϦετΛੜ͢ΔαʔϏεͷࣦഊ γεςϜσϑΥϧτͷʢύʔιφϥΠζ͞Ε͍ͯͳ͍ʣөըϦετΛฦ͢ w طʹऑ͕͔͍ͬͯΔ߹࣮ݧΛߦ͏લʹमਖ਼͢Δ
࣮ੈքͷࣄଟ༷Ͱ͋Δ • ݱ࣮ͷੈքͰൃੜ͢ΔՄೳੑͷ͋Δೖྗ͔ΒΠϕϯτΛαϯϓϦϯάͯ͠ೖ • Ҏલʹൃੜͨ͠ͷ࠶ൃΛ͙ͨΊɺͦΕʹؔ͢ΔΠϕϯτΛར༻͢Δ͜ͱͱͯ༗༻ w ԾϚγϯΠϯελϯεΛऴྃ w αʔϏεؒͷϦΫΤετʹϨΠςϯγΛૠೖ
w αʔϏεؒͷϦΫΤετͷࣦഊ w ෦αʔϏε͕ࣦഊ w ಛఆͷ"84ϦʔδϣϯΛ༻ෆೳʹ Chaos Kong
ຊ൪ڥͰݕূΛ࣮ߦ͢Δ ΧΦεΤϯδχΞϦϯάͰຊ൪ڥͰͷγεςϜશମͷ;Δ·͍ΛΓ͍ͨɻ ։ൃڥͰҎԼΛຊ൪ڥͱಉҰʹ͢Δ͜ͱ͍͠ɻ • γεςϜͷ࣋ͭσʔλεςʔτ • Ϣʔβ͔Βͷ༧֎ͷೖྗ • ଓ͍ͯ͠Δ֎෦αʔϏε
➡ DNS, ֎෦SaaS etc. • ϋΠϖʔεͰʑߋ৽͞Εଓ͚Δίʔυ w ຊ൪ڥͰݕূΛ࣮ߦ
ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ • ॳΊͷҰճΛखಈͰ࣮ݧΛߦͬͯΈΔͷ࣮ݧ͕ਖ਼͘͠ߦ͑Δ͔Λ֬ೝ͢Δͷʹྑ͍ • ҰํͰɺݱͷγεςϜͷෳࡶ͞ɺͲͷมߋ͕࣮ݧͷ݁ՌΛม͑Δ͔༧ଌͰ͖ͳ͍ ➡ ͯ͢ͷมߋ͕݁ՌʹӨڹ͢Δͱߟ͑Δ • લ߲Ͱड़ͨͱ͓Γɺຊ൪ڥৗʹมԽ
➡ ࣮ݧ݁Ռ͕୲อ͢Δ৴པੑ࣌ؒͱͱʹݮਰ • ΤϯδχΞϦϯάϦιʔε༗ݶ ➡ खಈ࣮ݧʹΑΔ։ൃεϐʔυͷԼຊస w ΧΦε࣮ݧࣗಈԽͷͨΊͷπʔϧͱϓϥοτϑΥʔϜͷ࡞
ӨڹൣғΛ࠷খԽ͢Δ • ຊ൪ڥͰͷ࣮ݧɺސ٬ʹ௧ΈΛҾ͖ى͜͢Մೳੑ͕͋Δ • ࣮ݧ͔ΒͷμϝʔδΛ࠷খݶʹ͑Δ͜ͱΧΦεΤϯδχΞͷٛ w ސ٬͕γεςϜΛར༻Ͱ͖ͳ͘ͳͬͨ߹ɺ͙͢ʹ࣮ݧΛऴྃ ۓٸఀࢭϘλϯͷ࣮
ಛఆ݅ʹΑΔࣗಈऴྃ w ҟৗ͕ൃੜͨ͠ࡍʹૉૣ͍ରԠ Ӧۀ࣌ؒதʹͷΈ࣮ݧΛ࣮ࢪ
োೖπʔϧ
Tool (N)umber of attack types available Enterprise support Can
halt attack in progress GUI Open source Software as a service (SaaS) Cloud agnostic Can attack container / pod Can attack using k8s primitives Can attack serverless Has API Bloomberg Powerful Seal 1 ✔ ✔ ✔ ✔ ✔ ChaosBlade 4 ✔ ✔ ✔ ChaosIQ Chaos Toolkit 3 Via vendor ✔ ✔ ✔ ✔ ✔ ✔ Gremlin Free 2 ✔ ✔ ✔ ✔ ✔ ✔ ✔ Gremlin ILFI 11 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Gremlin ALFI 1 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Istio 2 ✔ ✔ ✔ ✔ Netflix Chaos Monkey 1 ✔ Netflix Simian Army 2 ✔ New Relic Chaos Panda 2 ✔ Nutanix X- Ray 3 ✔ ✔ Pumba 6 ✔ ✔ ✔ https://www.gremlin.com/chaos-engineering/chaos-engineering-tools-comparison/ োೖπʔϧ݁ߏ๛
(SFNMJOɺࣾͰྲྀߦͬͯ·͢ • ݩNetflixͷΧΦεΤϯδχΞKolton AndrusΒʹΑΔSaaS • ʢEC2ͷ߹ʣ15ఔͰΧΦε࣮ݧ͕࢝ΊΒΕΔखܰ͞ • ࢹπʔϧSlackͱͷ౷߹؆୯ɻۓٸఀࢭϘλϯ࣮ࡁΈ
ΧΦεΤϯδχΞϦϯάͷ࢝Ίํʁ
࢝ΊΑ͏ͱ͢Δͱʜ w ຊ൪Ͱͷ࣮ݧϦεΫߴͯ͘ා͍ɻຊ൪ͰΓͨ͘ͳ͍ʜ w Ͱ࣮ݧͷͨΊ͚ͩͷڥΛҡ࣋͢Δͷେมͩ͠ɺαʔυ ύʔςΟ࿈ܞຊ൪ͱҧ͏ͱ͜Ζଟ͍ʜ োൃੜ࣌ͷϏδωεΠϯύΫτ w ػձଛࣦ
w ചΓ্͛ w ϒϥϯυΠϝʔδ w ิঈඅ༻ ΧΦεΤϯδχΞϦϯάಋೖίετ w ࣄނൃੜͷϦεΫ w ਓతϦιʔε w ৺ཧతϋʔυϧ VS ܦݧ͕ͳ͍ͱΧΦεΤϯδχΞϦϯάͷίετͷݟੵΓ͍͠ʜ ✖ ൃੜ֬
(BNFEBZͷεεϝ "84(BNFEBZ w ͞Εͨ"84ڥʹͲΜͲΜো͕ൃੜ͢ΔͷͰɺͦΕΛൃݟ͠मਖ਼ w ࣌ؒͰେྔͷোʹରԠ͢Δ͜ͱʹͳΔͷͰຊͷҙຯͰ"84ͷཧղ͕ࢼ͞ΕΔ w Ϋϥϝιࣾ(BNFEBZΛݱࡏاըதɻ͠धཁ͕͋Ε֎෦͚ʹʜ •
҆શͳڥͰͷΧΦεΤϯδχΞϦϯάͷ࿅श • είʔϓΛߜͬͯ࣌ؒఔɺোೖΛ࣮ࢪ • ࣗࣾͰग़དྷΔͷ͕ϕλʔ͕ͩAWS Gameday͋Δ https://awsgamedaymicroservicestokyo.splashthat.com/
None