Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
カオスエンジニアリングへの招待
Search
KyoheiSaiki
January 15, 2020
Programming
1
1.7k
カオスエンジニアリングへの招待
「AKIBA.AWS #15 ガチ編〜re:Invent振り返り その1〜」での発表資料です。
https://classmethod.connpass.com/event/159529/
KyoheiSaiki
January 15, 2020
Tweet
Share
More Decks by KyoheiSaiki
See All by KyoheiSaiki
Grafana OnCallによる通知
kyo1024
0
170
カオスエンジニアリングのススメ
kyo1024
1
1.5k
New Relic と Classmethod で実践するカオスエンジニアリング
kyo1024
0
2.5k
ANGEL_Dojo_最終発表_問題とミニブログで技術を学ぶ_エンジニア向け学習サービス_Loop_I_O.pdf
kyo1024
0
2.6k
問題とミニブログで技術を学ぶ エンジニア向け学習サービス Loop I/O
kyo1024
0
7k
Other Decks in Programming
See All in Programming
deno-redisの紹介とJSRパッケージの運用について (toranoana.deno #21)
uki00a
0
150
なぜ「共通化」を考え、失敗を繰り返すのか
rinchoku
1
560
CursorはMCPを使った方が良いぞ
taigakono
1
190
来たるべき 8.0 に備えて React 19 新機能と React Router 固有機能の取捨選択とすり合わせを考える
oukayuka
2
860
20250628_非エンジニアがバイブコーディングしてみた
ponponmikankan
0
470
datadog dash 2025 LLM observability for reliability and stability
ivry_presentationmaterials
0
120
High-Level Programming Languages in AI Era -Human Thought and Mind-
hayat01sh1da
PRO
0
540
GoのGenericsによるslice操作との付き合い方
syumai
3
690
Result型で“失敗”を型にするPHPコードの書き方
kajitack
4
500
Azure AI Foundryではじめてのマルチエージェントワークフロー
seosoft
0
140
地方に住むエンジニアの残酷な現実とキャリア論
ichimichi
5
1.4k
関数型まつり2025登壇資料「関数プログラミングと再帰」
taisontsukada
2
850
Featured
See All Featured
Designing Experiences People Love
moore
142
24k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
281
13k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
It's Worth the Effort
3n
185
28k
Stop Working from a Prison Cell
hatefulcrawdad
270
20k
Code Review Best Practice
trishagee
68
18k
The Cult of Friendly URLs
andyhume
79
6.5k
How to Think Like a Performance Engineer
csswizardry
24
1.7k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.7k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.2k
Practical Orchestrator
shlominoach
188
11k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.6k
Transcript
ΧΦεΤϯδχΞϦϯάͷট ࠤഢګฏ ,ZP ",*#""84ΨνฤʙSF*OWFOUৼΓฦΓͦͷʙ
8)0".* ࠤഢګฏʢ,ZPʣ w+0*/ w"84ࣄۀຊ෦ίϯαϧ෦ wιϦϡʔγϣϯΞʔΩςΫτ झຯɿ ϥΠϑαΠΤϯε
ԿͷͰ͔͢ʁ 2019/08/23
"84౦ژϦʔδϣϯͷେنো ͪͳΈʹೖࣾޙॳͷ͓٬༷ڥͰͷ࡞ۀ༧ఆͰͨ͠
"Everything fails, all the time" - Werner Vogels (CTO,
Amazon.com)
ࠓͷຊ ࣮ݧΛ௨ͯ͠γεςϜͷʢো࣌ͷʣৼΔ͍ʹ ࣗ৴Λ͚ͭΔ
ΧΦεΤϯδχΞϦϯά is Կ
ఆٛʢจݙΑΓʣ wΧΦεΤϯδχΞϦϯάɺຊ൪ڥͷߥΕڰͬͨঢ়ଶʹ͑ΔγεςϜͷೳྗ ʹࣗ৴Λ࣋ͨͤΔͨΊʹɺࢄγεςϜͰ࣮ݧ͢ΔֶͰ͢ Netflix paper ɻ wΧΦεΤϯδχΞϦϯάͷ࠷ऴతͳඪɺγεςϜͷ҆ఆੑͱճ෮ྗΛڧԽ͢ Δ͜ͱͰ͢ Gremlin
white paper ɻ resilience ʤපؾɾෆɾࠔɾۤڥͳͲ͔Βͷʥճ෮ྗɺཱͪ Δྗɺ෮׆ྗ ʤมܗ͞Ε͕ͨݩͷܗʹΔʥ෮ݩྗɺྗʢੑʣ ӳࣙPOUIF8&#ΑΓൈਮ
ఆٛʢͬ͘͟Γʣ wෳࡶͰڍಈ͕༧ෆೳͳʢࢄʣγεςϜʹରͯ͠ɺ োೖ ࣮ݧΛߦ͏͜ͱͰͦͷऑΛ͋Ϳ Γग़͢ ͋ͿΓग़ͨ͠ऑΛڧԽ͢Δ͜ͱͰΑΓճ෮ྗͷ͋ΔγεςϜΛߏங ऑͰͳ͔ͬͨ߹ɺͦ͜ʹࣗ৴͕࣋ͯΔ wোೖతͰͳ͘खஈ wߥͬΆ͍݈߁அͱ༧ࣹͷηοτͱ͍͏Πϝʔδ
Կ͕خ͍͠ͷʁ wͲΜͳγεςϜμϯͯ͠͠·͏ͱՁΛఏڙͰ͖ͳ͍ ճ෮ྗ্͕͕Δ͜ͱʢఔʹࠩ͋ΕͲʣ΄΅શͯͷγεςϜʹͱͬͯϋοϐʔ wചΓ্͛Լͷࢭ μϯλΠϜ͕ചΓ্͛ʹӨڹ͢Δ&$αΠτͳͲ wࣗͨͪͷϒϥϯυΠϝʔδΛकΔʢϓϥΠεϨεʣ
ͲΜͳاۀ͕ͬͯΔͷʁ ւ֎ ຊ
ΧΦεΤϯδχΞϦϯάͷݪଇ
w ௨ৗͷಈ࡞Λࣔ͢γεςϜͷଌఆՄೳͳग़ྗͱͯ͠ʮఆৗঢ়ଶʯΛఆٛ͢Δ͜ͱ͔Β࢝Ί·͢ w ͜ͷఆৗঢ়ଶɺରর܈͓Αͼ࣮ݧ܈ͷ྆ํͰܧଓ͢ΔͱԾఆ͠·͢ w αʔόʔͷΫϥογϡɺϋʔυυϥΠϒͷޡ࡞ಈɺωοτϫʔΫଓͷஅͳͲɺݱ࣮ੈքͷΠϕϯ τΛө͢ΔมΛಋೖ͠·͢ w ରর܈ͱ࣮ݧ܈ͱͷؒͷఆৗঢ়ଶͷҧ͍ΛௐΔ͜ͱʹΑͬͯԾઆΛূ͠Α͏ͱ͠·͢ ΧΦεΤϯδχΞϦϯάͷݪଇجຊݪଇ
https://principlesofchaos.org/?lang=ENcontent# ݕূʹ͓͚ΔΧΦε
ΧΦεΤϯδχΞϦϯάͷݪଇৄࡉͳݪଇ w ఆৗঢ়ଶʹ͓͚ΔৼΔ͍ͷԾઆΛཱͯΔ w ࣮ੈքͷࣄଟ༷Ͱ͋Δ w ຊ൪ڥͰݕূΛ࣮ߦ͢Δ w ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ
w ӨڹൣғΛ࠷খԽ͢Δ
ఆৗঢ়ଶJTԿ γεςϜ͕దʹಈ࡞ͯ͠ΔͬͯͲ͏͍͏͜ͱʁ
ఆৗঢ়ଶʹ͓͚ΔৼΔ͍ͷԾઆΛཱͯΔఆৗঢ়ଶͷѲ ϏδωεϝτϦΫεʹ w ސ٬Λࣦ͍ͬͯΔ͔ʁ w ސ٬αΠτͷॏཁͳػೳΛ࣮ߦͰ͖Δ͔ʁʢ&$αΠτͰΧʔτͷνΣοΫΞτՃͳͲʣ w ԆΛཧ༝ʹސ٬αʔϏεΛఀࢭ͢Δ͔ʁ w
ຖඵϏσΦͷετϦʔϛϯάΛ։࢝͢ΔϢʔβʔ 4USFBNJOHQFSTFDPOE 414 ࢀߟ w"NB[PO F#BZച্ w(PPHMF 'BDFCPPLࠂͷදࣔճ https://medium.com/netflix-techblog/sps-the-pulse-of-netflix-streaming-ae4db0e05f8a SPS
ఆৗঢ়ଶʹ͓͚ΔৼΔ͍ͷԾઆΛཱͯΔԾઆΛཱͯΔ • γεςϜʹΠϕϯτΛೖ͢Δͱɺఆৗঢ়ଶͷಈ࡞͕ͲͷΑ͏ʹมԽ͢Δ͔Λߟ͑Δ • ఆৗঢ়ଶதஅ͞ΕΔʁͦΕͱมԽ͠ͳ͍ʁ • தஅͷ߹ɺγεςϜग़ྗ͕૿Ճ·ͨݮগ͢Δͱ༧͍ͯ͠Δʁ w ೖ͢ΔΠϕϯτʹΑͬͯɺʮγεςϜͷಈ࡞͕ఆৗঢ়ଶ͔ΒมԽ͢Δ͜ͱͳ͍ʯͱ͍͏Ծઆͷͱ
࣮ݧΛ࣮ࢪ w ʢྫ Ϣʔβʔʹදࣔ͞ΕΔύʔιφϥΠζ͞ΕͨөըϦετΛੜ͢ΔαʔϏεͷࣦഊ γεςϜσϑΥϧτͷʢύʔιφϥΠζ͞Ε͍ͯͳ͍ʣөըϦετΛฦ͢ w طʹऑ͕͔͍ͬͯΔ߹࣮ݧΛߦ͏લʹमਖ਼͢Δ
࣮ੈքͷࣄଟ༷Ͱ͋Δ • ݱ࣮ͷੈքͰൃੜ͢ΔՄೳੑͷ͋Δೖྗ͔ΒΠϕϯτΛαϯϓϦϯάͯ͠ೖ • Ҏલʹൃੜͨ͠ͷ࠶ൃΛ͙ͨΊɺͦΕʹؔ͢ΔΠϕϯτΛར༻͢Δ͜ͱͱͯ༗༻ w ԾϚγϯΠϯελϯεΛऴྃ w αʔϏεؒͷϦΫΤετʹϨΠςϯγΛૠೖ
w αʔϏεؒͷϦΫΤετͷࣦഊ w ෦αʔϏε͕ࣦഊ w ಛఆͷ"84ϦʔδϣϯΛ༻ෆೳʹ Chaos Kong
ຊ൪ڥͰݕূΛ࣮ߦ͢Δ ΧΦεΤϯδχΞϦϯάͰຊ൪ڥͰͷγεςϜશମͷ;Δ·͍ΛΓ͍ͨɻ ։ൃڥͰҎԼΛຊ൪ڥͱಉҰʹ͢Δ͜ͱ͍͠ɻ • γεςϜͷ࣋ͭσʔλεςʔτ • Ϣʔβ͔Βͷ༧֎ͷೖྗ • ଓ͍ͯ͠Δ֎෦αʔϏε
➡ DNS, ֎෦SaaS etc. • ϋΠϖʔεͰʑߋ৽͞Εଓ͚Δίʔυ w ຊ൪ڥͰݕূΛ࣮ߦ
ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ • ॳΊͷҰճΛखಈͰ࣮ݧΛߦͬͯΈΔͷ࣮ݧ͕ਖ਼͘͠ߦ͑Δ͔Λ֬ೝ͢Δͷʹྑ͍ • ҰํͰɺݱͷγεςϜͷෳࡶ͞ɺͲͷมߋ͕࣮ݧͷ݁ՌΛม͑Δ͔༧ଌͰ͖ͳ͍ ➡ ͯ͢ͷมߋ͕݁ՌʹӨڹ͢Δͱߟ͑Δ • લ߲Ͱड़ͨͱ͓Γɺຊ൪ڥৗʹมԽ
➡ ࣮ݧ݁Ռ͕୲อ͢Δ৴པੑ࣌ؒͱͱʹݮਰ • ΤϯδχΞϦϯάϦιʔε༗ݶ ➡ खಈ࣮ݧʹΑΔ։ൃεϐʔυͷԼຊస w ΧΦε࣮ݧࣗಈԽͷͨΊͷπʔϧͱϓϥοτϑΥʔϜͷ࡞
ӨڹൣғΛ࠷খԽ͢Δ • ຊ൪ڥͰͷ࣮ݧɺސ٬ʹ௧ΈΛҾ͖ى͜͢Մೳੑ͕͋Δ • ࣮ݧ͔ΒͷμϝʔδΛ࠷খݶʹ͑Δ͜ͱΧΦεΤϯδχΞͷٛ w ސ٬͕γεςϜΛར༻Ͱ͖ͳ͘ͳͬͨ߹ɺ͙͢ʹ࣮ݧΛऴྃ ۓٸఀࢭϘλϯͷ࣮
ಛఆ݅ʹΑΔࣗಈऴྃ w ҟৗ͕ൃੜͨ͠ࡍʹૉૣ͍ରԠ Ӧۀ࣌ؒதʹͷΈ࣮ݧΛ࣮ࢪ
োೖπʔϧ
Tool (N)umber of attack types available Enterprise support Can
halt attack in progress GUI Open source Software as a service (SaaS) Cloud agnostic Can attack container / pod Can attack using k8s primitives Can attack serverless Has API Bloomberg Powerful Seal 1 ✔ ✔ ✔ ✔ ✔ ChaosBlade 4 ✔ ✔ ✔ ChaosIQ Chaos Toolkit 3 Via vendor ✔ ✔ ✔ ✔ ✔ ✔ Gremlin Free 2 ✔ ✔ ✔ ✔ ✔ ✔ ✔ Gremlin ILFI 11 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Gremlin ALFI 1 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Istio 2 ✔ ✔ ✔ ✔ Netflix Chaos Monkey 1 ✔ Netflix Simian Army 2 ✔ New Relic Chaos Panda 2 ✔ Nutanix X- Ray 3 ✔ ✔ Pumba 6 ✔ ✔ ✔ https://www.gremlin.com/chaos-engineering/chaos-engineering-tools-comparison/ োೖπʔϧ݁ߏ๛
(SFNMJOɺࣾͰྲྀߦͬͯ·͢ • ݩNetflixͷΧΦεΤϯδχΞKolton AndrusΒʹΑΔSaaS • ʢEC2ͷ߹ʣ15ఔͰΧΦε࣮ݧ͕࢝ΊΒΕΔखܰ͞ • ࢹπʔϧSlackͱͷ౷߹؆୯ɻۓٸఀࢭϘλϯ࣮ࡁΈ
ΧΦεΤϯδχΞϦϯάͷ࢝Ίํʁ
࢝ΊΑ͏ͱ͢Δͱʜ w ຊ൪Ͱͷ࣮ݧϦεΫߴͯ͘ා͍ɻຊ൪ͰΓͨ͘ͳ͍ʜ w Ͱ࣮ݧͷͨΊ͚ͩͷڥΛҡ࣋͢Δͷେมͩ͠ɺαʔυ ύʔςΟ࿈ܞຊ൪ͱҧ͏ͱ͜Ζଟ͍ʜ োൃੜ࣌ͷϏδωεΠϯύΫτ w ػձଛࣦ
w ചΓ্͛ w ϒϥϯυΠϝʔδ w ิঈඅ༻ ΧΦεΤϯδχΞϦϯάಋೖίετ w ࣄނൃੜͷϦεΫ w ਓతϦιʔε w ৺ཧతϋʔυϧ VS ܦݧ͕ͳ͍ͱΧΦεΤϯδχΞϦϯάͷίετͷݟੵΓ͍͠ʜ ✖ ൃੜ֬
(BNFEBZͷεεϝ "84(BNFEBZ w ͞Εͨ"84ڥʹͲΜͲΜো͕ൃੜ͢ΔͷͰɺͦΕΛൃݟ͠मਖ਼ w ࣌ؒͰେྔͷোʹରԠ͢Δ͜ͱʹͳΔͷͰຊͷҙຯͰ"84ͷཧղ͕ࢼ͞ΕΔ w Ϋϥϝιࣾ(BNFEBZΛݱࡏاըதɻ͠धཁ͕͋Ε֎෦͚ʹʜ •
҆શͳڥͰͷΧΦεΤϯδχΞϦϯάͷ࿅श • είʔϓΛߜͬͯ࣌ؒఔɺোೖΛ࣮ࢪ • ࣗࣾͰग़དྷΔͷ͕ϕλʔ͕ͩAWS Gameday͋Δ https://awsgamedaymicroservicestokyo.splashthat.com/
None