Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
カオスエンジニアリングへの招待
Search
KyoheiSaiki
January 15, 2020
Programming
1
1.7k
カオスエンジニアリングへの招待
「AKIBA.AWS #15 ガチ編〜re:Invent振り返り その1〜」での発表資料です。
https://classmethod.connpass.com/event/159529/
KyoheiSaiki
January 15, 2020
Tweet
Share
More Decks by KyoheiSaiki
See All by KyoheiSaiki
Grafana OnCallによる通知
kyo1024
0
170
カオスエンジニアリングのススメ
kyo1024
1
1.6k
New Relic と Classmethod で実践するカオスエンジニアリング
kyo1024
0
2.5k
ANGEL_Dojo_最終発表_問題とミニブログで技術を学ぶ_エンジニア向け学習サービス_Loop_I_O.pdf
kyo1024
0
2.6k
問題とミニブログで技術を学ぶ エンジニア向け学習サービス Loop I/O
kyo1024
0
7k
Other Decks in Programming
See All in Programming
Blazing Fast UI Development with Compose Hot Reload (droidcon New York 2025)
zsmb
1
290
明示と暗黙 ー PHPとGoの インターフェイスの違いを知る
shimabox
2
510
猫と暮らす Google Nest Cam生活🐈 / WebRTC with Google Nest Cam
yutailang0119
0
120
Webの外へ飛び出せ NativePHPが切り拓くPHPの未来
takuyakatsusa
2
550
“いい感じ“な定量評価を求めて - Four Keysとアウトカムの間の探求 -
nealle
1
10k
AIプログラマーDevinは PHPerの夢を見るか?
shinyasaita
1
220
GitHub Copilot and GitHub Codespaces Hands-on
ymd65536
2
150
なぜ「共通化」を考え、失敗を繰り返すのか
rinchoku
1
650
Rails Frontend Evolution: It Was a Setup All Along
skryukov
0
140
すべてのコンテキストを、 ユーザー価値に変える
applism118
3
1.3k
Rubyでやりたい駆動開発 / Ruby driven development
chobishiba
1
700
What Spring Developers Should Know About Jakarta EE
ivargrimstad
0
470
Featured
See All Featured
Facilitating Awesome Meetings
lara
54
6.4k
Making Projects Easy
brettharned
116
6.3k
KATA
mclloyd
30
14k
Raft: Consensus for Rubyists
vanstee
140
7k
How GitHub (no longer) Works
holman
314
140k
Making the Leap to Tech Lead
cromwellryan
134
9.4k
The Language of Interfaces
destraynor
158
25k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
How to Think Like a Performance Engineer
csswizardry
25
1.7k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
130
19k
A Tale of Four Properties
chriscoyier
160
23k
Transcript
ΧΦεΤϯδχΞϦϯάͷট ࠤഢګฏ ,ZP ",*#""84ΨνฤʙSF*OWFOUৼΓฦΓͦͷʙ
8)0".* ࠤഢګฏʢ,ZPʣ w+0*/ w"84ࣄۀຊ෦ίϯαϧ෦ wιϦϡʔγϣϯΞʔΩςΫτ झຯɿ ϥΠϑαΠΤϯε
ԿͷͰ͔͢ʁ 2019/08/23
"84౦ژϦʔδϣϯͷେنো ͪͳΈʹೖࣾޙॳͷ͓٬༷ڥͰͷ࡞ۀ༧ఆͰͨ͠
"Everything fails, all the time" - Werner Vogels (CTO,
Amazon.com)
ࠓͷຊ ࣮ݧΛ௨ͯ͠γεςϜͷʢো࣌ͷʣৼΔ͍ʹ ࣗ৴Λ͚ͭΔ
ΧΦεΤϯδχΞϦϯά is Կ
ఆٛʢจݙΑΓʣ wΧΦεΤϯδχΞϦϯάɺຊ൪ڥͷߥΕڰͬͨঢ়ଶʹ͑ΔγεςϜͷೳྗ ʹࣗ৴Λ࣋ͨͤΔͨΊʹɺࢄγεςϜͰ࣮ݧ͢ΔֶͰ͢ Netflix paper ɻ wΧΦεΤϯδχΞϦϯάͷ࠷ऴతͳඪɺγεςϜͷ҆ఆੑͱճ෮ྗΛڧԽ͢ Δ͜ͱͰ͢ Gremlin
white paper ɻ resilience ʤපؾɾෆɾࠔɾۤڥͳͲ͔Βͷʥճ෮ྗɺཱͪ Δྗɺ෮׆ྗ ʤมܗ͞Ε͕ͨݩͷܗʹΔʥ෮ݩྗɺྗʢੑʣ ӳࣙPOUIF8&#ΑΓൈਮ
ఆٛʢͬ͘͟Γʣ wෳࡶͰڍಈ͕༧ෆೳͳʢࢄʣγεςϜʹରͯ͠ɺ োೖ ࣮ݧΛߦ͏͜ͱͰͦͷऑΛ͋Ϳ Γग़͢ ͋ͿΓग़ͨ͠ऑΛڧԽ͢Δ͜ͱͰΑΓճ෮ྗͷ͋ΔγεςϜΛߏங ऑͰͳ͔ͬͨ߹ɺͦ͜ʹࣗ৴͕࣋ͯΔ wোೖతͰͳ͘खஈ wߥͬΆ͍݈߁அͱ༧ࣹͷηοτͱ͍͏Πϝʔδ
Կ͕خ͍͠ͷʁ wͲΜͳγεςϜμϯͯ͠͠·͏ͱՁΛఏڙͰ͖ͳ͍ ճ෮ྗ্͕͕Δ͜ͱʢఔʹࠩ͋ΕͲʣ΄΅શͯͷγεςϜʹͱͬͯϋοϐʔ wചΓ্͛Լͷࢭ μϯλΠϜ͕ചΓ্͛ʹӨڹ͢Δ&$αΠτͳͲ wࣗͨͪͷϒϥϯυΠϝʔδΛकΔʢϓϥΠεϨεʣ
ͲΜͳاۀ͕ͬͯΔͷʁ ւ֎ ຊ
ΧΦεΤϯδχΞϦϯάͷݪଇ
w ௨ৗͷಈ࡞Λࣔ͢γεςϜͷଌఆՄೳͳग़ྗͱͯ͠ʮఆৗঢ়ଶʯΛఆٛ͢Δ͜ͱ͔Β࢝Ί·͢ w ͜ͷఆৗঢ়ଶɺରর܈͓Αͼ࣮ݧ܈ͷ྆ํͰܧଓ͢ΔͱԾఆ͠·͢ w αʔόʔͷΫϥογϡɺϋʔυυϥΠϒͷޡ࡞ಈɺωοτϫʔΫଓͷஅͳͲɺݱ࣮ੈքͷΠϕϯ τΛө͢ΔมΛಋೖ͠·͢ w ରর܈ͱ࣮ݧ܈ͱͷؒͷఆৗঢ়ଶͷҧ͍ΛௐΔ͜ͱʹΑͬͯԾઆΛূ͠Α͏ͱ͠·͢ ΧΦεΤϯδχΞϦϯάͷݪଇجຊݪଇ
https://principlesofchaos.org/?lang=ENcontent# ݕূʹ͓͚ΔΧΦε
ΧΦεΤϯδχΞϦϯάͷݪଇৄࡉͳݪଇ w ఆৗঢ়ଶʹ͓͚ΔৼΔ͍ͷԾઆΛཱͯΔ w ࣮ੈքͷࣄଟ༷Ͱ͋Δ w ຊ൪ڥͰݕূΛ࣮ߦ͢Δ w ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ
w ӨڹൣғΛ࠷খԽ͢Δ
ఆৗঢ়ଶJTԿ γεςϜ͕దʹಈ࡞ͯ͠ΔͬͯͲ͏͍͏͜ͱʁ
ఆৗঢ়ଶʹ͓͚ΔৼΔ͍ͷԾઆΛཱͯΔఆৗঢ়ଶͷѲ ϏδωεϝτϦΫεʹ w ސ٬Λࣦ͍ͬͯΔ͔ʁ w ސ٬αΠτͷॏཁͳػೳΛ࣮ߦͰ͖Δ͔ʁʢ&$αΠτͰΧʔτͷνΣοΫΞτՃͳͲʣ w ԆΛཧ༝ʹސ٬αʔϏεΛఀࢭ͢Δ͔ʁ w
ຖඵϏσΦͷετϦʔϛϯάΛ։࢝͢ΔϢʔβʔ 4USFBNJOHQFSTFDPOE 414 ࢀߟ w"NB[PO F#BZച্ w(PPHMF 'BDFCPPLࠂͷදࣔճ https://medium.com/netflix-techblog/sps-the-pulse-of-netflix-streaming-ae4db0e05f8a SPS
ఆৗঢ়ଶʹ͓͚ΔৼΔ͍ͷԾઆΛཱͯΔԾઆΛཱͯΔ • γεςϜʹΠϕϯτΛೖ͢Δͱɺఆৗঢ়ଶͷಈ࡞͕ͲͷΑ͏ʹมԽ͢Δ͔Λߟ͑Δ • ఆৗঢ়ଶதஅ͞ΕΔʁͦΕͱมԽ͠ͳ͍ʁ • தஅͷ߹ɺγεςϜग़ྗ͕૿Ճ·ͨݮগ͢Δͱ༧͍ͯ͠Δʁ w ೖ͢ΔΠϕϯτʹΑͬͯɺʮγεςϜͷಈ࡞͕ఆৗঢ়ଶ͔ΒมԽ͢Δ͜ͱͳ͍ʯͱ͍͏Ծઆͷͱ
࣮ݧΛ࣮ࢪ w ʢྫ Ϣʔβʔʹදࣔ͞ΕΔύʔιφϥΠζ͞ΕͨөըϦετΛੜ͢ΔαʔϏεͷࣦഊ γεςϜσϑΥϧτͷʢύʔιφϥΠζ͞Ε͍ͯͳ͍ʣөըϦετΛฦ͢ w طʹऑ͕͔͍ͬͯΔ߹࣮ݧΛߦ͏લʹमਖ਼͢Δ
࣮ੈքͷࣄଟ༷Ͱ͋Δ • ݱ࣮ͷੈքͰൃੜ͢ΔՄೳੑͷ͋Δೖྗ͔ΒΠϕϯτΛαϯϓϦϯάͯ͠ೖ • Ҏલʹൃੜͨ͠ͷ࠶ൃΛ͙ͨΊɺͦΕʹؔ͢ΔΠϕϯτΛར༻͢Δ͜ͱͱͯ༗༻ w ԾϚγϯΠϯελϯεΛऴྃ w αʔϏεؒͷϦΫΤετʹϨΠςϯγΛૠೖ
w αʔϏεؒͷϦΫΤετͷࣦഊ w ෦αʔϏε͕ࣦഊ w ಛఆͷ"84ϦʔδϣϯΛ༻ෆೳʹ Chaos Kong
ຊ൪ڥͰݕূΛ࣮ߦ͢Δ ΧΦεΤϯδχΞϦϯάͰຊ൪ڥͰͷγεςϜશମͷ;Δ·͍ΛΓ͍ͨɻ ։ൃڥͰҎԼΛຊ൪ڥͱಉҰʹ͢Δ͜ͱ͍͠ɻ • γεςϜͷ࣋ͭσʔλεςʔτ • Ϣʔβ͔Βͷ༧֎ͷೖྗ • ଓ͍ͯ͠Δ֎෦αʔϏε
➡ DNS, ֎෦SaaS etc. • ϋΠϖʔεͰʑߋ৽͞Εଓ͚Δίʔυ w ຊ൪ڥͰݕূΛ࣮ߦ
ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ • ॳΊͷҰճΛखಈͰ࣮ݧΛߦͬͯΈΔͷ࣮ݧ͕ਖ਼͘͠ߦ͑Δ͔Λ֬ೝ͢Δͷʹྑ͍ • ҰํͰɺݱͷγεςϜͷෳࡶ͞ɺͲͷมߋ͕࣮ݧͷ݁ՌΛม͑Δ͔༧ଌͰ͖ͳ͍ ➡ ͯ͢ͷมߋ͕݁ՌʹӨڹ͢Δͱߟ͑Δ • લ߲Ͱड़ͨͱ͓Γɺຊ൪ڥৗʹมԽ
➡ ࣮ݧ݁Ռ͕୲อ͢Δ৴པੑ࣌ؒͱͱʹݮਰ • ΤϯδχΞϦϯάϦιʔε༗ݶ ➡ खಈ࣮ݧʹΑΔ։ൃεϐʔυͷԼຊస w ΧΦε࣮ݧࣗಈԽͷͨΊͷπʔϧͱϓϥοτϑΥʔϜͷ࡞
ӨڹൣғΛ࠷খԽ͢Δ • ຊ൪ڥͰͷ࣮ݧɺސ٬ʹ௧ΈΛҾ͖ى͜͢Մೳੑ͕͋Δ • ࣮ݧ͔ΒͷμϝʔδΛ࠷খݶʹ͑Δ͜ͱΧΦεΤϯδχΞͷٛ w ސ٬͕γεςϜΛར༻Ͱ͖ͳ͘ͳͬͨ߹ɺ͙͢ʹ࣮ݧΛऴྃ ۓٸఀࢭϘλϯͷ࣮
ಛఆ݅ʹΑΔࣗಈऴྃ w ҟৗ͕ൃੜͨ͠ࡍʹૉૣ͍ରԠ Ӧۀ࣌ؒதʹͷΈ࣮ݧΛ࣮ࢪ
োೖπʔϧ
Tool (N)umber of attack types available Enterprise support Can
halt attack in progress GUI Open source Software as a service (SaaS) Cloud agnostic Can attack container / pod Can attack using k8s primitives Can attack serverless Has API Bloomberg Powerful Seal 1 ✔ ✔ ✔ ✔ ✔ ChaosBlade 4 ✔ ✔ ✔ ChaosIQ Chaos Toolkit 3 Via vendor ✔ ✔ ✔ ✔ ✔ ✔ Gremlin Free 2 ✔ ✔ ✔ ✔ ✔ ✔ ✔ Gremlin ILFI 11 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Gremlin ALFI 1 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Istio 2 ✔ ✔ ✔ ✔ Netflix Chaos Monkey 1 ✔ Netflix Simian Army 2 ✔ New Relic Chaos Panda 2 ✔ Nutanix X- Ray 3 ✔ ✔ Pumba 6 ✔ ✔ ✔ https://www.gremlin.com/chaos-engineering/chaos-engineering-tools-comparison/ োೖπʔϧ݁ߏ๛
(SFNMJOɺࣾͰྲྀߦͬͯ·͢ • ݩNetflixͷΧΦεΤϯδχΞKolton AndrusΒʹΑΔSaaS • ʢEC2ͷ߹ʣ15ఔͰΧΦε࣮ݧ͕࢝ΊΒΕΔखܰ͞ • ࢹπʔϧSlackͱͷ౷߹؆୯ɻۓٸఀࢭϘλϯ࣮ࡁΈ
ΧΦεΤϯδχΞϦϯάͷ࢝Ίํʁ
࢝ΊΑ͏ͱ͢Δͱʜ w ຊ൪Ͱͷ࣮ݧϦεΫߴͯ͘ා͍ɻຊ൪ͰΓͨ͘ͳ͍ʜ w Ͱ࣮ݧͷͨΊ͚ͩͷڥΛҡ࣋͢Δͷେมͩ͠ɺαʔυ ύʔςΟ࿈ܞຊ൪ͱҧ͏ͱ͜Ζଟ͍ʜ োൃੜ࣌ͷϏδωεΠϯύΫτ w ػձଛࣦ
w ചΓ্͛ w ϒϥϯυΠϝʔδ w ิঈඅ༻ ΧΦεΤϯδχΞϦϯάಋೖίετ w ࣄނൃੜͷϦεΫ w ਓతϦιʔε w ৺ཧతϋʔυϧ VS ܦݧ͕ͳ͍ͱΧΦεΤϯδχΞϦϯάͷίετͷݟੵΓ͍͠ʜ ✖ ൃੜ֬
(BNFEBZͷεεϝ "84(BNFEBZ w ͞Εͨ"84ڥʹͲΜͲΜো͕ൃੜ͢ΔͷͰɺͦΕΛൃݟ͠मਖ਼ w ࣌ؒͰେྔͷোʹରԠ͢Δ͜ͱʹͳΔͷͰຊͷҙຯͰ"84ͷཧղ͕ࢼ͞ΕΔ w Ϋϥϝιࣾ(BNFEBZΛݱࡏاըதɻ͠धཁ͕͋Ε֎෦͚ʹʜ •
҆શͳڥͰͷΧΦεΤϯδχΞϦϯάͷ࿅श • είʔϓΛߜͬͯ࣌ؒఔɺোೖΛ࣮ࢪ • ࣗࣾͰग़དྷΔͷ͕ϕλʔ͕ͩAWS Gameday͋Δ https://awsgamedaymicroservicestokyo.splashthat.com/
None