Slide 1

Slide 1 text

©2019 Wantedly, Inc. Chaos Engineeringͷ खલͷ࿩ golang.tokyo #24 May 20, 2019 - Hazumi Ichijo (rerost/hazumirr) ΞϓϦέʔγϣϯΤϯδχΞ͚ͩͰͰ͖Δ͜ͱ

Slide 2

Slide 2 text

©2018 Wantedly, Inc. ࣗݾ঺հ Ұᑍ୺੅!IB[VNJSS!SFSPTU 4FSWFS4JEF&OHJOFFSJO8BOUFEMZ (P 3BJMT 3FBDUʜ ग़਎๺ւಓ

Slide 3

Slide 3 text

©2019 Wantedly, Inc. 1. αʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔ 2. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔

Slide 4

Slide 4 text

©2019 Wantedly, Inc. 1. αʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔ 2. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔ => ೺Ѳ͍ͯ͠Δͭ΋Γ͕ͩͦΕ͸ຊ౰͔ʁ

Slide 5

Slide 5 text

©2019 Wantedly, Inc.

Slide 6

Slide 6 text

©2019 Wantedly, Inc. ͦ͏ͩChaos Engineeringͩ

Slide 7

Slide 7 text

©2019 Wantedly, Inc. ͦ΋ͦ΋Chaos Engineeringͱ͸ʁ

Slide 8

Slide 8 text

©2019 Wantedly, Inc. > ΧΦεΤϯδχΞϦϯά͸ɺ෼ࢄγεςϜʹ͓͍ͯγεςϜ͕ෆ҆ ఆͳঢ়ଶʹ଱͑Δ͜ͱͷग़དྷΔ؀ڥΛߏங͢ΔͨΊͷݕূͷن཯ https://principlesofchaos.org/?lang=JAcontent#

Slide 9

Slide 9 text

©2019 Wantedly, Inc. 1. Πϯελϯε͕μ΢ϯ͢Δ 2. DB͕٧·ΔɾΞΫηεͰ͖ͳ͘ͳΔɹ 3. ґଘ͢ΔଞͷαʔϏε͕ෆ҆ఆʹͳΔ etc… ෆ҆ఆͳঢ়ଶͱ͸

Slide 10

Slide 10 text

©2019 Wantedly, Inc. 1. αʔϏε͕ෆ҆ఆͳ࣌ɺͲ͏͍ͬͨӨڹ͕ͰΔ͔ 2. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔ ஌Γ͍ͨ͜ͱ

Slide 11

Slide 11 text

©2019 Wantedly, Inc. αʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε 4FSWJDF" 4FSWJDF# 4FSWJDF$

Slide 12

Slide 12 text

©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF" 4FSWJDF# 4FSWJDF$ αʔϏε͕ෆ҆ఆͳ࣌

Slide 13

Slide 13 text

©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF" 4FSWJDF# 4FSWJDF$ αʔϏε͕ෆ҆ఆͳ࣌

Slide 14

Slide 14 text

©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF" 4FSWJDF# 4FSWJDF$ αʔϏε͕ෆ҆ఆͳ࣌

Slide 15

Slide 15 text

©2019 Wantedly, Inc. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε ଞͷαʔϏε

Slide 16

Slide 16 text

©2019 Wantedly, Inc. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε ଞͷαʔϏε

Slide 17

Slide 17 text

©2019 Wantedly, Inc. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε ଞͷαʔϏε

Slide 18

Slide 18 text

©2019 Wantedly, Inc. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε ଞͷαʔϏε

Slide 19

Slide 19 text

©2019 Wantedly, Inc. Ͳ͏͍ͬͨ͜ͱΛ͢Δ΂͖͔ʁ

Slide 20

Slide 20 text

©2019 Wantedly, Inc. 1. ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔ 2. ࣮ੈքͷࣄ৅͸ଟ༷Ͱ͋Δ 3. ຊ൪؀ڥͰݕূΛ࣮ߦ͢Δ 4. ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ 5. ӨڹൣғΛہॴԽ͢Δ https://principlesofchaos.org/?lang=JAcontent# Ͳ͏͍ͬͨ͜ͱΛߦ͏͔

Slide 21

Slide 21 text

©2019 Wantedly, Inc. 1. ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔ 2. ࣮ੈքͷࣄ৅͸ଟ༷Ͱ͋Δ 3. ຊ൪؀ڥͰݕূΛ࣮ߦ͢Δ 4. ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ 5. ӨڹൣғΛہॴԽ͢Δ https://principlesofchaos.org/?lang=JAcontent# 43&ͱͷڠྗ΍૊৫త ͳରԠ͕ඞཁ Ͳ͏͍ͬͨ͜ͱΛߦ͏͔

Slide 22

Slide 22 text

©2019 Wantedly, Inc. 1. 1ඵؒͷಈը࠶ੜ਺ͳͲ 2. APIͷϨεϙϯε଎౓ͳͲ 1. ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔ => ͦΕΛकΔͨΊͷ΋ͷ͕ॏཁɻͦͷ֬ೝ͕Ͱ͖Ε͹͋Δఔ౓҆৺ ྫ: AWS͕region͝ͱμ΢ϯͨ͠ࡍɺଞͷregionʹfail over͢ΔͳͲ ྫ

Slide 23

Slide 23 text

©2019 Wantedly, Inc. 1. Πϯελϯε͕μ΢ϯ͢Δ 2. DB͕٧·ΔɾΞΫηεͰ͖ͳ͘ͳΔɹ 3. ґଘ͢ΔଞͷαʔϏε͕ෆ҆ఆʹͳΔ etc… 2. ࣮ੈքͷࣄ৅͸ଟ༷Ͱ͋Δ => ͜͏͍ͬͨཁૉΛೖΕΒΕΔΑ͏ʹ͢Δ = Fault Injection

Slide 24

Slide 24 text

©2019 Wantedly, Inc. ͜ͷ2͚ͭͩͳΒQA؀ڥͰ ΞϓϦέʔγϣϯΤϯδχΞ͚ͩͰՄೳ

Slide 25

Slide 25 text

©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF" 4FSWJDF# Fault Injectionͱͦͷ֬ೝ

Slide 26

Slide 26 text

©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF" 4FSWJDF# Fault Injectionͱͦͷ֬ೝ ֬ೝ༻ͷίʔυΛೖΕΔͷ͕໘౗

Slide 27

Slide 27 text

©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε JOUFSDFQUPS 4FSWJDF" 4FSWJDF# Fault Injectionͱͦͷ֬ೝ

Slide 28

Slide 28 text

©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε  ൪୆Λฦ͢  ஗͘ͳΔ  ઃఆ͍ͯ͠ΔλΠϜΞ΢ τΑΓϨεϙϯε͕஗͍ JOUFSDFQUPS 4FSWJDF" 4FSWJDF# Fault Injectionͱͦͷ֬ೝ

Slide 29

Slide 29 text

©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε  ൪୆Λฦ͢  ஗͘ͳΔ  ઃఆ͍ͯ͠ΔλΠϜΞ΢ τΑΓϨεϙϯε͕஗͍ JOUFSDFQUPS 4FSWJDF" 4FSWJDF# { "interval": 1, "slow_response_option": { "enable": true, "time": 5 }, "server_error_option": { "enable": true } } Fault Injectionͱͦͷ֬ೝ

Slide 30

Slide 30 text

©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε  ൪୆Λฦ͢  ஗͘ͳΔ  ઃఆ͍ͯ͠ΔλΠϜΞ΢ τΑΓϨεϙϯε͕஗͍ JOUFSDFQUPS 4FSWJDF" 4FSWJDF# Fault Injectionͱͦͷ֬ೝ

Slide 31

Slide 31 text

©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε JOUFSDFQUPS 4FSWJDF" 4FSWJDF# Fault Injectionͱͦͷ֬ೝ

Slide 32

Slide 32 text

©2019 Wantedly, Inc. 4FSWJDF" 4FSWJDF# $MPVE1VC4VC 1VCMJTI 4VCTDSJCF Fault Injectionͱͦͷ֬ೝ

Slide 33

Slide 33 text

©2019 Wantedly, Inc. 4FSWJDF" 4FSWJDF# $MPVE1VC4VC 1VCMJTI 4VCTDSJCF Fault Injectionͱͦͷ֬ೝ

Slide 34

Slide 34 text

©2019 Wantedly, Inc. 4FSWJDF" 4FSWJDF# $MPVE1VC4VC  ॏෳ഑৴Λ͢Δ  1VCMJTIͨ͠ॱ൪Ͱ഑৴͠ͳ͍ 1VCMJTI 4VCTDSJCF Fault Injectionͱͦͷ֬ೝ

Slide 35

Slide 35 text

©2019 Wantedly, Inc. 4FSWJDF" 4FSWJDF# $MPVE1VC4VC  ॏෳ഑৴Λ͢Δ  1VCMJTIͨ͠ॱ൪Ͱ഑৴͠ͳ͍ 1VCMJTI 4VCTDSJCF ϥΠϒϥϦ͕ߦ͏ͷͰ JOUFSDFQUPSΛڬΊͳ͍ Fault Injectionͱͦͷ֬ೝ

Slide 36

Slide 36 text

©2019 Wantedly, Inc. 4FSWJDF" 4FSWJDF# $MPVE1VC4VC 1VCMJTI 4VCTDSJCF 1SPYZ 1SPYZ Fault Injectionͱͦͷ֬ೝ

Slide 37

Slide 37 text

©2019 Wantedly, Inc. 4FSWJDF" 4FSWJDF# $MPVE1VC4VC 1VCMJTI 4VCTDSJCF 1SPYZ 1SPYZ  ॏෳ഑৴Λ͢Δ  1VCMJTIͨ͠ॱ൪Ͱ഑৴͠ͳ͍ Fault Injectionͱͦͷ֬ೝ

Slide 38

Slide 38 text

©2019 Wantedly, Inc. https://github.com/rerost/unstable https://github.com/rerost/chaos-pubsub

Slide 39

Slide 39 text

©2019 Wantedly, Inc.

Slide 40

Slide 40 text

©2019 Wantedly, Inc. σϝϦοτ  ݱঢ়ͷ֬ೝ͸Ͱ͖Δ͕͍ͭյΕΔ͔Θ͔Βͳ͍ɹ  ςετ͢Δͷ͕ਓखͳͷͰ֬ೝ͕໘౗  ςετ͕໢ཏͰ͖͍ͯͳ͍Մೳੑ͕͋Δ

Slide 41

Slide 41 text

©2019 Wantedly, Inc. ·ͱΊ  खಈͰ΋͋Δఔ౓ͷݕূ͸Ͱ͖Δ  ςετ༻ͷίʔυ͸ൈ͖΍͍͢ܗʹ͓ͯ͘͠ɹ  2"؀ڥ͚ͩͰ΋ࢼ͓ͯ͘͠ҙຯ͸͋Δɹ  σϝϦοτ͸ଟ͍ͷͰҰൠʹ͓͢͢ΊͰ͖Δ ํ๏Ͱ͸ͳ͍