Before Chaos Engineering

Transcript

1. ©2019 Wantedly, Inc. Chaos Engineeringͷ खલͷ࿩ golang.tokyo #24 May 20,

   2019 - Hazumi Ichijo (rerost/hazumirr) ΞϓϦέʔγϣϯΤϯδχΞ͚ͩͰͰ͖Δ͜ͱ

2. ©2018 Wantedly, Inc. ࣗݾ঺հ Ұᑍ
   ୺੅
   !IB[VNJSS!SFSPTU
   4FSWFS
   4JEF
   &OHJOFFS
   JO
   8BOUFEMZ
   (P
   3BJMT
   3FBDUʜ
   ग़਎
   ๺ւಓ

3. ©2019 Wantedly, Inc. 1. αʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔ 2. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔

4. ©2019 Wantedly, Inc. 1. αʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔ 2. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔ => ೺Ѳ͍ͯ͠Δͭ΋Γ͕ͩͦΕ͸ຊ౰͔ʁ

5. ©2019 Wantedly, Inc.

6. ©2019 Wantedly, Inc. ͦ͏ͩChaos Engineeringͩ

7. ©2019 Wantedly, Inc. ͦ΋ͦ΋Chaos Engineeringͱ͸ʁ

8. ©2019 Wantedly, Inc. > ΧΦεΤϯδχΞϦϯά͸ɺ෼ࢄγεςϜʹ͓͍ͯγεςϜ͕ෆ҆ ఆͳঢ়ଶʹ଱͑Δ͜ͱͷग़དྷΔ؀ڥΛߏங͢ΔͨΊͷݕূͷن཯ https://principlesofchaos.org/?lang=JAcontent#

9. ©2019 Wantedly, Inc. 1. Πϯελϯε͕μ΢ϯ͢Δ 2. DB͕٧·ΔɾΞΫηεͰ͖ͳ͘ͳΔɹ 3. ґଘ͢ΔଞͷαʔϏε͕ෆ҆ఆʹͳΔ etc…

   ෆ҆ఆͳঢ়ଶͱ͸

10. ©2019 Wantedly, Inc. 1. αʔϏε͕ෆ҆ఆͳ࣌ɺͲ͏͍ͬͨӨڹ͕ͰΔ͔ 2. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔ ஌Γ͍ͨ͜ͱ

11. ©2019 Wantedly, Inc. αʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε 4FSWJDF
    " 4FSWJDF
    # 4FSWJDF
    $

12. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF
    " 4FSWJDF
    # 4FSWJDF
    $ αʔϏε͕ෆ҆ఆͳ࣌

13. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF
    " 4FSWJDF
    # 4FSWJDF
    $ αʔϏε͕ෆ҆ఆͳ࣌

14. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF
    " 4FSWJDF
    # 4FSWJDF
    $ αʔϏε͕ෆ҆ఆͳ࣌

15. ©2019 Wantedly, Inc. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε ଞͷαʔϏε

16. ©2019 Wantedly, Inc. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε ଞͷαʔϏε

17. ©2019 Wantedly, Inc. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε ଞͷαʔϏε

18. ©2019 Wantedly, Inc. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε ଞͷαʔϏε

19. ©2019 Wantedly, Inc. Ͳ͏͍ͬͨ͜ͱΛ͢Δ΂͖͔ʁ

20. ©2019 Wantedly, Inc. 1. ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔ 2. ࣮ੈքͷࣄ৅͸ଟ༷Ͱ͋Δ 3. ຊ൪؀ڥͰݕূΛ࣮ߦ͢Δ 4.

    ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ 5. ӨڹൣғΛہॴԽ͢Δ https://principlesofchaos.org/?lang=JAcontent# Ͳ͏͍ͬͨ͜ͱΛߦ͏͔

21. ©2019 Wantedly, Inc. 1. ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔ 2. ࣮ੈքͷࣄ৅͸ଟ༷Ͱ͋Δ 3. ຊ൪؀ڥͰݕূΛ࣮ߦ͢Δ 4.

    ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ 5. ӨڹൣғΛہॴԽ͢Δ https://principlesofchaos.org/?lang=JAcontent# 43&ͱͷڠྗ΍૊৫త ͳରԠ͕ඞཁ Ͳ͏͍ͬͨ͜ͱΛߦ͏͔

22. ©2019 Wantedly, Inc. 1. 1ඵؒͷಈը࠶ੜ਺ͳͲ 2. APIͷϨεϙϯε଎౓ͳͲ 1. ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔ =>

    ͦΕΛकΔͨΊͷ΋ͷ͕ॏཁɻͦͷ֬ೝ͕Ͱ͖Ε͹͋Δఔ౓҆৺ ྫ: AWS͕region͝ͱμ΢ϯͨ͠ࡍɺଞͷregionʹfail over͢ΔͳͲ ྫ

23. ©2019 Wantedly, Inc. 1. Πϯελϯε͕μ΢ϯ͢Δ 2. DB͕٧·ΔɾΞΫηεͰ͖ͳ͘ͳΔɹ 3. ґଘ͢ΔଞͷαʔϏε͕ෆ҆ఆʹͳΔ etc…

    2. ࣮ੈքͷࣄ৅͸ଟ༷Ͱ͋Δ => ͜͏͍ͬͨཁૉΛೖΕΒΕΔΑ͏ʹ͢Δ = Fault Injection

24. ©2019 Wantedly, Inc. ͜ͷ2͚ͭͩͳΒQA؀ڥͰ ΞϓϦέʔγϣϯΤϯδχΞ͚ͩͰՄೳ

25. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF
    " 4FSWJDF
    # Fault Injectionͱͦͷ֬ೝ

26. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF
    " 4FSWJDF
    # Fault Injectionͱͦͷ֬ೝ ֬ೝ༻ͷίʔυΛೖΕΔͷ͕໘౗

27. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε JOUFSDFQUPS 4FSWJDF
    " 4FSWJDF
    # Fault Injectionͱͦͷ֬ೝ

28. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε  ൪୆Λฦ͢
     ஗͘ͳΔ
     ઃఆ͍ͯ͠ΔλΠϜΞ΢

    τΑΓϨεϙϯε͕஗͍
    JOUFSDFQUPS 4FSWJDF
    " 4FSWJDF
    # Fault Injectionͱͦͷ֬ೝ

29. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε  ൪୆Λฦ͢
     ஗͘ͳΔ
     ઃఆ͍ͯ͠ΔλΠϜΞ΢

    τΑΓϨεϙϯε͕஗͍
    JOUFSDFQUPS 4FSWJDF
    " 4FSWJDF
    # { "interval": 1, "slow_response_option": { "enable": true, "time": 5 }, "server_error_option": { "enable": true } } Fault Injectionͱͦͷ֬ೝ

30. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε  ൪୆Λฦ͢
     ஗͘ͳΔ
     ઃఆ͍ͯ͠ΔλΠϜΞ΢

    τΑΓϨεϙϯε͕஗͍
    JOUFSDFQUPS 4FSWJDF
    " 4FSWJDF
    # Fault Injectionͱͦͷ֬ೝ

31. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε JOUFSDFQUPS 4FSWJDF
    " 4FSWJDF
    # Fault Injectionͱͦͷ֬ೝ

32. ©2019 Wantedly, Inc. 4FSWJDF
    " 4FSWJDF
    # $MPVE
    1VC4VC 1VCMJTI 4VCTDSJCF Fault Injectionͱͦͷ֬ೝ

33. ©2019 Wantedly, Inc. 4FSWJDF
    " 4FSWJDF
    # $MPVE
    1VC4VC 1VCMJTI 4VCTDSJCF Fault Injectionͱͦͷ֬ೝ

34. ©2019 Wantedly, Inc. 4FSWJDF
    " 4FSWJDF
    # $MPVE
    1VC4VC  ॏෳ഑৴Λ͢Δ
     1VCMJTIͨ͠ॱ൪Ͱ഑৴͠ͳ͍

    1VCMJTI 4VCTDSJCF Fault Injectionͱͦͷ֬ೝ

35. ©2019 Wantedly, Inc. 4FSWJDF
    " 4FSWJDF
    # $MPVE
    1VC4VC  ॏෳ഑৴Λ͢Δ
     1VCMJTIͨ͠ॱ൪Ͱ഑৴͠ͳ͍

    1VCMJTI 4VCTDSJCF ϥΠϒϥϦ͕ߦ͏ͷͰ JOUFSDFQUPSΛڬΊͳ͍
    Fault Injectionͱͦͷ֬ೝ

36. ©2019 Wantedly, Inc. 4FSWJDF
    " 4FSWJDF
    # $MPVE
    1VC4VC 1VCMJTI 4VCTDSJCF 1SPYZ 1SPYZ

    Fault Injectionͱͦͷ֬ೝ

37. ©2019 Wantedly, Inc. 4FSWJDF
    " 4FSWJDF
    # $MPVE
    1VC4VC 1VCMJTI 4VCTDSJCF 1SPYZ 1SPYZ

     ॏෳ഑৴Λ͢Δ
     1VCMJTIͨ͠ॱ൪Ͱ഑৴͠ͳ͍ Fault Injectionͱͦͷ֬ೝ

38. ©2019 Wantedly, Inc. https://github.com/rerost/unstable https://github.com/rerost/chaos-pubsub

39. ©2019 Wantedly, Inc.

40. ©2019 Wantedly, Inc. σϝϦοτ  ݱঢ়ͷ֬ೝ͸Ͱ͖Δ͕͍ͭյΕΔ͔Θ͔Βͳ͍ɹ
     ςετ͢Δͷ͕ਓखͳͷͰ֬ೝ͕໘౗
     ςετ͕໢ཏͰ͖͍ͯͳ͍Մೳੑ͕͋Δ

41. ©2019 Wantedly, Inc. ·ͱΊ  खಈͰ΋͋Δఔ౓ͷݕূ͸Ͱ͖Δ
     ςετ༻ͷίʔυ͸ൈ͖΍͍͢ܗʹ͓ͯ͘͠ɹ
     2"؀ڥ͚ͩͰ΋ࢼ͓ͯ͘͠ҙຯ͸͋Δɹ
    

     σϝϦοτ͸ଟ͍ͷͰҰൠʹ͓͢͢ΊͰ͖Δ ํ๏Ͱ͸ͳ͍