Before Chaos Engineering

Before Chaos Engineering

9ec2ceee3f782e64fd67fa7d82b6c1b4?s=128

Hazumi Ichijo

May 20, 2019
Tweet

Transcript

  1. ©2019 Wantedly, Inc. Chaos Engineeringͷ खલͷ࿩ golang.tokyo #24 May 20,

    2019 - Hazumi Ichijo (rerost/hazumirr) ΞϓϦέʔγϣϯΤϯδχΞ͚ͩͰͰ͖Δ͜ͱ
  2. ©2018 Wantedly, Inc. ࣗݾ঺հ Ұᑍ୺੅!IB[VNJSS!SFSPTU 4FSWFS4JEF&OHJOFFSJO8BOUFEMZ (P 3BJMT 3FBDUʜ ग़਎๺ւಓ

  3. ©2019 Wantedly, Inc. 1. αʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔ 2. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔

  4. ©2019 Wantedly, Inc. 1. αʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔ 2. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔ => ೺Ѳ͍ͯ͠Δͭ΋Γ͕ͩͦΕ͸ຊ౰͔ʁ

  5. ©2019 Wantedly, Inc.

  6. ©2019 Wantedly, Inc. ͦ͏ͩChaos Engineeringͩ

  7. ©2019 Wantedly, Inc. ͦ΋ͦ΋Chaos Engineeringͱ͸ʁ

  8. ©2019 Wantedly, Inc. > ΧΦεΤϯδχΞϦϯά͸ɺ෼ࢄγεςϜʹ͓͍ͯγεςϜ͕ෆ҆ ఆͳঢ়ଶʹ଱͑Δ͜ͱͷग़དྷΔ؀ڥΛߏங͢ΔͨΊͷݕূͷن཯ https://principlesofchaos.org/?lang=JAcontent#

  9. ©2019 Wantedly, Inc. 1. Πϯελϯε͕μ΢ϯ͢Δ 2. DB͕٧·ΔɾΞΫηεͰ͖ͳ͘ͳΔɹ 3. ґଘ͢ΔଞͷαʔϏε͕ෆ҆ఆʹͳΔ etc…

    ෆ҆ఆͳঢ়ଶͱ͸
  10. ©2019 Wantedly, Inc. 1. αʔϏε͕ෆ҆ఆͳ࣌ɺͲ͏͍ͬͨӨڹ͕ͰΔ͔ 2. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔ ஌Γ͍ͨ͜ͱ

  11. ©2019 Wantedly, Inc. αʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε 4FSWJDF" 4FSWJDF# 4FSWJDF$

  12. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF" 4FSWJDF# 4FSWJDF$ αʔϏε͕ෆ҆ఆͳ࣌

  13. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF" 4FSWJDF# 4FSWJDF$ αʔϏε͕ෆ҆ఆͳ࣌

  14. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF" 4FSWJDF# 4FSWJDF$ αʔϏε͕ෆ҆ఆͳ࣌

  15. ©2019 Wantedly, Inc. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε ଞͷαʔϏε

  16. ©2019 Wantedly, Inc. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε ଞͷαʔϏε

  17. ©2019 Wantedly, Inc. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε ଞͷαʔϏε

  18. ©2019 Wantedly, Inc. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε ଞͷαʔϏε

  19. ©2019 Wantedly, Inc. Ͳ͏͍ͬͨ͜ͱΛ͢Δ΂͖͔ʁ

  20. ©2019 Wantedly, Inc. 1. ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔ 2. ࣮ੈքͷࣄ৅͸ଟ༷Ͱ͋Δ 3. ຊ൪؀ڥͰݕূΛ࣮ߦ͢Δ 4.

    ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ 5. ӨڹൣғΛہॴԽ͢Δ https://principlesofchaos.org/?lang=JAcontent# Ͳ͏͍ͬͨ͜ͱΛߦ͏͔
  21. ©2019 Wantedly, Inc. 1. ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔ 2. ࣮ੈքͷࣄ৅͸ଟ༷Ͱ͋Δ 3. ຊ൪؀ڥͰݕূΛ࣮ߦ͢Δ 4.

    ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ 5. ӨڹൣғΛہॴԽ͢Δ https://principlesofchaos.org/?lang=JAcontent# 43&ͱͷڠྗ΍૊৫త ͳରԠ͕ඞཁ Ͳ͏͍ͬͨ͜ͱΛߦ͏͔
  22. ©2019 Wantedly, Inc. 1. 1ඵؒͷಈը࠶ੜ਺ͳͲ 2. APIͷϨεϙϯε଎౓ͳͲ 1. ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔ =>

    ͦΕΛकΔͨΊͷ΋ͷ͕ॏཁɻͦͷ֬ೝ͕Ͱ͖Ε͹͋Δఔ౓҆৺ ྫ: AWS͕region͝ͱμ΢ϯͨ͠ࡍɺଞͷregionʹfail over͢ΔͳͲ ྫ
  23. ©2019 Wantedly, Inc. 1. Πϯελϯε͕μ΢ϯ͢Δ 2. DB͕٧·ΔɾΞΫηεͰ͖ͳ͘ͳΔɹ 3. ґଘ͢ΔଞͷαʔϏε͕ෆ҆ఆʹͳΔ etc…

    2. ࣮ੈքͷࣄ৅͸ଟ༷Ͱ͋Δ => ͜͏͍ͬͨཁૉΛೖΕΒΕΔΑ͏ʹ͢Δ = Fault Injection
  24. ©2019 Wantedly, Inc. ͜ͷ2͚ͭͩͳΒQA؀ڥͰ ΞϓϦέʔγϣϯΤϯδχΞ͚ͩͰՄೳ

  25. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF" 4FSWJDF# Fault Injectionͱͦͷ֬ೝ

  26. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF" 4FSWJDF# Fault Injectionͱͦͷ֬ೝ ֬ೝ༻ͷίʔυΛೖΕΔͷ͕໘౗

  27. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε JOUFSDFQUPS 4FSWJDF" 4FSWJDF# Fault Injectionͱͦͷ֬ೝ

  28. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε  ൪୆Λฦ͢  ஗͘ͳΔ  ઃఆ͍ͯ͠ΔλΠϜΞ΢

    τΑΓϨεϙϯε͕஗͍ JOUFSDFQUPS 4FSWJDF" 4FSWJDF# Fault Injectionͱͦͷ֬ೝ
  29. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε  ൪୆Λฦ͢  ஗͘ͳΔ  ઃఆ͍ͯ͠ΔλΠϜΞ΢

    τΑΓϨεϙϯε͕஗͍ JOUFSDFQUPS 4FSWJDF" 4FSWJDF# { "interval": 1, "slow_response_option": { "enable": true, "time": 5 }, "server_error_option": { "enable": true } } Fault Injectionͱͦͷ֬ೝ
  30. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε  ൪୆Λฦ͢  ஗͘ͳΔ  ઃఆ͍ͯ͠ΔλΠϜΞ΢

    τΑΓϨεϙϯε͕஗͍ JOUFSDFQUPS 4FSWJDF" 4FSWJDF# Fault Injectionͱͦͷ֬ೝ
  31. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε JOUFSDFQUPS 4FSWJDF" 4FSWJDF# Fault Injectionͱͦͷ֬ೝ

  32. ©2019 Wantedly, Inc. 4FSWJDF" 4FSWJDF# $MPVE1VC4VC 1VCMJTI 4VCTDSJCF Fault Injectionͱͦͷ֬ೝ

  33. ©2019 Wantedly, Inc. 4FSWJDF" 4FSWJDF# $MPVE1VC4VC 1VCMJTI 4VCTDSJCF Fault Injectionͱͦͷ֬ೝ

  34. ©2019 Wantedly, Inc. 4FSWJDF" 4FSWJDF# $MPVE1VC4VC  ॏෳ഑৴Λ͢Δ  1VCMJTIͨ͠ॱ൪Ͱ഑৴͠ͳ͍

    1VCMJTI 4VCTDSJCF Fault Injectionͱͦͷ֬ೝ
  35. ©2019 Wantedly, Inc. 4FSWJDF" 4FSWJDF# $MPVE1VC4VC  ॏෳ഑৴Λ͢Δ  1VCMJTIͨ͠ॱ൪Ͱ഑৴͠ͳ͍

    1VCMJTI 4VCTDSJCF ϥΠϒϥϦ͕ߦ͏ͷͰ JOUFSDFQUPSΛڬΊͳ͍ Fault Injectionͱͦͷ֬ೝ
  36. ©2019 Wantedly, Inc. 4FSWJDF" 4FSWJDF# $MPVE1VC4VC 1VCMJTI 4VCTDSJCF 1SPYZ 1SPYZ

    Fault Injectionͱͦͷ֬ೝ
  37. ©2019 Wantedly, Inc. 4FSWJDF" 4FSWJDF# $MPVE1VC4VC 1VCMJTI 4VCTDSJCF 1SPYZ 1SPYZ

     ॏෳ഑৴Λ͢Δ  1VCMJTIͨ͠ॱ൪Ͱ഑৴͠ͳ͍ Fault Injectionͱͦͷ֬ೝ
  38. ©2019 Wantedly, Inc. https://github.com/rerost/unstable https://github.com/rerost/chaos-pubsub

  39. ©2019 Wantedly, Inc.

  40. ©2019 Wantedly, Inc. σϝϦοτ  ݱঢ়ͷ֬ೝ͸Ͱ͖Δ͕͍ͭյΕΔ͔Θ͔Βͳ͍ɹ  ςετ͢Δͷ͕ਓखͳͷͰ֬ೝ͕໘౗  ςετ͕໢ཏͰ͖͍ͯͳ͍Մೳੑ͕͋Δ

  41. ©2019 Wantedly, Inc. ·ͱΊ  खಈͰ΋͋Δఔ౓ͷݕূ͸Ͱ͖Δ  ςετ༻ͷίʔυ͸ൈ͖΍͍͢ܗʹ͓ͯ͘͠ɹ  2"؀ڥ͚ͩͰ΋ࢼ͓ͯ͘͠ҙຯ͸͋Δɹ

     σϝϦοτ͸ଟ͍ͷͰҰൠʹ͓͢͢ΊͰ͖Δ ํ๏Ͱ͸ͳ͍