Re:silience から始めるカオスエンジニアリング生活 / A Life of Chaos Engineering Starting with Resilience

E4619fc2a039391a1677beeac58dd487?s=47 itkq
February 27, 2019

Re:silience から始めるカオスエンジニアリング生活 / A Life of Chaos Engineering Starting with Resilience

E4619fc2a039391a1677beeac58dd487?s=128

itkq

February 27, 2019
Tweet

Transcript

  1. @itkq ٕज़෦ SRE άϧʔϓ Re:silience ͔Β࢝ΊΔ ΧΦεΤϯδχΞϦϯάੜ׆

  2. whoami 2 • 2018 ೥৽ଔೖࣾ • ٕज़෦ SRE άϧʔϓ •

    ΧΦεେਉ • औΓ૊Έ: ଱ো֐ੑͷ޲্ͳͲ • @itkq
  3. Chaos Engineering 3 Χ Φ ε Τ ϯ δ χ

    Ξ Ϧ ϯ ά
  4. Chaos Engineering ͱ͸ 4 • Principles of Chaos Engineering* •

    Ξφϩδʔ: ༧๷઀छ ** ΧΦεΤϯδχΞϦϯά͸ɺ෼ࢄγεςϜʹ͓͍ͯγεςϜ͕ ෆ҆ఆͳঢ়ଶʹ଱͑Δ͜ͱͷग़དྷΔ؀ڥΛߏங͢ΔͨΊͷݕূ ͷن཯Ͱ͢ɻ l ** https://www.quora.com/What-is-chaos-engineering * https://principlesofchaos.org/?lang=JAcontent#
  5. Chaos Engineering ΍͍ͬͯ͘ 5 • Chaos Monkey* ͰϥϯμϜʹΠϯελϯεΛམͱ͠·͢ * https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey

  6. Chaos Engineering ΍͍ͬͯ͘ 6 • Chaos Monkey* ͰϥϯμϜʹΠϯελϯεΛམͱ͠·͢ མͱ͠·ͤΜ *

    https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
  7. Chaos Engineering ͬͯͳΜ͚ͩͬ 7 • Chaos Engineering Chaos Monkey? •

    “ϥϯμϜʹམͱ͢͜ͱ” ͸ॏཁͰ͸ͳ͍ • ͳͥ Chaos Engineering ͕ඞཁͳͷ͔?
  8. Agenda 8 • ΫοΫύου͕ Chaos Engineering Λ΍Δཧ༝ • ϚΠΫϩαʔϏεΞʔΩςΫνϟͷো֐΁ͷ໰୊ҙࣝ •

    ΫοΫύουͰͲͷΑ͏ʹ Chaos Engineering Λ࢝Ί͔ͨ
  9. Microserivces* ΁ͷಓͷΓ 9 Ϛ Π Ϋ ϩ α ʔ Ϗ

    ε Ξ ʔ Ω ς Ϋ ν ϟ * https://microservices.io/
  10. 10 Developers cookpad.com* * https://www.slideshare.net/hashikem/rails-presentation-667968 2008

  11. 11 The Recipe for the World’s Largest Rails Monolith *

    * https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-largest-rails-monolith 2015 cookpad.com
  12. 12 The Recipe for the World’s Largest Rails Monolith *

    * https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-largest-rails-monolith cookpad.com େϞϊϦεظ αʔϏε։ൃ଎౓ͷ௿Լ ϦϦʔε଎౓ͷ௿Լ
  13. 13 New service! API cookpad.com

  14. 14 New Internal Service! API cookpad.com

  15. 15 API cookpad.com αʔϏεͷ։ൃ଎౓ͷ޲্ ϦϦʔε଎౓ͷ޲্ ෳࡶੑͷ૿Ճ Microservices ΁ͷҠߦ

  16. 16 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ

  17. 17 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ

  18. 18 ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ cookpad.com

  19. 19 ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ cookpad.com େ

    Microservices ظ αʔϏεؒ௨৴ͷঢ়گ͕͙͢ʹ෼͔Βͳ͍ ࿈࠯తো֐
  20. 20 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ Service

    Mesh! α ʔ Ϗ ε ϝ ο γ ϡ
  21. Service Mesh in Cookpad* 21 Control plane Data plane cookpad/itacho

    App Service A ࣮ࡍͷ௨৴ ௨৴ͷઃఆ Service B Service C ΞϓϦέʔγϣϯϨΠϠʔ ͔Β௨৴Λ੾Γ཭ͨ͠ IUUQTUFDIMJGFDPPLQBEDPNFOUSZ
  22. 22 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ Service

    Mesh! αʔϏεؒ௨৴ͷঢ়گ೺Ѳ͕Ͱ͖Δ ద੾ͳϦτϥΠઃఆͳͲʹΑΓҰ࣌తͳো֐ʹ଱͑ΒΕΔ Service Mesh ಋೖ
  23. 23 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ ☠

  24. 24 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ ☠

    ☠ “Everything fails, all the time”
  25. 25 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ cookpad.com

    ґଘઌ͕͋ΔαʔϏε ্ྲྀ͕ো֐ʹͳͬͨ৔߹ ʮҾ͖ͣΒΕΔʯ΂͖…?
  26. 26 ো֐ʹର͢Δ໰୊ҙࣝ • αʔϏε͸ʮਖ਼͘͠ʯյΕΔ͔ • ଱ো֐ੑ (Resilience) ͕͋Δ͔ʁ • ো֐࣌ʹͲͷΑ͏ͳڍಈΛ͢Δͷ͔

    • ͢΂ͯͷτϨʔε͸೉͍͠ • ҙਤతʹো֐Λىͯ͜͠ΈΔ → Fault Injection
  27. 27 Chaos Engineering for us ΧΦεΤϯδχΞϦϯά͸ɺ෼ࢄγεςϜʹ͓͍ͯγεςϜ͕ ෆ҆ఆͳঢ়ଶʹ଱͑Δ͜ͱͷग़དྷΔ؀ڥΛߏங͢ΔͨΊͷݕূ ͷن཯Ͱ͢ɻ l Microservices

    Ͱൃੜ͢Δো֐ʹର͢Δ
 ϓϩΞΫςΟϒͳΞϓϩʔν
  28. 28 “Chaos Engineering
 ΍͍ͬͯ͘એݴ”

  29. 29 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ ☠

    ☠ Important Layer: Network
  30. Network Fault Injection Platform 30

  31. Service Mesh in Cookpad* 31 Control plane Data plane cookpad/itacho

    App Service A Service B Service C Fault Injection Return 503 + 5s IUUQTUFDIMJGFDPPLQBEDPNFOUSZ
  32. Network Fault Injection Platform 32 Control plane Data plane cookpad/itacho

    App Service A Service B Service C hako-console* UI ૢ࡞ IUUQTUFDIMJGFDPPLQBEDPNFOUSZ New!
  33. ͲͷΑ͏ʹ Chaos Engineering Λ ࢝ΊΔ͔ 33

  34. 34 Known unknowns

  35. 35 Known knowns γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͓ͬͯΓ γεςϜ B ʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌͍ͬͯΔ γεςϜ

    A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͍ͬͯΔ͕ ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍ γεςϜ B ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛߟ͑ͨ͜ͱ͕ͳ͍͠ ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍ Known unknowns Unknown unknowns ஌ࣝͷ޿͕Δํ޲
  36. 36 γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͓ͬͯΓ γεςϜ B ʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌͍ͬͯΔ γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͍ͬͯΔ͕

    ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍ γεςϜ B ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛߟ͑ͨ͜ͱ͕ͳ͍͠ ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍ Chaos Engineering ஌ࣝͷ޿͕Δํ޲ Known knowns Known unknowns Unknown unknowns
  37. 37 Fault Injection Testing ຊ൪؀ڥͰͷܧଓతͳΧΦε࣮ݧ ஌ࣝͷ޿͕Δํ޲ Chaos Engineering Known knowns

    Known unknowns Unknown unknowns
  38. 38 Fault Injection Testing ຊ൪؀ڥͰͷܧଓతͳΧΦε࣮ݧ ஌ࣝͷ޿͕Δํ޲ Chaos Engineering Known knowns

    Known unknowns Unknown unknowns ·ͣ͸͔͜͜Β࢝ΊΔ
  39. 39 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά Network Fault Injection

    ຊ൪ͱಉ͡ϝογϡߏ଄ εςʔδϯά
  40. 40 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά Network Fault Injection

    ຊ൪ͱಉ͡ϝογϡߏ଄ εςʔδϯά εςʔδϯάͰ Fault Injection Testing Ωϟογϡʹؔ܎͢Δ଱ো֐ੑͷ໰୊ΛൃݟɾվળͰ͖ͨ
  41. ·ͱΊ 41 • ϚΠΫϩαʔϏε࣌୅ͷো֐ʹର͢Δ໰୊ҙࣝ • Network Fault Injection Λى఺ͱ͢Δ Chaos

    Engineering ΁ • Known-unknowns ͷݕূɾվળ͔Β࢝ΊΔ • ࠓޙ΋γεςϜશମͷ଱ো֐ੑͷ޲্ʹऔΓ૊Ή