Re:silience から始めるカオスエンジニアリング生活 / A Life of Chaos Engineering Starting with Resilience

https://techconf.cookpad.com/2019/ で話す (話した) 資料です。
下記スライド内のリンクです。
- https://principlesofchaos.org/?lang=JAcontent#
- https://www.quora.com/What-is-chaos-engineering
- https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
- https://microservices.io/
- https://www.slideshare.net/hashikem/rails-presentation-667968
- https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-largest-rails-monolith
- https://techlife.cookpad.com/entry/2018/05/08/080000
- https://techlife.cookpad.com/entry/2018/04/02/140846

Transcript

1. @itkq ٕज़෦ SRE άϧʔϓ Re:silience ͔Β࢝ΊΔ ΧΦεΤϯδχΞϦϯάੜ׆
2. whoami 2 • 2018 ೥৽ଔೖࣾ • ٕज़෦ SRE άϧʔϓ • ΧΦεେਉ • औΓ૊Έ: ଱ো֐ੑͷ޲্ͳͲ • @itkq
3. Chaos Engineering 3 Χ Φ ε Τ ϯ δ χ Ξ Ϧ ϯ ά
4. Chaos Engineering ͱ͸ 4 • Principles of Chaos Engineering* • Ξφϩδʔ: ༧๷઀छ ** ΧΦεΤϯδχΞϦϯά͸ɺ෼ࢄγεςϜʹ͓͍ͯγεςϜ͕ ෆ҆ఆͳঢ়ଶʹ଱͑Δ͜ͱͷग़དྷΔ؀ڥΛߏங͢ΔͨΊͷݕূ ͷن཯Ͱ͢ɻ l ** https://www.quora.com/What-is-chaos-engineering * https://principlesofchaos.org/?lang=JAcontent#
5. Chaos Engineering ΍͍ͬͯ͘ 5 • Chaos Monkey* ͰϥϯμϜʹΠϯελϯεΛམͱ͠·͢ * https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
6. Chaos Engineering ΍͍ͬͯ͘ 6 • Chaos Monkey* ͰϥϯμϜʹΠϯελϯεΛམͱ͠·͢ མͱ͠·ͤΜ * https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
7. Chaos Engineering ͬͯͳΜ͚ͩͬ 7 • Chaos Engineering Chaos Monkey? • “ϥϯμϜʹམͱ͢͜ͱ” ͸ॏཁͰ͸ͳ͍ • ͳͥ Chaos Engineering ͕ඞཁͳͷ͔?
8. Agenda 8 • ΫοΫύου͕ Chaos Engineering Λ΍Δཧ༝ • ϚΠΫϩαʔϏεΞʔΩςΫνϟͷো֐΁ͷ໰୊ҙࣝ • ΫοΫύουͰͲͷΑ͏ʹ Chaos Engineering Λ࢝Ί͔ͨ
9. Microserivces* ΁ͷಓͷΓ 9 Ϛ Π Ϋ ϩ α ʔ Ϗ ε Ξ ʔ Ω ς Ϋ ν ϟ * https://microservices.io/
10. 10 Developers cookpad.com* * https://www.slideshare.net/hashikem/rails-presentation-667968 2008
11. 11 The Recipe for the World’s Largest Rails Monolith * * https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-largest-rails-monolith 2015 cookpad.com
12. 12 The Recipe for the World’s Largest Rails Monolith * * https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-largest-rails-monolith cookpad.com େϞϊϦεظ αʔϏε։ൃ଎౓ͷ௿Լ ϦϦʔε଎౓ͷ௿Լ
13. 13 New service! API cookpad.com
14. 14 New Internal Service! API cookpad.com
15. 15 API cookpad.com αʔϏεͷ։ൃ଎౓ͷ޲্ ϦϦʔε଎౓ͷ޲্ ෳࡶੑͷ૿Ճ Microservices ΁ͷҠߦ
16. 16 cookpad.com ೝূ Ϣʔβ৘ใ "#
    ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ
17. 17 cookpad.com ೝূ Ϣʔβ৘ใ "#
    ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ
18. 18 ೝূ Ϣʔβ৘ใ "#
    ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ cookpad.com
19. 19 ೝূ Ϣʔβ৘ใ "#
    ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ cookpad.com େ Microservices ظ αʔϏεؒ௨৴ͷঢ়گ͕͙͢ʹ෼͔Βͳ͍ ࿈࠯తো֐
20. 20 cookpad.com ೝূ Ϣʔβ৘ใ "#
    ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ Service Mesh! α ʔ Ϗ ε ϝ ο γ ϡ
21. Service Mesh in Cookpad* 21 Control plane Data plane cookpad/itacho App Service A ࣮ࡍͷ௨৴ ௨৴ͷઃఆ Service B Service C ΞϓϦέʔγϣϯϨΠϠʔ
    ͔Β௨৴Λ੾Γ཭ͨ͠
    IUUQTUFDIMJGFDPPLQBEDPNFOUSZ
22. 22 cookpad.com ೝূ Ϣʔβ৘ใ "#
    ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ Service Mesh! αʔϏεؒ௨৴ͷঢ়گ೺Ѳ͕Ͱ͖Δ ద੾ͳϦτϥΠઃఆͳͲʹΑΓҰ࣌తͳো֐ʹ଱͑ΒΕΔ Service Mesh ಋೖ
23. 23 cookpad.com ೝূ Ϣʔβ৘ใ "#
    ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ ☠ ☠
24. 24 cookpad.com ೝূ Ϣʔβ৘ใ "#
    ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ ☠ ☠ “Everything fails, all the time”
25. 25 cookpad.com ೝূ Ϣʔβ৘ใ "#
    ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ cookpad.com ґଘઌ͕͋ΔαʔϏε ্ྲྀ͕ো֐ʹͳͬͨ৔߹ ʮҾ͖ͣΒΕΔʯ΂͖…?
26. 26 ো֐ʹର͢Δ໰୊ҙࣝ • αʔϏε͸ʮਖ਼͘͠ʯյΕΔ͔ • ଱ো֐ੑ (Resilience) ͕͋Δ͔ʁ • ো֐࣌ʹͲͷΑ͏ͳڍಈΛ͢Δͷ͔ • ͢΂ͯͷτϨʔε͸೉͍͠ • ҙਤతʹো֐Λىͯ͜͠ΈΔ → Fault Injection
27. 27 Chaos Engineering for us ΧΦεΤϯδχΞϦϯά͸ɺ෼ࢄγεςϜʹ͓͍ͯγεςϜ͕ ෆ҆ఆͳঢ়ଶʹ଱͑Δ͜ͱͷग़དྷΔ؀ڥΛߏங͢ΔͨΊͷݕূ ͷن཯Ͱ͢ɻ l Microservices Ͱൃੜ͢Δো֐ʹର͢Δ
 ϓϩΞΫςΟϒͳΞϓϩʔν
28. 28 “Chaos Engineering
 ΍͍ͬͯ͘એݴ”
29. 29 cookpad.com ೝূ Ϣʔβ৘ใ "#
    ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ ☠ ☠ Important Layer: Network
30. Network Fault Injection Platform 30
31. Service Mesh in Cookpad* 31 Control plane Data plane cookpad/itacho App Service A Service B Service C Fault Injection Return 503 + 5s
    IUUQTUFDIMJGFDPPLQBEDPNFOUSZ
32. Network Fault Injection Platform 32 Control plane Data plane cookpad/itacho App Service A Service B Service C hako-console* UI ૢ࡞
    IUUQTUFDIMJGFDPPLQBEDPNFOUSZ New!
33. ͲͷΑ͏ʹ Chaos Engineering Λ ࢝ΊΔ͔ 33
34. 34 Known unknowns
35. 35 Known knowns γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͓ͬͯΓ γεςϜ B ʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌͍ͬͯΔ γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͍ͬͯΔ͕ ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍ γεςϜ B ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛߟ͑ͨ͜ͱ͕ͳ͍͠ ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍ Known unknowns Unknown unknowns ஌ࣝͷ޿͕Δํ޲
36. 36 γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͓ͬͯΓ γεςϜ B ʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌͍ͬͯΔ γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͍ͬͯΔ͕ ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍ γεςϜ B ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛߟ͑ͨ͜ͱ͕ͳ͍͠ ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍ Chaos Engineering ஌ࣝͷ޿͕Δํ޲ Known knowns Known unknowns Unknown unknowns
37. 37 Fault Injection Testing ຊ൪؀ڥͰͷܧଓతͳΧΦε࣮ݧ ஌ࣝͷ޿͕Δํ޲ Chaos Engineering Known knowns Known unknowns Unknown unknowns
38. 38 Fault Injection Testing ຊ൪؀ڥͰͷܧଓతͳΧΦε࣮ݧ ஌ࣝͷ޿͕Δํ޲ Chaos Engineering Known knowns Known unknowns Unknown unknowns ·ͣ͸͔͜͜Β࢝ΊΔ
39. 39 cookpad.com ೝূ Ϣʔβ৘ใ "#
    ςετ ݕࡧ ΫοΫύουϒϩά Network Fault Injection ຊ൪ͱಉ͡ϝογϡߏ଄ εςʔδϯά
40. 40 cookpad.com ೝূ Ϣʔβ৘ใ "#
    ςετ ݕࡧ ΫοΫύουϒϩά Network Fault Injection ຊ൪ͱಉ͡ϝογϡߏ଄ εςʔδϯά εςʔδϯάͰ Fault Injection Testing Ωϟογϡʹؔ܎͢Δ଱ো֐ੑͷ໰୊ΛൃݟɾվળͰ͖ͨ
41. ·ͱΊ 41 • ϚΠΫϩαʔϏε࣌୅ͷো֐ʹର͢Δ໰୊ҙࣝ • Network Fault Injection Λى఺ͱ͢Δ Chaos Engineering ΁ • Known-unknowns ͷݕূɾվળ͔Β࢝ΊΔ • ࠓޙ΋γεςϜશମͷ଱ো֐ੑͷ޲্ʹऔΓ૊Ή