Re:silience から始めるカオスエンジニアリング生活 / A Life of Chaos Engineering Starting with Resilience

https://techconf.cookpad.com/2019/ で話す (話した) 資料です。
下記スライド内のリンクです。
- https://principlesofchaos.org/?lang=JAcontent#
- https://www.quora.com/What-is-chaos-engineering
- https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
- https://microservices.io/
- https://www.slideshare.net/hashikem/rails-presentation-667968
- https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-largest-rails-monolith
- https://techlife.cookpad.com/entry/2018/05/08/080000
- https://techlife.cookpad.com/entry/2018/04/02/140846

Transcript

1. @itkq
   ٕज़෦ SRE άϧʔϓ
   Re:silience ͔Β࢝ΊΔ
   ΧΦεΤϯδχΞϦϯάੜ׆
   

   View Slide

2. whoami
   2
   • 2018 ೥৽ଔೖࣾ
   • ٕज़෦ SRE άϧʔϓ
   • ΧΦεେਉ
   • औΓ૊Έ: ଱ো֐ੑͷ޲্ͳͲ
   • @itkq
   

   View Slide

3. Chaos Engineering
   3
   Χ Φ ε Τ ϯ δ χ Ξ Ϧ ϯ ά
   

   View Slide

4. Chaos Engineering ͱ͸
   4
   • Principles of Chaos Engineering*
   • Ξφϩδʔ: ༧๷઀छ **
   ΧΦεΤϯδχΞϦϯά͸ɺ෼ࢄγεςϜʹ͓͍ͯγεςϜ͕
   ෆ҆ఆͳঢ়ଶʹ଱͑Δ͜ͱͷग़དྷΔ؀ڥΛߏங͢ΔͨΊͷݕূ
   ͷن཯Ͱ͢ɻ
   l
   ** https://www.quora.com/What-is-chaos-engineering
   * https://principlesofchaos.org/?lang=JAcontent#
   

   View Slide

5. Chaos Engineering ΍͍ͬͯ͘
   5
   • Chaos Monkey* ͰϥϯμϜʹΠϯελϯεΛམͱ͠·͢
   * https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
   

   View Slide

6. Chaos Engineering ΍͍ͬͯ͘
   6
   • Chaos Monkey* ͰϥϯμϜʹΠϯελϯεΛམͱ͠·͢
   མͱ͠·ͤΜ
   * https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
   

   View Slide

7. Chaos Engineering ͬͯͳΜ͚ͩͬ
   7
   • Chaos Engineering Chaos Monkey?
   • “ϥϯμϜʹམͱ͢͜ͱ” ͸ॏཁͰ͸ͳ͍
   • ͳͥ Chaos Engineering ͕ඞཁͳͷ͔?
   

   View Slide

8. Agenda
   8
   • ΫοΫύου͕ Chaos Engineering Λ΍Δཧ༝
   • ϚΠΫϩαʔϏεΞʔΩςΫνϟͷো֐΁ͷ໰୊ҙࣝ
   • ΫοΫύουͰͲͷΑ͏ʹ Chaos Engineering Λ࢝Ί͔ͨ
   

   View Slide

9. Microserivces*
   ΁ͷಓͷΓ
   9
   Ϛ Π Ϋ ϩ α ʔ Ϗ ε Ξ ʔ Ω ς Ϋ ν ϟ
   * https://microservices.io/
   

   View Slide

10. 10
    Developers
    cookpad.com*
    * https://www.slideshare.net/hashikem/rails-presentation-667968
    2008
    

    View Slide

11. 11
    The Recipe for
    the World’s Largest Rails Monolith *
    * https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-largest-rails-monolith
    2015
    cookpad.com
    

    View Slide

12. 12
    The Recipe for
    the World’s Largest Rails Monolith *
    * https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-largest-rails-monolith
    cookpad.com
    େϞϊϦεظ
    αʔϏε։ൃ଎౓ͷ௿Լ
    ϦϦʔε଎౓ͷ௿Լ
    

    View Slide

13. 13
    New service!
    API
    cookpad.com
    

    View Slide

14. 14
    New Internal Service!
    API
    cookpad.com
    

    View Slide

15. 15
    API
    cookpad.com
    αʔϏεͷ։ൃ଎౓ͷ޲্
    ϦϦʔε଎౓ͷ޲্
    ෳࡶੑͷ૿Ճ
    Microservices ΁ͷҠߦ
    

    View Slide

16. 16
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ
    

    View Slide

17. 17
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ
    
    

    View Slide

18. 18
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ
    
    
    
    
    cookpad.com
    

    View Slide

19. 19
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ
    
    
    
    
    cookpad.com
    େ Microservices ظ
    αʔϏεؒ௨৴ͷঢ়گ͕͙͢ʹ෼͔Βͳ͍
    ࿈࠯తো֐
    

    View Slide

20. 20
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ
    Service Mesh!
    α ʔ Ϗ ε ϝ ο γ ϡ
    

    View Slide

21. Service Mesh in Cookpad*
    21
    Control plane
    Data plane
    cookpad/itacho
    App
    Service A
    ࣮ࡍͷ௨৴
    ௨৴ͷઃఆ
    Service B
    Service C
    ΞϓϦέʔγϣϯϨΠϠʔ
    ͔Β௨৴Λ੾Γ཭ͨ͠
    IUUQTUFDIMJGFDPPLQBEDPNFOUSZ
    

    View Slide

22. 22
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ
    Service Mesh!
    αʔϏεؒ௨৴ͷঢ়گ೺Ѳ͕Ͱ͖Δ
    ద੾ͳϦτϥΠઃఆͳͲʹΑΓҰ࣌తͳো֐ʹ଱͑ΒΕΔ
    Service Mesh ಋೖ
    

    View Slide

23. 23
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ
    ☠
    ☠
    

    View Slide

24. 24
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ
    ☠
    ☠
    “Everything fails, all the time”
    

    View Slide

25. 25
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ
    cookpad.com
    ґଘઌ͕͋ΔαʔϏε
    ্ྲྀ͕ো֐ʹͳͬͨ৔߹
    ʮҾ͖ͣΒΕΔʯ΂͖…?
    

    View Slide

26. 26
    ো֐ʹର͢Δ໰୊ҙࣝ
    • αʔϏε͸ʮਖ਼͘͠ʯյΕΔ͔
    • ଱ো֐ੑ (Resilience) ͕͋Δ͔ʁ
    • ো֐࣌ʹͲͷΑ͏ͳڍಈΛ͢Δͷ͔
    • ͢΂ͯͷτϨʔε͸೉͍͠
    • ҙਤతʹো֐Λىͯ͜͠ΈΔ → Fault Injection
    

    View Slide

27. 27
    Chaos Engineering for us
    ΧΦεΤϯδχΞϦϯά͸ɺ෼ࢄγεςϜʹ͓͍ͯγεςϜ͕
    ෆ҆ఆͳঢ়ଶʹ଱͑Δ͜ͱͷग़དྷΔ؀ڥΛߏங͢ΔͨΊͷݕূ
    ͷن཯Ͱ͢ɻ
    l
    Microservices Ͱൃੜ͢Δো֐ʹର͢Δ

    ϓϩΞΫςΟϒͳΞϓϩʔν
    

    View Slide

28. 28
    “Chaos Engineering

    ΍͍ͬͯ͘એݴ”
    

    View Slide

29. 29
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ
    ☠
    ☠
    Important Layer: Network
    

    View Slide

30. Network Fault
    Injection Platform
    30
    

    View Slide

31. Service Mesh in Cookpad*
    31
    Control plane
    Data plane
    cookpad/itacho
    App
    Service A
    Service B
    Service C
    Fault Injection
    Return 503
    + 5s
    IUUQTUFDIMJGFDPPLQBEDPNFOUSZ
    

    View Slide

32. Network Fault Injection Platform
    32
    Control plane
    Data plane
    cookpad/itacho
    App
    Service A
    Service B
    Service C
    hako-console*
    UI
    ૢ࡞
    IUUQTUFDIMJGFDPPLQBEDPNFOUSZ
    New!
    

    View Slide

33. ͲͷΑ͏ʹ
    Chaos Engineering Λ
    ࢝ΊΔ͔
    33
    

    View Slide

34. 34
    Known unknowns
    

    View Slide

35. 35
    Known knowns
    γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͓ͬͯΓ
    γεςϜ B ʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌͍ͬͯΔ
    γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͍ͬͯΔ͕
    ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍
    γεςϜ B ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛߟ͑ͨ͜ͱ͕ͳ͍͠
    ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍
    Known unknowns
    Unknown unknowns
    ஌ࣝͷ޿͕Δํ޲
    

    View Slide

36. 36
    γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͓ͬͯΓ
    γεςϜ B ʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌͍ͬͯΔ
    γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͍ͬͯΔ͕
    ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍
    γεςϜ B ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛߟ͑ͨ͜ͱ͕ͳ͍͠
    ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍
    Chaos Engineering
    ஌ࣝͷ޿͕Δํ޲
    Known knowns
    Known unknowns
    Unknown unknowns
    

    View Slide

37. 37
    Fault Injection Testing
    ຊ൪؀ڥͰͷܧଓతͳΧΦε࣮ݧ
    ஌ࣝͷ޿͕Δํ޲
    Chaos Engineering
    Known knowns
    Known unknowns
    Unknown unknowns
    

    View Slide

38. 38
    Fault Injection Testing
    ຊ൪؀ڥͰͷܧଓతͳΧΦε࣮ݧ
    ஌ࣝͷ޿͕Δํ޲
    Chaos Engineering
    Known knowns
    Known unknowns
    Unknown unknowns
    ·ͣ͸͔͜͜Β࢝ΊΔ
    

    View Slide

39. 39
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    Network Fault Injection
    ຊ൪ͱಉ͡ϝογϡߏ଄
    εςʔδϯά
    

    View Slide

40. 40
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    Network Fault Injection
    ຊ൪ͱಉ͡ϝογϡߏ଄
    εςʔδϯά
    εςʔδϯάͰ Fault Injection Testing
    Ωϟογϡʹؔ܎͢Δ଱ো֐ੑͷ໰୊ΛൃݟɾվળͰ͖ͨ
    

    View Slide

41. ·ͱΊ
    41
    • ϚΠΫϩαʔϏε࣌୅ͷো֐ʹର͢Δ໰୊ҙࣝ
    • Network Fault Injection Λى఺ͱ͢Δ Chaos Engineering ΁
    • Known-unknowns ͷݕূɾվળ͔Β࢝ΊΔ
    • ࠓޙ΋γεςϜશମͷ଱ো֐ੑͷ޲্ʹऔΓ૊Ή
    

    View Slide