Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Re:silience から始めるカオスエンジニアリング生活 / A Life of Chaos Engineering Starting with Resilience

itkq
February 27, 2019

Re:silience から始めるカオスエンジニアリング生活 / A Life of Chaos Engineering Starting with Resilience

itkq

February 27, 2019
Tweet

More Decks by itkq

Other Decks in Technology

Transcript

  1. @itkq
    ٕज़෦ SRE άϧʔϓ
    Re:silience ͔Β࢝ΊΔ
    ΧΦεΤϯδχΞϦϯάੜ׆

    View Slide

  2. whoami
    2
    • 2018 ೥৽ଔೖࣾ
    • ٕज़෦ SRE άϧʔϓ
    • ΧΦεେਉ
    • औΓ૊Έ: ଱ো֐ੑͷ޲্ͳͲ
    • @itkq

    View Slide

  3. Chaos Engineering
    3
    Χ Φ ε Τ ϯ δ χ Ξ Ϧ ϯ ά

    View Slide

  4. Chaos Engineering ͱ͸
    4
    • Principles of Chaos Engineering*
    • Ξφϩδʔ: ༧๷઀छ **
    ΧΦεΤϯδχΞϦϯά͸ɺ෼ࢄγεςϜʹ͓͍ͯγεςϜ͕
    ෆ҆ఆͳঢ়ଶʹ଱͑Δ͜ͱͷग़དྷΔ؀ڥΛߏங͢ΔͨΊͷݕূ
    ͷن཯Ͱ͢ɻ
    l
    ** https://www.quora.com/What-is-chaos-engineering
    * https://principlesofchaos.org/?lang=JAcontent#

    View Slide

  5. Chaos Engineering ΍͍ͬͯ͘
    5
    • Chaos Monkey* ͰϥϯμϜʹΠϯελϯεΛམͱ͠·͢
    * https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey

    View Slide

  6. Chaos Engineering ΍͍ͬͯ͘
    6
    • Chaos Monkey* ͰϥϯμϜʹΠϯελϯεΛམͱ͠·͢
    མͱ͠·ͤΜ
    * https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey

    View Slide

  7. Chaos Engineering ͬͯͳΜ͚ͩͬ
    7
    • Chaos Engineering Chaos Monkey?
    • “ϥϯμϜʹམͱ͢͜ͱ” ͸ॏཁͰ͸ͳ͍
    • ͳͥ Chaos Engineering ͕ඞཁͳͷ͔?

    View Slide

  8. Agenda
    8
    • ΫοΫύου͕ Chaos Engineering Λ΍Δཧ༝
    • ϚΠΫϩαʔϏεΞʔΩςΫνϟͷো֐΁ͷ໰୊ҙࣝ
    • ΫοΫύουͰͲͷΑ͏ʹ Chaos Engineering Λ࢝Ί͔ͨ

    View Slide

  9. Microserivces*
    ΁ͷಓͷΓ
    9
    Ϛ Π Ϋ ϩ α ʔ Ϗ ε Ξ ʔ Ω ς Ϋ ν ϟ
    * https://microservices.io/

    View Slide

  10. 10
    Developers
    cookpad.com*
    * https://www.slideshare.net/hashikem/rails-presentation-667968
    2008

    View Slide

  11. 11
    The Recipe for
    the World’s Largest Rails Monolith *
    * https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-largest-rails-monolith
    2015
    cookpad.com

    View Slide

  12. 12
    The Recipe for
    the World’s Largest Rails Monolith *
    * https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-largest-rails-monolith
    cookpad.com
    େϞϊϦεظ
    αʔϏε։ൃ଎౓ͷ௿Լ
    ϦϦʔε଎౓ͷ௿Լ

    View Slide

  13. 13
    New service!
    API
    cookpad.com

    View Slide

  14. 14
    New Internal Service!
    API
    cookpad.com

    View Slide

  15. 15
    API
    cookpad.com
    αʔϏεͷ։ൃ଎౓ͷ޲্
    ϦϦʔε଎౓ͷ޲্
    ෳࡶੑͷ૿Ճ
    Microservices ΁ͷҠߦ

    View Slide

  16. 16
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ

    View Slide

  17. 17
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ

    View Slide

  18. 18
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ




    cookpad.com

    View Slide

  19. 19
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ




    cookpad.com
    େ Microservices ظ
    αʔϏεؒ௨৴ͷঢ়گ͕͙͢ʹ෼͔Βͳ͍
    ࿈࠯తো֐

    View Slide

  20. 20
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ
    Service Mesh!
    α ʔ Ϗ ε ϝ ο γ ϡ

    View Slide

  21. Service Mesh in Cookpad*
    21
    Control plane
    Data plane
    cookpad/itacho
    App
    Service A
    ࣮ࡍͷ௨৴
    ௨৴ͷઃఆ
    Service B
    Service C
    ΞϓϦέʔγϣϯϨΠϠʔ
    ͔Β௨৴Λ੾Γ཭ͨ͠
    IUUQTUFDIMJGFDPPLQBEDPNFOUSZ

    View Slide

  22. 22
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ
    Service Mesh!
    αʔϏεؒ௨৴ͷঢ়گ೺Ѳ͕Ͱ͖Δ
    ద੾ͳϦτϥΠઃఆͳͲʹΑΓҰ࣌తͳো֐ʹ଱͑ΒΕΔ
    Service Mesh ಋೖ

    View Slide

  23. 23
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ


    View Slide

  24. 24
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ


    “Everything fails, all the time”

    View Slide

  25. 25
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ
    cookpad.com
    ґଘઌ͕͋ΔαʔϏε
    ্ྲྀ͕ো֐ʹͳͬͨ৔߹
    ʮҾ͖ͣΒΕΔʯ΂͖…?

    View Slide

  26. 26
    ো֐ʹର͢Δ໰୊ҙࣝ
    • αʔϏε͸ʮਖ਼͘͠ʯյΕΔ͔
    • ଱ো֐ੑ (Resilience) ͕͋Δ͔ʁ
    • ো֐࣌ʹͲͷΑ͏ͳڍಈΛ͢Δͷ͔
    • ͢΂ͯͷτϨʔε͸೉͍͠
    • ҙਤతʹো֐Λىͯ͜͠ΈΔ → Fault Injection

    View Slide

  27. 27
    Chaos Engineering for us
    ΧΦεΤϯδχΞϦϯά͸ɺ෼ࢄγεςϜʹ͓͍ͯγεςϜ͕
    ෆ҆ఆͳঢ়ଶʹ଱͑Δ͜ͱͷग़དྷΔ؀ڥΛߏங͢ΔͨΊͷݕূ
    ͷن཯Ͱ͢ɻ
    l
    Microservices Ͱൃੜ͢Δো֐ʹର͢Δ

    ϓϩΞΫςΟϒͳΞϓϩʔν

    View Slide

  28. 28
    “Chaos Engineering

    ΍͍ͬͯ͘એݴ”

    View Slide

  29. 29
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    DPPLQBE57
    ΫοΫύουϚʔτ


    Important Layer: Network

    View Slide

  30. Network Fault
    Injection Platform
    30

    View Slide

  31. Service Mesh in Cookpad*
    31
    Control plane
    Data plane
    cookpad/itacho
    App
    Service A
    Service B
    Service C
    Fault Injection
    Return 503
    + 5s
    IUUQTUFDIMJGFDPPLQBEDPNFOUSZ

    View Slide

  32. Network Fault Injection Platform
    32
    Control plane
    Data plane
    cookpad/itacho
    App
    Service A
    Service B
    Service C
    hako-console*
    UI
    ૢ࡞
    IUUQTUFDIMJGFDPPLQBEDPNFOUSZ
    New!

    View Slide

  33. ͲͷΑ͏ʹ
    Chaos Engineering Λ
    ࢝ΊΔ͔
    33

    View Slide

  34. 34
    Known unknowns

    View Slide

  35. 35
    Known knowns
    γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͓ͬͯΓ
    γεςϜ B ʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌͍ͬͯΔ
    γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͍ͬͯΔ͕
    ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍
    γεςϜ B ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛߟ͑ͨ͜ͱ͕ͳ͍͠
    ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍
    Known unknowns
    Unknown unknowns
    ஌ࣝͷ޿͕Δํ޲

    View Slide

  36. 36
    γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͓ͬͯΓ
    γεςϜ B ʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌͍ͬͯΔ
    γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͍ͬͯΔ͕
    ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍
    γεςϜ B ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛߟ͑ͨ͜ͱ͕ͳ͍͠
    ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍
    Chaos Engineering
    ஌ࣝͷ޿͕Δํ޲
    Known knowns
    Known unknowns
    Unknown unknowns

    View Slide

  37. 37
    Fault Injection Testing
    ຊ൪؀ڥͰͷܧଓతͳΧΦε࣮ݧ
    ஌ࣝͷ޿͕Δํ޲
    Chaos Engineering
    Known knowns
    Known unknowns
    Unknown unknowns

    View Slide

  38. 38
    Fault Injection Testing
    ຊ൪؀ڥͰͷܧଓతͳΧΦε࣮ݧ
    ஌ࣝͷ޿͕Δํ޲
    Chaos Engineering
    Known knowns
    Known unknowns
    Unknown unknowns
    ·ͣ͸͔͜͜Β࢝ΊΔ

    View Slide

  39. 39
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    Network Fault Injection
    ຊ൪ͱಉ͡ϝογϡߏ଄
    εςʔδϯά

    View Slide

  40. 40
    cookpad.com
    ೝূ
    Ϣʔβ৘ใ
    "#ςετ
    ݕࡧ
    ΫοΫύουϒϩά
    Network Fault Injection
    ຊ൪ͱಉ͡ϝογϡߏ଄
    εςʔδϯά
    εςʔδϯάͰ Fault Injection Testing
    Ωϟογϡʹؔ܎͢Δ଱ো֐ੑͷ໰୊ΛൃݟɾվળͰ͖ͨ

    View Slide

  41. ·ͱΊ
    41
    • ϚΠΫϩαʔϏε࣌୅ͷো֐ʹର͢Δ໰୊ҙࣝ
    • Network Fault Injection Λى఺ͱ͢Δ Chaos Engineering ΁
    • Known-unknowns ͷݕূɾվળ͔Β࢝ΊΔ
    • ࠓޙ΋γεςϜશମͷ଱ো֐ੑͷ޲্ʹऔΓ૊Ή

    View Slide