Re:silience から始めるカオスエンジニアリング生活 / A Life of Chaos Engineering Starting with Resilience

by itkq

Slide 1

Slide 1 text

@itkq ٕज़෦ SRE άϧʔϓ Re:silience ͔Β࢝ΊΔ ΧΦεΤϯδχΞϦϯάੜ׆

Slide 2

Slide 2 text

whoami 2 • 2018 ೥৽ଔೖࣾ • ٕज़෦ SRE άϧʔϓ • ΧΦεେਉ • औΓ૊Έ: ଱ো֐ੑͷ޲্ͳͲ • @itkq

Slide 3

Slide 3 text

Chaos Engineering 3 Χ Φ ε Τ ϯ δ χ Ξ Ϧ ϯ ά

Slide 4

Slide 4 text

Chaos Engineering ͱ͸ 4 • Principles of Chaos Engineering* • Ξφϩδʔ: ༧๷઀छ ** ΧΦεΤϯδχΞϦϯά͸ɺ෼ࢄγεςϜʹ͓͍ͯγεςϜ͕ ෆ҆ఆͳঢ়ଶʹ଱͑Δ͜ͱͷग़དྷΔ؀ڥΛߏங͢ΔͨΊͷݕূ ͷن཯Ͱ͢ɻ l ** https://www.quora.com/What-is-chaos-engineering * https://principlesofchaos.org/?lang=JAcontent#

Slide 5

Slide 5 text

Chaos Engineering ΍͍ͬͯ͘ 5 • Chaos Monkey* ͰϥϯμϜʹΠϯελϯεΛམͱ͠·͢ * https://github.com/Netﬂix/SimianArmy/wiki/Chaos-Monkey

Slide 6

Slide 6 text

Chaos Engineering ΍͍ͬͯ͘ 6 • Chaos Monkey* ͰϥϯμϜʹΠϯελϯεΛམͱ͠·͢ མͱ͠·ͤΜ * https://github.com/Netﬂix/SimianArmy/wiki/Chaos-Monkey

Slide 7

Slide 7 text

Chaos Engineering ͬͯͳΜ͚ͩͬ 7 • Chaos Engineering Chaos Monkey? • “ϥϯμϜʹམͱ͢͜ͱ” ͸ॏཁͰ͸ͳ͍ • ͳͥ Chaos Engineering ͕ඞཁͳͷ͔?

Slide 8

Slide 8 text

Agenda 8 • ΫοΫύου͕ Chaos Engineering Λ΍Δཧ༝ • ϚΠΫϩαʔϏεΞʔΩςΫνϟͷো֐΁ͷ໰୊ҙࣝ • ΫοΫύουͰͲͷΑ͏ʹ Chaos Engineering Λ࢝Ί͔ͨ

Slide 9

Slide 9 text

Microserivces* ΁ͷಓͷΓ 9 Ϛ Π Ϋ ϩ α ʔ Ϗ ε Ξ ʔ Ω ς Ϋ ν ϟ * https://microservices.io/

Slide 10

Slide 10 text

10 Developers cookpad.com* * https://www.slideshare.net/hashikem/rails-presentation-667968 2008

Slide 11

Slide 11 text

11 The Recipe for the World’s Largest Rails Monolith * * https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-largest-rails-monolith 2015 cookpad.com

Slide 12

Slide 12 text

12 The Recipe for the World’s Largest Rails Monolith * * https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-largest-rails-monolith cookpad.com େϞϊϦεظ αʔϏε։ൃ଎౓ͷ௿Լ ϦϦʔε଎౓ͷ௿Լ

Slide 13

Slide 13 text

13 New service! API cookpad.com

Slide 14

Slide 14 text

14 New Internal Service! API cookpad.com

Slide 15

Slide 15 text

15 API cookpad.com αʔϏεͷ։ൃ଎౓ͷ޲্ ϦϦʔε଎౓ͷ޲্ ෳࡶੑͷ૿Ճ Microservices ΁ͷҠߦ

Slide 16

Slide 16 text

16 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ

Slide 17

Slide 17 text

17 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ

Slide 18

Slide 18 text

18 ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ cookpad.com

Slide 19

Slide 19 text

19 ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ cookpad.com େ Microservices ظ αʔϏεؒ௨৴ͷঢ়گ͕͙͢ʹ෼͔Βͳ͍ ࿈࠯తো֐

Slide 20

Slide 20 text

20 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ Service Mesh! α ʔ Ϗ ε ϝ ο γ ϡ

Slide 21

Slide 21 text

Service Mesh in Cookpad* 21 Control plane Data plane cookpad/itacho App Service A ࣮ࡍͷ௨৴ ௨৴ͷઃఆ Service B Service C ΞϓϦέʔγϣϯϨΠϠʔ ͔Β௨৴Λ੾Γ཭ͨ͠ IUUQTUFDIMJGFDPPLQBEDPNFOUSZ

Slide 22

Slide 22 text

22 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ Service Mesh! αʔϏεؒ௨৴ͷঢ়گ೺Ѳ͕Ͱ͖Δ ద੾ͳϦτϥΠઃఆͳͲʹΑΓҰ࣌తͳো֐ʹ଱͑ΒΕΔ Service Mesh ಋೖ

Slide 23

Slide 23 text

23 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ ☠ ☠

Slide 24

Slide 24 text

24 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ ☠ ☠ “Everything fails, all the time”

Slide 25

Slide 25 text

25 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ cookpad.com ґଘઌ͕͋ΔαʔϏε ্ྲྀ͕ো֐ʹͳͬͨ৔߹ ʮҾ͖ͣΒΕΔʯ΂͖…?

Slide 26

Slide 26 text

26 ো֐ʹର͢Δ໰୊ҙࣝ • αʔϏε͸ʮਖ਼͘͠ʯյΕΔ͔ • ଱ো֐ੑ (Resilience) ͕͋Δ͔ʁ • ো֐࣌ʹͲͷΑ͏ͳڍಈΛ͢Δͷ͔ • ͢΂ͯͷτϨʔε͸೉͍͠ • ҙਤతʹো֐Λىͯ͜͠ΈΔ → Fault Injection

Slide 27

Slide 27 text

27 Chaos Engineering for us ΧΦεΤϯδχΞϦϯά͸ɺ෼ࢄγεςϜʹ͓͍ͯγεςϜ͕ ෆ҆ఆͳঢ়ଶʹ଱͑Δ͜ͱͷग़དྷΔ؀ڥΛߏங͢ΔͨΊͷݕূ ͷن཯Ͱ͢ɻ l Microservices Ͱൃੜ͢Δো֐ʹର͢Δ  ϓϩΞΫςΟϒͳΞϓϩʔν

Slide 28

Slide 28 text

28 “Chaos Engineering  ΍͍ͬͯ͘એݴ”

Slide 29

Slide 29 text

29 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά DPPLQBE57 ΫοΫύουϚʔτ ☠ ☠ Important Layer: Network

Slide 30

Slide 30 text

Network Fault Injection Platform 30

Slide 31

Slide 31 text

Service Mesh in Cookpad* 31 Control plane Data plane cookpad/itacho App Service A Service B Service C Fault Injection Return 503 + 5s IUUQTUFDIMJGFDPPLQBEDPNFOUSZ

Slide 32

Slide 32 text

Network Fault Injection Platform 32 Control plane Data plane cookpad/itacho App Service A Service B Service C hako-console* UI ૢ࡞ IUUQTUFDIMJGFDPPLQBEDPNFOUSZ New!

Slide 33

Slide 33 text

ͲͷΑ͏ʹ Chaos Engineering Λ ࢝ΊΔ͔ 33

Slide 34

Slide 34 text

34 Known unknowns

Slide 35

Slide 35 text

35 Known knowns γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͓ͬͯΓ γεςϜ B ʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌͍ͬͯΔ γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͍ͬͯΔ͕ ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍ γεςϜ B ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛߟ͑ͨ͜ͱ͕ͳ͍͠ ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍ Known unknowns Unknown unknowns ஌ࣝͷ޿͕Δํ޲

Slide 36

Slide 36 text

36 γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͓ͬͯΓ γεςϜ B ʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌͍ͬͯΔ γεςϜ A ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛ஌͍ͬͯΔ͕ ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍ γεςϜ B ͕Ԡ౴͠ͳ͘ͳΔ͜ͱΛߟ͑ͨ͜ͱ͕ͳ͍͠ ଞͷγεςϜʹͲΕ͙Β͍ӨڹΛٴ΅͔͢஌Βͳ͍ Chaos Engineering ஌ࣝͷ޿͕Δํ޲ Known knowns Known unknowns Unknown unknowns

Slide 37

Slide 37 text

37 Fault Injection Testing ຊ൪؀ڥͰͷܧଓతͳΧΦε࣮ݧ ஌ࣝͷ޿͕Δํ޲ Chaos Engineering Known knowns Known unknowns Unknown unknowns

Slide 38

Slide 38 text

38 Fault Injection Testing ຊ൪؀ڥͰͷܧଓతͳΧΦε࣮ݧ ஌ࣝͷ޿͕Δํ޲ Chaos Engineering Known knowns Known unknowns Unknown unknowns ·ͣ͸͔͜͜Β࢝ΊΔ

Slide 39

Slide 39 text

39 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά Network Fault Injection ຊ൪ͱಉ͡ϝογϡߏ଄ εςʔδϯά

Slide 40

Slide 40 text

40 cookpad.com ೝূ Ϣʔβ৘ใ "#ςετ ݕࡧ ΫοΫύουϒϩά Network Fault Injection ຊ൪ͱಉ͡ϝογϡߏ଄ εςʔδϯά εςʔδϯάͰ Fault Injection Testing Ωϟογϡʹؔ܎͢Δ଱ো֐ੑͷ໰୊ΛൃݟɾվળͰ͖ͨ

Slide 41

Slide 41 text

·ͱΊ 41 • ϚΠΫϩαʔϏε࣌୅ͷো֐ʹର͢Δ໰୊ҙࣝ • Network Fault Injection Λى఺ͱ͢Δ Chaos Engineering ΁ • Known-unknowns ͷݕূɾվળ͔Β࢝ΊΔ • ࠓޙ΋γεςϜશମͷ଱ো֐ੑͷ޲্ʹऔΓ૊Ή