SREの組織的実践 / Organized SRE

Slide 1

Slide 1 text

Slide 2

Slide 2 text

2 ࣗݾ঺հ ௶಺ ༎थ / Ώ͏͏͖ https://yuuk.io/ ܦྺ גࣜձࣾ͸ͯͳ WebΦϖϨʔγϣϯΤϯδχΞɾSRE ͘͞ΒΠϯλʔωοτגࣜձࣾ ͘͞ΒΠϯλʔωοτݚڀॴ ݚڀһ WebαʔϏεͷ ։ൃɾӡ༻ WebɾΠϯλʔωοτ ج൫ٕज़ݚڀ 5೥ ݱࡏ Site Reliability Engineeringશൠ ಛʹ Monitoring, Data-Intensive Applications ઐ໳෼໺

Slide 3

Slide 3 text

https://blog.yuuk.io/entry/2019/thinking-sre

Slide 4

Slide 4 text

4 ίϯϐϡʔλʔγεςϜͷӡ༻ʹ͍ͭͯɼ3೥΄Ͳલʹࣗ෼ ͕΋͍ͬͯͨ࣍ͷ2ͭͷ໰୊ҙࣝʹର͢Δճ౴Λ঺հ͢Δ • ӡ༻ͷ͋Δ΂͖͕࢟Θ͔Βͳ͍ • ͋Δ΂͖࢟΁޲͔͏ͨΊʹ·ͣԿΛ͢Ε͹͍͍͔͕Θ͔Β ͳ͍ ͓఻͑͢Δ͜ͱ

Slide 5

Slide 5 text

5 1. SREͷ໨ࢦ͢ͱ͜Ζ 2. SREͷ͋Δ΂͖࢟ 3. SREͷ૊৫త࣮ફ 4. ·ͱΊ ΞδΣϯμ

Slide 6

Slide 6 text

1. SREͷ໨ࢦ͢ͱ͜Ζ

Slide 7

Slide 7 text

7 ιϑτ΢ΣΞͷϥΠϑαΠΫϧͷ௕ظԽ ɾੲ͸ιϑτ΢ΣΞΛ։ൃ͠ೲ඼ͯ͠ऴΘΓͱ͍͏ϥΠϑαΠΫϧ ɾݱࡏ͸ɼιϑτ΢ΣΞΛαʔϏεͱͯ͠ఏڙ͞ΕɼαϒεΫϦϓ γϣϯํࣜʹΑΓྉۚΛࢧ෷͏ܗଶ ɾιϑτ΢ΣΞͷαʔϏεఏڙظ͕ؒ10೥Ҏ্ͱ͍͏ͷ΋౰ͨΓલʹ ɾ͕ͨͬͯ͠ɼαʔϏεఏڙ͠ଓ͚ΔͨΊͷӡ༻͕ॏཁʹͳ͖ͬͯͯ ͍Δ

Slide 8

Slide 8 text

8 ࠓޙ΋ӡ༻ͷॏཁੑ͸ߴ·͍ͬͯ͘ ίϯϐϡʔλʔ γεςϜ Ϣʔβʔ ։ൃӡ༻ऀ ӡ༻ ӡ༻ • ίϯϐϡʔλʔγεςϜͱਓؒͷؔ܎ੑ͸Ϣʔβʔͱ։ൃӡ༻ऀʹ෼͚ΒΕΔ • ࠷ۙ͸ػցֶशͳͲʹΑΓɼϢʔβʔ͕ίϯϐϡʔλʔʹσʔλΛೖྗͯ͠ҭ ͯΔ(ӡ༻͢Δ)ͱ͍͏࣌୅ʹͳ͖ͬͯͨ • Ϣʔβʔͱ։ൃӡ༻ऀͷ۠ผ͕ঃʑʹᐆດʹͳΓɼҰൠͷਓʑ͕ӡ༻ʹؔΘΔ Α͏ʹͳ͍ͬͯ͘ͷͰ͸ͳ͍͔

Slide 9

Slide 9 text

9 ίϯϐϡʔλʔͷӡ༻Λָ͘͢͠Δ ɾίϯϐϡʔλʔ͕ݡ͘ͳͬͯ΋ਓ͕͍ؒͯॳΊͯҙຯ͕͋ΔͨΊɼਓ ؒΛൈ͖ʹͯ͠ίϯϐϡʔλ͕׬શࣗ཯͢Δͱ͍͏͜ͱ͸ͳ͍ ɾਓؒ΋ίϯϐϡʔλʔ΋ޓ͍ʹ׬શͰ͸ͳ͍ͨΊɼӡ༻͸࢒Γଓ͚Δ ɾӡ༻͕࢒Γଓ͚ΔͷͰ͋Ε͹ɼָ͘͠ӡ༻͢Δ͜ͱ͕ॏཁʹͳΔ ɾӡ༻͔Βҭ੒ɼҭ੒͔Βର࿩ͱܗΛม͑ͯਐԽ͍ͤͯ͘͞ ӡ༻ ҭ੒ ର࿩

Slide 10

Slide 10 text

10 ϋʔυϧ: γεςϜมߋͷڪා ɾΠϯλʔωοτ΍΢ΣϒͷීٴͱൃలʹΑΓɼγεςϜ͕େن໛Խ ͔ͭෳࡶԽ͍ͯ͠Δ ɾਓ͕͍ؒͭͬͯ͘Δ͸ͣͷγεςϜͰ΋มߋʹΑΓԿ͕ى͖Δ͔Θ ͔Βͳ͘ͳ͖͍ͬͯͯΔ ɾݱ৔ͷΤϯδχΞ͸ɼγεςϜͷมߋ͕ා͘ͳ͍ͬͯΔ ɾා͘ͳͬͨ݁Ռɼ௚ۙͷ৴པੑΛߴΊΔͨΊʹγεςϜΛมߋ͠ͳ ͘ͳΔ ɾมߋ͠ͳ͔ͬͨ݁ՌɼΤϯδχΞ͕ӡ༻ͰΧόʔ͢ΔΑ͏ʹͳΓɼ ݁Ռతʹӡ༻ෛ୲͕૿େ

Slide 11

Slide 11 text

11 ࣦഊΛڐ༰͢Δͱ͍͏ٯసͷൃ૝ ɾڪාͷཧ༝͸৴པੑͷ௿Լ ɾ͔͠͠ɼ100%ͷ৴པੑΛୡ੒͢Δͱ͍͏ͷ͸ݱ࣮తͰ͸ͳ͍ ɾͦ͜ͰɼࣦഊΛڐ༰͢ΔલఏͰӡ༻ΛઃܭΛ͢Δ ɾ৴པੑΛ޲্ͤ͞Δ͚ͩͰͳ͘ɼͱ͖ʹ͸͋͑ͯ௿Լͤͯ͞มߋ଎ ౓ΛߴΊΔ ɾ৴པੑΛ࣠ʹγεςϜӡ༻ΛମܥԽͨ͠Site Reliability Engineering ͷొ৔

Slide 12

Slide 12 text

12 SREͱ͸ͳʹ͔ ɾSRE͸ɼ৴པੑΛ޲্ͤ͞Δ͚ͩͰͳ͘ɼ௿Լͤ͞Δ͜ͱ΋ؚΉ ɾ͔͠͠ɼ௿Լͤ͞Δͱ͍ͬͯ΋ఔ౓͕ෆ໌Ͱ͋Ε͹ɼ৴པੑͷ໨ඪ ஋ΛԼճΔՄೳੑ͕͋Δ ɾͦ͜Ͱɼ্Լͤ͞Δ৴པੑͷఔ౓Λݟੵ΋Γɼ৴པੑΛ੍ޚ͢Δ͜ ͱͰมߋ଎౓Λ࠷େԽ͢Δ͜ͱΛ໨తͱ͢Δ ɾSREͱ͸ʮαΠτ৴པੑΛ੍ޚ͢ΔͨΊͷ޻ֶʯ ɾ৴པੑͱ͍͏कΓΛ੍ޚ͢Δ͜ͱͰɼ҆৺ͯ͠มߋ଎౓ͱ͍͏߈Ί ʹసͣΔ͜ͱ͕Ͱ͖Δ

Slide 13

Slide 13 text

2. SREͷ͋Δ΂͖࢟

Slide 14

Slide 14 text

14 ৴པੑͷఆٛͱܭଌ ɾ“γεςϜ͕ٻΊΒΕΔػೳΛɼఆΊΒΕͨ৚݅ͷԼͰɼఆΊΒΕͨ ظؒʹΘͨΓɼো֐Λى͜͢͜ͱͳ࣮͘ߦ͢Δ֬཰” ɾॻ੶ʮPractical Reliability EngineeringʯΑΓ ɾ৴པੑΛఆྔతʹSLO(αʔϏεϨϕϧ໨ඪ)ͱͯ͠ઃఆ͢Δ ɾSLOྫ) ϦΫΤετͷԠ౴࣌ؒͷ95%ile͕300msҎԼͷঢ়ଶ͕࢛൒ ظதͷ99.8%ͷظؒҡ࣋͞Ε͍ͯΔ͜ͱ ɾSLOΛա৒ୡ੒ͤͣʹϐολϦୡ੒͢Δ͜ͱΛ໨ࢦ͢

Slide 15

Slide 15 text

15 ৴པੑͷ֊૚ ɾ৴པੑͷ֊૚͸ɼ৴པੑΛ੍ޚ͢ΔͨΊͷج൫ ɾϞχλϦϯά͕ͳ͚Ε͹ɼ৴པੑͷܭଌ͕ͦ΋ͦ ΋Ͱ͖ͳ͍ Betsy Beyer et. al.ฤ, "SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά", ΦϥΠϦʔδϟύϯ, p108 ਤIII -1αʔϏεͷ৴པੑͷ֊૚ ɾΠϯγσϯτରԠମ੍͕ͳ͚Ε͹ɼ৴པੑ ͕ଛͳΘΕͯ΋ճ෮Ͱ͖ͳ͍ ɾϙετϞʔςϜ͕ͳ͚Ε͹ಉ͡Πϯγσϯ τΛ܁Γฦ͠ɼ৴པੑ͕ଛͳΘΕ΍͘͢ͳ Δ ɾ…

Slide 16

Slide 16 text

16 มߋ଎౓ͷ޲্ ɾSRE͸ɺιϑτ΢ΣΞΤϯδχΞͷνʔϜʹΑΓैདྷͷγεςϜ؅ཧ Λ࠶ߏ੒ͨ͠΋ͷͱݴΘΕΔ͜ͱ͕͋Δ ɾιϑτ΢ΣΞΤϯδχΞϦϯάʹΑΓɼࣗಈԽΛ͢͢Ίɼมߋ଎౓Λ ߴΊ͍ͯ͘ ɾਓؒΑΓιϑτ΢ΣΞͷ΄͏͕ಈ࡞͕଎͍ͨΊɼো֐͔Βͷ෮چ࣌ؒ ΋ιϑτ΢ΣΞͰճ෮͞Εͨ΄͏͕଎͍ (ྫ)ࣗಈϑΣΠϧΦʔόʔ ɾ݁Ռతʹɼιϑτ΢ΣΞΤϯδχΞϦϯάʹΑΓɼ৴པੑΛߴΊͭ ͭɼมߋ଎౓΋ߴ·͍ͬͯ͘

Slide 17

Slide 17 text

17 ࢀߟจݙ ɾBetsy Beyer, et.al, “SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά ――Googleͷ ৴པੑΛࢧ͑ΔΤϯδχΞϦϯάνʔϜ”, ΦϥΠϦʔɾδϟύϯ, 2017. ɾStephen Thorne, et.al, “The Site Reliability Workbook”, O'Reilly Media, Inc., 2018. ɾDavid N. Blank-Edelman, “Seeking SRE: Conversations About Running Production Systems at Scale”, O'Reilly Media, Inc., 2018. ɾCraig Sebenik, “SRE at a Startup: Lessons From LinkedIn”, USENIX SREcon16. ɾͦͷଞɼSREcon, SRE LoungeͷൃදࢿྉͳͲ SREͷ͋Δ΂͖࢟ʹ͍ͭͯٞ࿦͍ͯ͠Δจݙ͸ଟ਺͋Δ

Slide 18

Slide 18 text

3. SREͷ૊৫త࣮ફ

Slide 19

Slide 19 text

19 ޻ֶ: ٕܳͱ૊৫ ٕܳ(Art) ޻ֶ(Engineering) ૊৫(Organization) ɾशಘٕͨ͠Λ൸࿐͢Δ ɾίʔυ͕͔͚Δ ɾΦϖϨʔγϣϯ͕Ͱ͖Δ ɾҰਓͰ͸Ͱ͖ͳ͍͜ͱΛ΍Δ ɾٕज़Λ࣮༻Խ͢Δ

Slide 20

Slide 20 text

20 SREͷ૊৫త࣮ફʹඞཁͳཁૉ ɾSREͷΧϧνϟʔͷਁಁ ɾSREͷεΩϧηοτΛ΋ͭΤϯδχΞ/Ϛωʔδϟʔͷ࠾༻΍ҭ੒ ɾSREͷ۩ମతͳ࣮ફ಺༰Λཧղ͍ͯ͠Δ͜ͱ ɾSREΛϓϩδΣΫτͱͯ͠ਐΊΒΕΔ͜ͱ ࠓ೔͸͜͜ʹϑΥʔΧε

Slide 21

Slide 21 text

21 SRE͸࣮ફͷ։͕࢝೉͍͠ ɾച্΍རӹʹ௚઀ؔΘΔ࢓ࣄ͸ࣗવͱਐΈ΍͍͢ ɾSRE͸ച্ͱ௚઀ؔ࿈͠ͳ͍͜ͱ͕ಛ௃ ɾ௚ۙͷച্΍ػೳ։ൃΛ༏ઌͯ͠ɼৗ࣌ൃੜ͢ΔγεςϜߏஙґཔ λεΫͱӡ༻λεΫͰ೔ৗ͕ຒ΋Ε͕ͪʹͳΔ ɾ෺ࣄΛલʹਐΊΔͨΊʹɼݱ৔ͷΤϯδχΞ͕໰୊ҙࣝΛ΋ͬͯɼ ௕ظత͔ͭϘτϜΞοϓʹղܾʹ޲͔͏ඞཁ͕͋Δ

Slide 22

Slide 22 text

22 ࠷ॳͷΞΫγϣϯ https://blog.shibayu36.org/entry/2019/03/04/180000

Slide 23

Slide 23 text

23 େ͖ͳ෺ࣄΛલʹਐΊΔϑϨʔϜ ໰୊ҙࣝͷ੔ཧ ՝୊ͷ͋ͿΓग़͠ ՝୊ͷ༏ઌ౓ઃఆ ϓϩδΣΫτԽ ϓϩδΣΫτਪਐ ͍͖ͳΓ͜ͷ͋ͨΓͷ ۩ମతΠγϡʔ͔Β ࢝Ί͕ͪ େ͖ͳ෺ࣄΛલʹਐΊΔͨΊʹ ઓུతʹΞΫγϣϯΛ࣮ߦ͢Δ

Slide 24

Slide 24 text

24 ࣗ෼ͨͪͷ໰୊ҙࣝΛ੔ཧ͢Δ ɾঢ়ଶ΍ղܾࡦ͸໰୊Ͱ͸ͳ͍ ɾ✘ SREνʔϜ͕͍ͳ͍͜ͱ͕໰୊Ͱ͋Δ ɾ✘ SRE͕ྲྀߦ͍ͯ͠ΔͷͰSREʹ໊લΛม͑Δ ɾҎԼͷΑ͏ʹSREʹ௚઀ؔΘΔΑ͏ʹ໰୊ҙࣝΛ੔ཧ͢Δ ɾ○ αʔϏεͷ৴པੑ͕ϓϩμΫτΦʔφʔͷظ଴஋ΛԼճ͍ͬͯΔ ɾ○ γεςϜͷมߋ଎౓͕νʔϜͷظ଴஋ΛԼճ͍ͬͯΔ ɾ○ Πϯϑϥඅ༻͕ࣄۀརӹΛѹഭ͍ͯ͠Δ

Slide 25

Slide 25 text

25 ՝୊ͷ͋ͿΓग़͠ ௿৴པੑ ௿มߋ଎౓ ߴΠϯϑϥඅ༻ ɾ͜Ε·Ͱͷো֐ه࿥Λચ͍ग़͠ɼಉҰݪҼͷෳ਺ͷো֐΍Өڹͷେ͖ ͔ͬͨো֐Λચ͍ग़͢ ɾ௚ۙͷ਺ϲ݄ͷλεΫͷத͔Βมߋ଎౓ͷ஗͍΋ͷΛચ͍ग़͠ ɾΠϯϑϥඅ༻ͷ಺༁Λ෼ੳ͠ɼ಺༁ͷେ͖͍΋ͷʹ͍ͭͯɼඅ༻࡟ݮ ͷͨΊͷղܾࡦΛߟ͑Δ ɾ࣍ͷ3ͭͷSREͳΒͰ͸ͷ؍఺͔Β՝୊Λ͋ͿΓग़͢

Slide 26

Slide 26 text

26 ϓϩδΣΫτԽ ɾେ͖ͳ՝୊΍ଟ਺ͷ՝୊Λݸਓͷۭ͍ͨ࣌ؒͰ՝୊ΛਐΊΔʹ͸ݶ ք͕͋Δ ɾ͋ͿΓग़ͨ͠՝୊Λ૊৫Ͱʮ΍Δ΋ͷʯͱ͢Δ ɾνʔϜ໨ඪ΍ݸਓ໨ඪΛઃఆ͢Δ૊৫จԽ͕͋Ε͹໨ඪʹ૊ΈࠐΉ ɾਓ਺΍طଘͷλεΫྔ͕ωοΫͳΒɼαʔϏε։ൃͱಉ͡Α͏ʹε λοϑΛ࠾༻ͨ͠ΓɼνʔϜؒͰௐ੔ͯ͠طଘͷλεΫΛফ͢ ɾ͔͠͠ɼ໨ඪΛཱ͔ͯͨΒͱ͍࣮ͬͯࡍʹୡ੒Ͱ͖Δͱ͸ݶΒͳ͍

Slide 27

Slide 27 text

27 ϓϩδΣΫτਪਐ ɾ໨ඪͷୡ੒͕೉͍͠ཧ༝ ɾݟੵ΋Γ͕؁͘ແཧͳ໨ඪʹͳ͍ͬͯΔ ɾλεΫͷ෼ղ΍ॱংཱͯͷࣦഊ ɾׂΓࠐΈଟ਺ͰϝΠϯλεΫ͕ਐ·ͳ͍ ɾϓϩδΣΫτϚωδϝϯτ͸ઐ໳ٕज़Ͱ͋ ΓɼֶशՄೳͳεΩϧ ɾΞδϟΠϧ։ൃͳͲͷطଘͷϓϥΫςΟε ΛԠ༻͢Δ

Slide 28

Slide 28 text

28 ϓϩδΣΫτਪਐͷϘτϧωοΫ ɾෳ਺ͷνʔϜΛ·͕ͨͬͨίϛϡχέʔγϣϯ଎౓ > ୯ҰνʔϜ಺ ͷίϛϡχέʔγϣϯ଎౓ ɾίϛϡχέʔγϣϯ଎౓ΛߴΊΔͨΊʹɼνʔϜ಺ʹίϛϡχέʔ γϣϯ͕ด͡ΔΑ͏ʹ૊৫ߏ଄Λมߋ͢Δ ͸΍͍ ͓͍ͦ ͸΍͍

Slide 29

Slide 29 text

29 ։ൃɾӡ༻૊৫ͷߏ଄ͷମܥ https://slide.meguro.ryuzee.com/slides/75

Slide 30

Slide 30 text

30 ૊৫ߏ଄ͷมߋྫ https://slide.meguro.ryuzee.com/slides/75 ɾࠨਤ) Πϯϑϥ෦໳ͱ։ൃ෦໳ͷίϛϡχέʔγϣϯଟ਺ ɾӈਤ) ίϛϡχέʔγϣϯΛ୯ҰνʔϜʹ׬݁ => ηϧϑαʔϏεԽ

Slide 31

Slide 31 text

31 ηϧϑαʔϏεԽ ɾαʔϏε։ൃνʔϜ͕ࣗ਎ͰαʔϏεͷӡ༻Λ͢Δ ɾSREνʔϜ͸ϓϥοτϑΥʔϜΛ։ൃ͠ɼαʔϏενʔϜ΁ఏڙ͢Δ ɾʮΦʔφʔγοϓʯʮࣗݾ૊৫ԽʯͳͲ͕ؔ࿈Ωʔϫʔυ ɾࠃ֎Ͱ͸Amazon, Netﬂix, SpotifyͳͲ͕࣮ફ [1] ɾࠃ಺Ͱ͸ϝϧΧϦ[1]ɼΫοΫύου[2]ɼ͸ͯͳ[3]ͳͲ(ܟশུ)͕࣮ફ [1]: ϚΠΫϩαʔϏενʔϜฤ੒ͷϕετϓϥΫςΟεͱϝϧΧϦͰͷߏ૝, https://tech.mercari.com/entry/2018/12/ 01/200159 [2]: ΫοΫύουSRE͕ޠΔάϩʔόϧαʔϏε΁ͷ௅ઓ, https://logmi.jp/tech/articles/321120#s6 [3]: ಥܸʂྡͷDevOps ʲ͸ͯͳฤʳ, https://dev.classmethod.jp/devops/dev-ops-hatena/

Slide 32

Slide 32 text

32 ηϧϑαʔϏεԽͷઌʹ ɾ෺ࣄΛલʹਐΊΔϑϨʔϜΛνʔϜ୯ҐͰ ߴ଎ʹ࣮ߦͰ͖Δ ɾػೳ։ൃͱӡ༻ͷόϥϯεΛνʔϜ୯Ґ ͰղܾͰ͖Δ ɾSRE໨ඪΛࣄۀ։ൃ໨ඪͱಉ͡νʔϜ໨ ඪʹઃఆͰ͖ΔͳͲ ໰୊ҙࣝͷ੔ཧ ՝୊ͷ͋ͿΓग़͠ ՝୊ͷ༏ઌ౓ઃఆ ϓϩδΣΫτԽ ϓϩδΣΫτਪਐ

Slide 33

Slide 33 text

4. ·ͱΊ

Slide 34

Slide 34 text

34 ·ͱΊ ɾӡ༻Λָ͘͢͠Δະདྷ΁ͷେ͖ͳϋʔυϧͱͯ͠มߋͷڪා͕͋Δ ɾSREʹΑΓɼ৴པੑΛ੍ޚ͠ɼมߋͷڪාΛࠀ෰͠ɼมߋ଎౓Λ޲ ্ͤ͞Δ ɾSREͷ࣮ફ͍ͯ͠Δঢ়ଶͱ͸ɼSLOΛઃఆ => ৴པੑͷ֊૚ߏங => ιϑτ΢ΣΞΤϯδχΞϦϯάʹΑΓมߋ଎౓޲্ ɾSREͷ૊৫తʹ࣮ફ͢Δʹ͸ɼෳ਺ͷਓ͕ू·ͬͨ૊৫ͷதͰɼ෺ ࣄΛલʹਐΊΔϑϨʔϜϫʔΫΛ਎ʹ͚ͭΔ ɾ૊৫ߏ଄ͱͯ͠ɼࣗݾ૊৫Խ͞ΕͨηϧϑαʔϏεΛ໨ࢦ͢