SREの組織的実践 / Organized SRE

エンジニリング組織の作り方 @ Sansan Innovation Lab
@yuuk1t / id:y_uuki
https://sansan.connpass.com/event/125822/

Transcript

1. ͘͞ΒΠϯλʔωοτ
   גࣜձࣾ
   (C) Copyright 1996-2019 SAKURA Internet Inc
   ͘͞ΒΠϯλʔωοτ
   ݚڀॴ
   SREͷ૊৫త࣮ફ
   2019/05/09 ݚڀһ ௶಺ ༎थ
   ΤϯδχϦϯά૊৫ͷ࡞Γํ @ Sansan Innovation Lab
   @yuuk1t / id:y_uuki
   

   View Slide

2. 2
   ࣗݾ঺հ
   ௶಺ ༎थ / Ώ͏͏͖ https://yuuk.io/
   ܦྺ
   גࣜձࣾ͸ͯͳ
   WebΦϖϨʔγϣϯΤϯδχΞɾSRE
   ͘͞ΒΠϯλʔωοτגࣜձࣾ
   ͘͞ΒΠϯλʔωοτݚڀॴ ݚڀһ
   WebαʔϏεͷ
   ։ൃɾӡ༻
   WebɾΠϯλʔωοτ
   ج൫ٕज़ݚڀ
   5೥
   ݱࡏ
   Site Reliability Engineeringશൠ
   ಛʹ Monitoring, Data-Intensive Applications
   ઐ໳෼໺
   

   View Slide

3. https://blog.yuuk.io/entry/2019/thinking-sre
   

   View Slide

4. 4
   ίϯϐϡʔλʔγεςϜͷӡ༻ʹ͍ͭͯɼ3೥΄Ͳલʹࣗ෼
   ͕΋͍ͬͯͨ࣍ͷ2ͭͷ໰୊ҙࣝʹର͢Δճ౴Λ঺հ͢Δ
   • ӡ༻ͷ͋Δ΂͖͕࢟Θ͔Βͳ͍
   • ͋Δ΂͖࢟΁޲͔͏ͨΊʹ·ͣԿΛ͢Ε͹͍͍͔͕Θ͔Β
   ͳ͍
   ͓఻͑͢Δ͜ͱ
   

   View Slide

5. 5
   1. SREͷ໨ࢦ͢ͱ͜Ζ
   2. SREͷ͋Δ΂͖࢟
   3. SREͷ૊৫త࣮ફ
   4. ·ͱΊ
   ΞδΣϯμ
   

   View Slide

6. 1.
   SREͷ໨ࢦ͢ͱ͜Ζ
   

   View Slide

7. 7
   ιϑτ΢ΣΞͷϥΠϑαΠΫϧͷ௕ظԽ
   ɾੲ͸ιϑτ΢ΣΞΛ։ൃ͠ೲ඼ͯ͠ऴΘΓͱ͍͏ϥΠϑαΠΫϧ
   ɾݱࡏ͸ɼιϑτ΢ΣΞΛαʔϏεͱͯ͠ఏڙ͞ΕɼαϒεΫϦϓ
   γϣϯํࣜʹΑΓྉۚΛࢧ෷͏ܗଶ
   ɾιϑτ΢ΣΞͷαʔϏεఏڙظ͕ؒ10೥Ҏ্ͱ͍͏ͷ΋౰ͨΓલʹ
   ɾ͕ͨͬͯ͠ɼαʔϏεఏڙ͠ଓ͚ΔͨΊͷӡ༻͕ॏཁʹͳ͖ͬͯͯ
   ͍Δ
   

   View Slide

8. 8
   ࠓޙ΋ӡ༻ͷॏཁੑ͸ߴ·͍ͬͯ͘
   ίϯϐϡʔλʔ
   γεςϜ
   Ϣʔβʔ ։ൃӡ༻ऀ
   ӡ༻
   ӡ༻
   • ίϯϐϡʔλʔγεςϜͱਓؒͷؔ܎ੑ͸Ϣʔβʔͱ։ൃӡ༻ऀʹ෼͚ΒΕΔ
   • ࠷ۙ͸ػցֶशͳͲʹΑΓɼϢʔβʔ͕ίϯϐϡʔλʔʹσʔλΛೖྗͯ͠ҭ
   ͯΔ(ӡ༻͢Δ)ͱ͍͏࣌୅ʹͳ͖ͬͯͨ
   • Ϣʔβʔͱ։ൃӡ༻ऀͷ۠ผ͕ঃʑʹᐆດʹͳΓɼҰൠͷਓʑ͕ӡ༻ʹؔΘΔ
   Α͏ʹͳ͍ͬͯ͘ͷͰ͸ͳ͍͔
   

   View Slide

9. 9
   ίϯϐϡʔλʔͷӡ༻Λָ͘͢͠Δ
   ɾίϯϐϡʔλʔ͕ݡ͘ͳͬͯ΋ਓ͕͍ؒͯॳΊͯҙຯ͕͋ΔͨΊɼਓ
   ؒΛൈ͖ʹͯ͠ίϯϐϡʔλ͕׬શࣗ཯͢Δͱ͍͏͜ͱ͸ͳ͍
   ɾਓؒ΋ίϯϐϡʔλʔ΋ޓ͍ʹ׬શͰ͸ͳ͍ͨΊɼӡ༻͸࢒Γଓ͚Δ
   ɾӡ༻͕࢒Γଓ͚ΔͷͰ͋Ε͹ɼָ͘͠ӡ༻͢Δ͜ͱ͕ॏཁʹͳΔ
   ɾӡ༻͔Βҭ੒ɼҭ੒͔Βର࿩ͱܗΛม͑ͯਐԽ͍ͤͯ͘͞
   ӡ༻ ҭ੒ ର࿩
   

   View Slide

10. 10
    ϋʔυϧ: γεςϜมߋͷڪා
    ɾΠϯλʔωοτ΍΢ΣϒͷීٴͱൃలʹΑΓɼγεςϜ͕େن໛Խ
    ͔ͭෳࡶԽ͍ͯ͠Δ
    ɾਓ͕͍ؒͭͬͯ͘Δ͸ͣͷγεςϜͰ΋มߋʹΑΓԿ͕ى͖Δ͔Θ
    ͔Βͳ͘ͳ͖͍ͬͯͯΔ
    ɾݱ৔ͷΤϯδχΞ͸ɼγεςϜͷมߋ͕ා͘ͳ͍ͬͯΔ
    ɾා͘ͳͬͨ݁Ռɼ௚ۙͷ৴པੑΛߴΊΔͨΊʹγεςϜΛมߋ͠ͳ
    ͘ͳΔ
    ɾมߋ͠ͳ͔ͬͨ݁ՌɼΤϯδχΞ͕ӡ༻ͰΧόʔ͢ΔΑ͏ʹͳΓɼ
    ݁Ռతʹӡ༻ෛ୲͕૿େ
    

    View Slide

11. 11
    ࣦഊΛڐ༰͢Δͱ͍͏ٯసͷൃ૝
    ɾڪාͷཧ༝͸৴པੑͷ௿Լ
    ɾ͔͠͠ɼ100%ͷ৴པੑΛୡ੒͢Δͱ͍͏ͷ͸ݱ࣮తͰ͸ͳ͍
    ɾͦ͜ͰɼࣦഊΛڐ༰͢ΔલఏͰӡ༻ΛઃܭΛ͢Δ
    ɾ৴པੑΛ޲্ͤ͞Δ͚ͩͰͳ͘ɼͱ͖ʹ͸͋͑ͯ௿Լͤͯ͞มߋ଎
    ౓ΛߴΊΔ
    ɾ৴པੑΛ࣠ʹγεςϜӡ༻ΛମܥԽͨ͠Site Reliability Engineering
    ͷొ৔
    

    View Slide

12. 12
    SREͱ͸ͳʹ͔
    ɾSRE͸ɼ৴པੑΛ޲্ͤ͞Δ͚ͩͰͳ͘ɼ௿Լͤ͞Δ͜ͱ΋ؚΉ
    ɾ͔͠͠ɼ௿Լͤ͞Δͱ͍ͬͯ΋ఔ౓͕ෆ໌Ͱ͋Ε͹ɼ৴པੑͷ໨ඪ
    ஋ΛԼճΔՄೳੑ͕͋Δ
    ɾͦ͜Ͱɼ্Լͤ͞Δ৴པੑͷఔ౓Λݟੵ΋Γɼ৴པੑΛ੍ޚ͢Δ͜
    ͱͰมߋ଎౓Λ࠷େԽ͢Δ͜ͱΛ໨తͱ͢Δ
    ɾSREͱ͸ʮαΠτ৴པੑΛ੍ޚ͢ΔͨΊͷ޻ֶʯ
    ɾ৴པੑͱ͍͏कΓΛ੍ޚ͢Δ͜ͱͰɼ҆৺ͯ͠มߋ଎౓ͱ͍͏߈Ί
    ʹసͣΔ͜ͱ͕Ͱ͖Δ
    

    View Slide

13. 2.
    SREͷ͋Δ΂͖࢟
    

    View Slide

14. 14
    ৴པੑͷఆٛͱܭଌ
    ɾ“γεςϜ͕ٻΊΒΕΔػೳΛɼఆΊΒΕͨ৚݅ͷԼͰɼఆΊΒΕͨ
    ظؒʹΘͨΓɼো֐Λى͜͢͜ͱͳ࣮͘ߦ͢Δ֬཰”
    ɾॻ੶ʮPractical Reliability EngineeringʯΑΓ
    ɾ৴པੑΛఆྔతʹSLO(αʔϏεϨϕϧ໨ඪ)ͱͯ͠ઃఆ͢Δ
    ɾSLOྫ) ϦΫΤετͷԠ౴࣌ؒͷ95%ile͕300msҎԼͷঢ়ଶ͕࢛൒
    ظதͷ99.8%ͷظؒҡ࣋͞Ε͍ͯΔ͜ͱ
    ɾSLOΛա৒ୡ੒ͤͣʹϐολϦୡ੒͢Δ͜ͱΛ໨ࢦ͢
    

    View Slide

15. 15
    ৴པੑͷ֊૚
    ɾ৴པੑͷ֊૚͸ɼ৴པੑΛ੍ޚ͢ΔͨΊͷج൫
    ɾϞχλϦϯά͕ͳ͚Ε͹ɼ৴པੑͷܭଌ͕ͦ΋ͦ
    ΋Ͱ͖ͳ͍
    Betsy Beyer et. al.ฤ, "SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά", ΦϥΠϦʔδϟύϯ,
    
    p108 ਤIII -1αʔϏεͷ৴པੑͷ֊૚
    ɾΠϯγσϯτରԠମ੍͕ͳ͚Ε͹ɼ৴པੑ
    ͕ଛͳΘΕͯ΋ճ෮Ͱ͖ͳ͍
    ɾϙετϞʔςϜ͕ͳ͚Ε͹ಉ͡Πϯγσϯ
    τΛ܁Γฦ͠ɼ৴པੑ͕ଛͳΘΕ΍͘͢ͳ
    Δ
    ɾ…
    

    View Slide

16. 16
    มߋ଎౓ͷ޲্
    ɾSRE͸ɺιϑτ΢ΣΞΤϯδχΞͷνʔϜʹΑΓैདྷͷγεςϜ؅ཧ
    Λ࠶ߏ੒ͨ͠΋ͷͱݴΘΕΔ͜ͱ͕͋Δ
    ɾιϑτ΢ΣΞΤϯδχΞϦϯάʹΑΓɼࣗಈԽΛ͢͢Ίɼมߋ଎౓Λ
    ߴΊ͍ͯ͘
    ɾਓؒΑΓιϑτ΢ΣΞͷ΄͏͕ಈ࡞͕଎͍ͨΊɼো֐͔Βͷ෮چ࣌ؒ
    ΋ιϑτ΢ΣΞͰճ෮͞Εͨ΄͏͕଎͍ (ྫ)ࣗಈϑΣΠϧΦʔόʔ
    ɾ݁Ռతʹɼιϑτ΢ΣΞΤϯδχΞϦϯάʹΑΓɼ৴པੑΛߴΊͭ
    ͭɼมߋ଎౓΋ߴ·͍ͬͯ͘
    

    View Slide

17. 17
    ࢀߟจݙ
    ɾBetsy Beyer, et.al, “SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά ――Googleͷ
    ৴པੑΛࢧ͑ΔΤϯδχΞϦϯάνʔϜ”, ΦϥΠϦʔɾδϟύϯ, 2017.
    ɾStephen Thorne, et.al, “The Site Reliability Workbook”, O'Reilly Media, Inc., 2018.
    ɾDavid N. Blank-Edelman, “Seeking SRE: Conversations About Running
    Production Systems at Scale”, O'Reilly Media, Inc., 2018.
    ɾCraig Sebenik, “SRE at a Startup: Lessons From LinkedIn”, USENIX SREcon16.
    ɾͦͷଞɼSREcon, SRE LoungeͷൃදࢿྉͳͲ
    SREͷ͋Δ΂͖࢟ʹ͍ͭͯٞ࿦͍ͯ͠Δจݙ͸ଟ਺͋Δ
    

    View Slide

18. 3.
    SREͷ૊৫త࣮ફ
    

    View Slide

19. 19
    ޻ֶ: ٕܳͱ૊৫
    ٕܳ(Art)
    ޻ֶ(Engineering)
    ૊৫(Organization)
    ɾशಘٕͨ͠Λ൸࿐͢Δ
    ɾίʔυ͕͔͚Δ
    ɾΦϖϨʔγϣϯ͕Ͱ͖Δ
    ɾҰਓͰ͸Ͱ͖ͳ͍͜ͱΛ΍Δ
    ɾٕज़Λ࣮༻Խ͢Δ
    

    View Slide

20. 20
    SREͷ૊৫త࣮ફʹඞཁͳཁૉ
    ɾSREͷΧϧνϟʔͷਁಁ
    ɾSREͷεΩϧηοτΛ΋ͭΤϯδχΞ/Ϛωʔδϟʔͷ࠾༻΍ҭ੒
    ɾSREͷ۩ମతͳ࣮ફ಺༰Λཧղ͍ͯ͠Δ͜ͱ
    ɾSREΛϓϩδΣΫτͱͯ͠ਐΊΒΕΔ͜ͱ
    ࠓ೔͸͜͜ʹϑΥʔΧε
    

    View Slide

21. 21
    SRE͸࣮ફͷ։͕࢝೉͍͠
    ɾച্΍རӹʹ௚઀ؔΘΔ࢓ࣄ͸ࣗવͱਐΈ΍͍͢
    ɾSRE͸ച্ͱ௚઀ؔ࿈͠ͳ͍͜ͱ͕ಛ௃
    ɾ௚ۙͷച্΍ػೳ։ൃΛ༏ઌͯ͠ɼৗ࣌ൃੜ͢ΔγεςϜߏஙґཔ
    λεΫͱӡ༻λεΫͰ೔ৗ͕ຒ΋Ε͕ͪʹͳΔ
    ɾ෺ࣄΛલʹਐΊΔͨΊʹɼݱ৔ͷΤϯδχΞ͕໰୊ҙࣝΛ΋ͬͯɼ
    ௕ظత͔ͭϘτϜΞοϓʹղܾʹ޲͔͏ඞཁ͕͋Δ
    

    View Slide

22. 22
    ࠷ॳͷΞΫγϣϯ
    https://blog.shibayu36.org/entry/2019/03/04/180000
    

    View Slide

23. 23
    େ͖ͳ෺ࣄΛલʹਐΊΔϑϨʔϜ
    ໰୊ҙࣝͷ੔ཧ
    ՝୊ͷ͋ͿΓग़͠
    ՝୊ͷ༏ઌ౓ઃఆ
    ϓϩδΣΫτԽ
    ϓϩδΣΫτਪਐ ͍͖ͳΓ͜ͷ͋ͨΓͷ
    ۩ମతΠγϡʔ͔Β
    ࢝Ί͕ͪ
    େ͖ͳ෺ࣄΛલʹਐΊΔͨΊʹ
    ઓུతʹΞΫγϣϯΛ࣮ߦ͢Δ
    

    View Slide

24. 24
    ࣗ෼ͨͪͷ໰୊ҙࣝΛ੔ཧ͢Δ
    ɾঢ়ଶ΍ղܾࡦ͸໰୊Ͱ͸ͳ͍
    ɾ✘ SREνʔϜ͕͍ͳ͍͜ͱ͕໰୊Ͱ͋Δ
    ɾ✘ SRE͕ྲྀߦ͍ͯ͠ΔͷͰSREʹ໊લΛม͑Δ
    ɾҎԼͷΑ͏ʹSREʹ௚઀ؔΘΔΑ͏ʹ໰୊ҙࣝΛ੔ཧ͢Δ
    ɾ○ αʔϏεͷ৴པੑ͕ϓϩμΫτΦʔφʔͷظ଴஋ΛԼճ͍ͬͯΔ
    ɾ○ γεςϜͷมߋ଎౓͕νʔϜͷظ଴஋ΛԼճ͍ͬͯΔ
    ɾ○ Πϯϑϥඅ༻͕ࣄۀརӹΛѹഭ͍ͯ͠Δ
    

    View Slide

25. 25
    ՝୊ͷ͋ͿΓग़͠
    ௿৴པੑ
    ௿มߋ଎౓
    ߴΠϯϑϥඅ༻
    ɾ͜Ε·Ͱͷো֐ه࿥Λચ͍ग़͠ɼಉҰݪҼͷෳ਺ͷো֐΍Өڹͷେ͖
    ͔ͬͨো֐Λચ͍ग़͢
    ɾ௚ۙͷ਺ϲ݄ͷλεΫͷத͔Βมߋ଎౓ͷ஗͍΋ͷΛચ͍ग़͠
    ɾΠϯϑϥඅ༻ͷ಺༁Λ෼ੳ͠ɼ಺༁ͷେ͖͍΋ͷʹ͍ͭͯɼඅ༻࡟ݮ
    ͷͨΊͷղܾࡦΛߟ͑Δ
    ɾ࣍ͷ3ͭͷSREͳΒͰ͸ͷ؍఺͔Β՝୊Λ͋ͿΓग़͢
    

    View Slide

26. 26
    ϓϩδΣΫτԽ
    ɾେ͖ͳ՝୊΍ଟ਺ͷ՝୊Λݸਓͷۭ͍ͨ࣌ؒͰ՝୊ΛਐΊΔʹ͸ݶ
    ք͕͋Δ
    ɾ͋ͿΓग़ͨ͠՝୊Λ૊৫Ͱʮ΍Δ΋ͷʯͱ͢Δ
    ɾνʔϜ໨ඪ΍ݸਓ໨ඪΛઃఆ͢Δ૊৫จԽ͕͋Ε͹໨ඪʹ૊ΈࠐΉ
    ɾਓ਺΍طଘͷλεΫྔ͕ωοΫͳΒɼαʔϏε։ൃͱಉ͡Α͏ʹε
    λοϑΛ࠾༻ͨ͠ΓɼνʔϜؒͰௐ੔ͯ͠طଘͷλεΫΛফ͢
    ɾ͔͠͠ɼ໨ඪΛཱ͔ͯͨΒͱ͍࣮ͬͯࡍʹୡ੒Ͱ͖Δͱ͸ݶΒͳ͍
    

    View Slide

27. 27
    ϓϩδΣΫτਪਐ
    ɾ໨ඪͷୡ੒͕೉͍͠ཧ༝
    ɾݟੵ΋Γ͕؁͘ແཧͳ໨ඪʹͳ͍ͬͯΔ
    ɾλεΫͷ෼ղ΍ॱংཱͯͷࣦഊ
    ɾׂΓࠐΈଟ਺ͰϝΠϯλεΫ͕ਐ·ͳ͍
    ɾϓϩδΣΫτϚωδϝϯτ͸ઐ໳ٕज़Ͱ͋
    ΓɼֶशՄೳͳεΩϧ
    ɾΞδϟΠϧ։ൃͳͲͷطଘͷϓϥΫςΟε
    ΛԠ༻͢Δ
    

    View Slide

28. 28
    ϓϩδΣΫτਪਐͷϘτϧωοΫ
    ɾෳ਺ͷνʔϜΛ·͕ͨͬͨίϛϡχέʔγϣϯ଎౓ > ୯ҰνʔϜ಺
    ͷίϛϡχέʔγϣϯ଎౓
    ɾίϛϡχέʔγϣϯ଎౓ΛߴΊΔͨΊʹɼνʔϜ಺ʹίϛϡχέʔ
    γϣϯ͕ด͡ΔΑ͏ʹ૊৫ߏ଄Λมߋ͢Δ
    ͸΍͍ ͓͍ͦ ͸΍͍
    

    View Slide

29. 29
    ։ൃɾӡ༻૊৫ͷߏ଄ͷମܥ
    https://slide.meguro.ryuzee.com/slides/75
    

    View Slide

30. 30
    ૊৫ߏ଄ͷมߋྫ
    https://slide.meguro.ryuzee.com/slides/75
    ɾࠨਤ) Πϯϑϥ෦໳ͱ։ൃ෦໳ͷίϛϡχέʔγϣϯଟ਺
    ɾӈਤ) ίϛϡχέʔγϣϯΛ୯ҰνʔϜʹ׬݁ => ηϧϑαʔϏεԽ
    

    View Slide

31. 31
    ηϧϑαʔϏεԽ
    ɾαʔϏε։ൃνʔϜ͕ࣗ਎ͰαʔϏεͷӡ༻Λ͢Δ
    ɾSREνʔϜ͸ϓϥοτϑΥʔϜΛ։ൃ͠ɼαʔϏενʔϜ΁ఏڙ͢Δ
    ɾʮΦʔφʔγοϓʯʮࣗݾ૊৫ԽʯͳͲ͕ؔ࿈Ωʔϫʔυ
    ɾࠃ֎Ͱ͸Amazon, Netflix, SpotifyͳͲ͕࣮ફ [1]
    ɾࠃ಺Ͱ͸ϝϧΧϦ[1]ɼΫοΫύου[2]ɼ͸ͯͳ[3]ͳͲ(ܟশུ)͕࣮ફ
    [1]: ϚΠΫϩαʔϏενʔϜฤ੒ͷϕετϓϥΫςΟεͱϝϧΧϦͰͷߏ૝, https://tech.mercari.com/entry/2018/12/
    01/200159
    [2]: ΫοΫύουSRE͕ޠΔάϩʔόϧαʔϏε΁ͷ௅ઓ, https://logmi.jp/tech/articles/321120#s6
    [3]: ಥܸʂྡͷDevOps ʲ͸ͯͳฤʳ, https://dev.classmethod.jp/devops/dev-ops-hatena/
    

    View Slide

32. 32
    ηϧϑαʔϏεԽͷઌʹ
    ɾ෺ࣄΛલʹਐΊΔϑϨʔϜΛνʔϜ୯ҐͰ
    ߴ଎ʹ࣮ߦͰ͖Δ
    ɾػೳ։ൃͱӡ༻ͷόϥϯεΛνʔϜ୯Ґ
    ͰղܾͰ͖Δ
    ɾSRE໨ඪΛࣄۀ։ൃ໨ඪͱಉ͡νʔϜ໨
    ඪʹઃఆͰ͖ΔͳͲ
    ໰୊ҙࣝͷ੔ཧ
    ՝୊ͷ͋ͿΓग़͠
    ՝୊ͷ༏ઌ౓ઃఆ
    ϓϩδΣΫτԽ
    ϓϩδΣΫτਪਐ
    

    View Slide

33. 4.
    ·ͱΊ
    

    View Slide

34. 34
    ·ͱΊ
    ɾӡ༻Λָ͘͢͠Δະདྷ΁ͷେ͖ͳϋʔυϧͱͯ͠มߋͷڪා͕͋Δ
    ɾSREʹΑΓɼ৴པੑΛ੍ޚ͠ɼมߋͷڪාΛࠀ෰͠ɼมߋ଎౓Λ޲
    ্ͤ͞Δ
    ɾSREͷ࣮ફ͍ͯ͠Δঢ়ଶͱ͸ɼSLOΛઃఆ => ৴པੑͷ֊૚ߏங =>
    ιϑτ΢ΣΞΤϯδχΞϦϯάʹΑΓมߋ଎౓޲্
    ɾSREͷ૊৫తʹ࣮ફ͢Δʹ͸ɼෳ਺ͷਓ͕ू·ͬͨ૊৫ͷதͰɼ෺
    ࣄΛલʹਐΊΔϑϨʔϜϫʔΫΛ਎ʹ͚ͭΔ
    ɾ૊৫ߏ଄ͱͯ͠ɼࣗݾ૊৫Խ͞ΕͨηϧϑαʔϏεΛ໨ࢦ͢
    

    View Slide