よりUXに近いSLI・SLOの運用による可用性の再設計

SRE NEXT 2022
https://sre-next.dev/2022/schedule/#jp47

Transcript

1. ΑΓUXʹ͍ۙSLIɾSLOͷӡ
   ༻ʹΑΔՄ༻ੑͷ࠶ઃܭ
   Ӭ໺ Ұഅ
   

   View Slide

2. ࣗݾ঺հ
   ● 2019/4ʙ λοϓϧʹiOSΤϯδχΞͱͯ͠
   δϣΠϯ
   
   
   ● 2021೥ΑΓSREνʔϜʹϞόΠϧΤϯδχ
   Ξͱͯ͠ॴଐ
   
   
   ● ݱࡏ͸αʔϏεͷ৴པੑΛߴΊΔϛογϣ
   ϯʹऔΓ૊ΜͰ͍·͢
   Ӭ໺ Ұഅ
   

   View Slide

3. λοϓϧͷαʔϏεʹ͍ͭͯ
   https://tapple.me/
   
   
   2014೥5݄ʹϦϦʔεͨ͠ࠃ಺࠷େن໛ͷϚονϯάΞϓϦͰ͢ɻ1,000छҎ্ͷσʔτϓϥϯ͔Β૬ख
   Λืूͯ͠ϚονϯάͰ͖Δʮ͓Ͱ͔͚ػೳʯΛఏڙ͠ɺ೥ऩɾ৬ۀͳͲͷ৚͚݅ͩͰͳ͘ɺझຯΛ͖ͬ
   ͔͚ʹࣗ෼Β͘͠աͤ͝Δύʔτφʔͱग़ձ͑Δͷ͕ಛ௃Ͱɺ20୅ͷஉঁΛத৺ʹࢧ࣋Λड͚͍ͯ·͢ɻ
   

   View Slide

4. ● SREνʔϜʹϞ
   バ
   ΠϧΤϯ
   ジ
   χΞ(iOSɺAndroid֤1ਓ)
   が
   ઐଐ
   で
   ॴଐ͢Δܗ
   
   
   ● ΑΓϢʔβʔʹ͍ۙ෦෼ͰͷܭଌɺBFFͷਪਐͱ͍ͬͨόοΫΤϯυSRE͚ͩͰ͸खΛ৳͹ͮ͠Β
   ͍෦෼ʹ΋ΞδϦςΟߴ͘औΓ૊ΊΔΑ͏ʹ͢Δ໨త
   
   
   ● Πϯϑϥ΍όοΫΤϯυͷઐ໳ੑͷ֫ಘ΋
   SREνʔϜͷMobileΤϯδχΞʹ͍ͭͯ
   

   View Slide

5. ࠓճͷൃද಺༰
   SLI / SLOΛϞόΠϧΞϓϦέʔγϣϯଆͰܭଌ͞ΕΔࢦඪΛ΋ͱʹ࠶ઃܭ͠ɺΑΓUXʹ͍ۙࢦඪΛ໨ࢦ
   ͢औΓ૊Έͷ঺հ
   
   
   ● ৴པੑΛճ෮ͤ͞ΔͨΊͷΞΫγϣϯ΍ɺΠϯγσϯτ΁ͷରॲΛఆྔతʹߦ͑ΔΑ͏ʹ
   
   
   ● ৴པੑΛఆྔతʹ೺ѲͰ͖Δ͜ͱͰͷ҆৺ײ
   
   
   ● ࣄۀ൑அ΍༏ઌ౓ͷίϛϡχέʔγϣϯʹ͓͚Δڞ௨ݴޠʹ
   

   View Slide

6. ैདྷͷSLI/SLOͷӡ༻
   

   View Slide

7. όοΫΤϯυ
   
   
   ● APIαʔόʔ, BFFαʔόʔͦΕͧΕͷϨΠςϯγΛSLIͱͯ͠ར༻
   
   
   ● ϞχλϦϯάɺΞϥʔτڞʹDataDogͰ؅ཧ
   
   
   ● ҧ൓࣌΍ɺΤϥʔόδΣοτΛஶ͘͠ফඅ͢Δ৔߹͸ଈ࣌ো֐ରԠ
   ैདྷͷSLI/SLOͷӡ༻ — όοΫΤϯυ —
   https://developers.cyberagent.co.jp/blog/archives/20354/
   

   View Slide

8. ΫϥΠΞϯτ
   
   
   ΫϥΠΞϯτଆͰ͸ಠࣗͷSLIΛӡ༻
   
   
   ● Ϋϥογϡ཰ɺύʔεΤϥʔ਺ɺϑϨʔϜϨʔτ
   ैདྷͷSLI/SLOͷӡ༻ — ΫϥΠΞϯτ —
   

   View Slide

9. ● Firebase Performance, CrashlyticsͷϩάΛར༻͠ɺϕϩγςΟΞϥʔτΛར༻
   
   
   ● Data PortalͰμογϡϘʔυ؅ཧ
   
   
   ● ҧ൓࣌͸Өڹͷൣғɺػೳͷॏཁ౓ʹԠͯ͡ΞΫγϣϯΛܾఆ
   

   ैདྷͷSLI/SLOͷӡ༻ — ΫϥΠΞϯτ —
   ΠϯύΫτ ΞΫγϣϯ
   High
   ࢪࡦΛͱΊͯͰ΋νʔϜશମͰՄೳͳݶΓϦιʔεΛͭ͗ࠐΈɺ
   
   
   ϗοτϑΟοΫε΋ݕ౼
   Middle
   ୲౰ͷΤϯδχΞΛΞαΠϯ্ͨ͠Ͱɺ࣍ճϦϦʔεͰඞͣରԠɻ
   
   
   ඞཁͰ͋Ε͹ࢪࡦͷख΋ͱΊΔ
   Low
   ୲౰ͷΤϯδχΞΛΞαΠϯ্ͨ͠Ͱɺ
   
   
   ։ൃϦιʔε͕ۭ͖࣍ୈదٓରԠ͢Δ
   

   View Slide

10. SLO͕αʔϏεͷՄ༻ੑͷ൑அ࣠ʹͳ͍ͬͯͳ͍
    

    

    

    1. emergencyϨϕϧͷΞϥʔτ͕ൃՐ
    
    
    2. ΤϯδχΞ͕֤ࣗӨڹൣғʢػೳɺϢʔβʔ਺ʣௐࠪ
    
    
    3. ΞΫγϣϯܾఆʢ͓஌ΒͤɺTwitterࠂ஌ɺ޿ࠂఀࢭɺϝϯςΠϯʣ
    

    

    

    1. emergencyϨϕϧͷΞϥʔτ͕ൃՐ
    
    
    2. SLI/SLOμογϡϘʔυͰӨڹൣғ֬ೝ
    
    
    3. ӨڹͷͰ͍ͯΔSLIɺΤϥʔόδΣοτͷফඅ͔Β࣍ͷΞΫγϣϯΛܾఆ
    ैདྷͷSLI/SLOͷӡ༻ — ՝୊ᶃ —
    ຊདྷͷཧ૝
    ݱࡏͷӡ༻
    

    View Slide

11. όοΫΤϯυɺΫϥΠΞϯτʹดͨ͡ࢦඪͦΕͧΕͷΈͰ͸Մ༻ੑΛอূͰ͖ͳ͍
    
    
    

    

    αʔόʔͷSLO͕ຬͨ͞Ε͍ͯͯ΋ɺ
    
    
    ● ૬खͷϝοηʔδ͕ݟΕ͍ͯͳ͍
    

    → ΫϥΠΞϯτଆͰͷύʔεΤϥʔ͕ݪҼͰϨεϙϯεΛड৴Ͱ͖ͯͳ͍
    
    
    ● ৽نొ࿥Ͱ͖ͳ͍
    

    → ֎෦SNSଆͰো֐ൃੜɺΦϯϘʔσΟϯάϑϩʔ͕ແݶϧʔϓ
    

    

    

    ● ϝοηʔδը໘Λ։͍ͨϢʔβʔͷ○%͕ϝοηʔδͷӾཡʹ੒ޭ͍ͯ͠Δঢ়ଶ͔Θ͔Δ
    
    
    ● ৽نొ࿥Λ։࢝ͨ͠Ϣʔβʔͷ○%͕৽نొ࿥ʹ੒ޭ͍ͯ͠Δ͔Θ͔Δ
    ैདྷͷSLI/SLOͷӡ༻ — ՝୊ᶄ —
    ຊདྷͷཧ૝
    ݱࡏͷӡ༻
    

    View Slide

12. ໨ࢦ͢ཧ૝ঢ়ଶ
    ● Ϣʔβʔͷظ଴͢ΔUXΛΧόʔ͢Δࢦඪઃఆ
    

    ● SLI / SLOͷҧ൓ʹΑΔϢʔβʔӨڹ೺ѲΛ༰қʹ
    

    ● ϏδωεɺΫϥΠΞϯτɺαʔόʔؒͰαʔϏεঢ়ଶͷڞ௨ݴޠΛ࣋ͭ
    

    View Slide

13. ΑΓUXʹ͍ۙSLIɾSLOͷઃܭ
    

    View Slide

14. Ϣʔβʔͷظ଴͢ΔUXΛΧόʔ͢ΔSLIઃఆ
    ϝϯόʔ͔ΒʮαʔϏεͷՄ༻ੑʯͷٵ্͍͛
    
    
    ● औΓ͍ͨࢦඪɺఆٛɺܭଌํ๏ɺѱԽͨ͠৔߹ʹଛͳΘΕΔUXΛώΞϦϯά
    
    
    ● ϑΥʔΧε͢ΔࢦඪɺίΞͳମݧΛ࢈ΜͰ͍ΔUXΛܾఆ͢Δ
    
    
    ϚΠΫϩαʔϏε౳ͷυϝΠϯϞσϧͱ߹கͤ͞Δ
    

    View Slide

15. ϓϥοτϑΥʔϜͷఏڙ͢Δࢦඪͷ࠾༻
    Apple, Google͕σΟϕϩούʔʹఏڙ͢ΔSLI
    
    
    ● ϓϥοτϑΥʔϜଆͷߟ͑Δྑ͍ΞϓϦͷఆٛ
    
    
    ● ΞϓϦετΞ΍OSϨϕϧͰӨڹ͕͋Δ͜ͱ΋
    
    
    ○ ASOʢApp Store Optimizationʣ
    
    
    ○ ༏ઌͯ͠ΞϯΠϯετʔϧΛଅ͞ΕΔ
    
    
    ○ ΞϓϦͷόοΫάϥ΢ϯυͰͷੜଘ࣌ؒ
    
    
    ● λοϓϧͰ͸ԼهͷࢦඪΛSLIͱͯ͠࠾༻
    
    
    ○ ΞϓϦαΠζ
    
    
    ○ શମΫϥογϡ཰ʢANRʣ
    
    
    ○ ىಈ࣌ؒ
    https://developers.cyberagent.co.jp/blog/archives/20354/
    

    View Slide

16. SLIҧ൓Λ࠶ݱͨ͠ΞϓϦΛ࢖ͬͨSLOܾΊ
    ● աڈͷΠϯγσϯτ࣌ͷࢦඪ΍ɺࣄલώΞϦϯά͔Βɺ͖͍͠஋ʹ͋ͨΓΛ͚ͭΔ
    

    ● σόοάػೳͰৄࡉͳ͖͍͠஋Λௐ੔ՄೳͳΞϓϦΛ༻ҙ͠ɺ৮Γͳ͕Βٞ࿦
    

    View Slide

17. ࠶ఆٛͨ͠SLI
    όοΫΤϯυʢܧଓʣ
    
    
    ● ϨΠςϯγʔ
    

    ΫϥΠΞϯτʢʹϓϥοτϑΥʔϜࢦඪʣ
    
    
    ● ىಈ࣌ؒɺશମΫϥογϡ཰ɺΞϓϦαΠζ
    

    ίΞػೳʢʹػೳ͝ͱͷՄ༻ੑʣ
    
    
    ● ϩάΠϯɺ৽نొ࿥ɺΧʔυϑϦοΫɺ
    

    ϝοηʔδɺ͓Ͱ͔͚ɺWish
    

    Մ༻ੑΛ΋ͱʹͨ͠SLIΛઃఆ
    
    
    

    ڞ௨
    
    
    Ϋϥογϡ / ANR
    
    
    ػೳ͝ͱͷΩʔϝτϦΫε
    
    
    ϝοηʔδ: ಡΈࠐΈ଎౓
    
    
    ΧʔυϑϦοΫ: ࠶ϑϦοΫՄೳʹͳΔ·Ͱͷ࣌ؒ
    

    ɾɾɾ
    

    View Slide

18. ΫϥΠΞϯτଆͰͷSLI / SLOͷܭଌ
    

    View Slide

19. ΫϥΠΞϯτଆͰͷSLI / SLOͷܭଌ
    1. ·ͣ͸Firebase (+BigQuery) ʹ׬݁ͯ͠ϛχϚϜʹՔಇ
    
    
    ● ػೳతʹे෼ͳ͜ͱΛݕূ
    
    
    2. Monitoring, DashboardΛDataDogʹҠߦʢ͜Ε͔Βʣ
    
    
    ● Ξϥʔτͷࣗ༝౓Λ্͛Δ
    
    
    ● ଞͷϩάͰར༻ͯ͠ΔαʔϏεͱ߹ΘͤΔ
    

    View Slide

20. ΫϥΠΞϯτଆͰͷϞχλϦϯάπʔϧͷݕ౼
    ͻͱ·ͣྉۚ໘Λߟྀ͠FirebaseΛར༻͠ɺʮӡ༻Λ։࢝ → ՝୊͕͋Ε͹ϒϥογϡΞοϓʯͷํ਑ʹ
    

    1. Firebase Crashlytics / Performance + BigQuery
    
    
    ● Firebaseࣗମ͸ແྉͷͨΊBigQueryͷQuery, Active StorageͷྉۚͷΈ
    
    
    ● Crashlytics, Performance͸طଘͰ΋ར༻͍ͯ͠ΔͨΊಋೖίετ͕௿͍
    
    
    ● BigQueryʹ஝ੵ͢Δ͜ͱͰൺֱతࣗ༝౓͸ߴ͘ར༻Ͱ͖Δ
    
    
    2. DataDog RUM
    
    
    ● طଘͷϞχλϦϯάπʔϧͱͯ͠DataDogΛར༻͍ͯ͠ΔͨΊπʔϧΛ౷ҰͰ͖Δ
    
    
    ● ηογϣϯ୯Ґͷ՝ۚମܥͷͨΊɺࠓճ͸ྉۚ໘Ͱஅ೦
    
    
    3. New Relic One
    
    
    ● όοΫΤϯυͰDataDogΛར༻͍ͯ͠Δ͜ͱɺྉۚ໘Λഎܠʹஅ೦
    

    View Slide

21. FirebaseͰߏ੒͢ΔϛχϚϜͳܭଌϑϩʔ
    Firebase ( Crashlytics + Performance ) ͷΈͰϛχϚϜͳϑϩʔ͸࣮ݱՄೳ
    
    
    ● BigQueryʹExportʢϦΞϧλΠϜ΋Մೳʣ
    
    
    ● BigQuery্ʹμογϡϘʔυදࣔ༻ͷதؒςʔϒϧ࡞੒
    
    
    ● Data PortalͰμογϡϘʔυදࣔ
    )LUHEDVH
    3HUIRUPDQFH
    )LUHEDVH
    &UDVKO\WLFV
    
    8VHU
    &UDVK
    .H\0HWULFV
    *RRJOH'DWD3RUWDO
    $XWR([SRUW
    %LJ4XHU\
    ,QWHUPHGLDWHWDEOH
    %LJ4XHU\
    

    View Slide

22. Ξϥʔτͷઃఆͷࣗ༝౓
    Firebase Performance
    
    
    ● ࡞੒ͨ͠ࢦඪ͝ͱʹΞϥʔτͷઃఆ͕Մೳ
    
    
    ● ͖͍͠஋Λ50ʙ95ύʔηϯλΠϧͷࢦఆ͕Մೳ
    
    
    Firebase Crashlytics
    
    
    ● ௚ۙ1࣌ؒͷηογϣϯʹର͢ΔΫϥογϡ཰
    
    
    ● ͖͍͠஋Λ0.1%ʙ1.0%ͰίϯτϩʔϧՄೳ
    

    View Slide

23. Ϋϥογϡϩάͷ෼ྨ
    https://firebase.google.com/docs/crashlytics/customize-crash-reports
    import UIKit
    
    
    import FirebaseCrashlytics
    
    
    final class LoginViewController: UIViewController {
    
    
    
    
    // ը໘Λදࣔ͢ΔλΠϛϯάͰCrashlyticsʹΧελϜΩʔΛઃఆ
    
    
    override func viewWillAppear(_ animated: Bool) {
    
    
    super.viewWillAppear(animated)
    
    
    Crashlytics.crashlytics().setCustomValue("login", forKey: "domain")
    
    
    }
    
    
    iOSΞϓϦͰͷϩάΠϯը໘ʹର͢Δ࣮૷ྫ
    ֤ػೳͷը໘දࣔ࣌ʹCrashlyticsʹΧελϜΩʔΛઃఆ͢Δ
    
    
    ● ΧελϜΩʔΛ΋ͱʹBigQuery (Data Portal) ্ͰػೳυϝΠϯ͝ͱͷΫϥογϡΛूܭ
    
    
    ● ػೳ͝ͱͷΞϥʔτ͸ઃఆͰ͖ͳ͍ͨΊɺVelocity Alert → μογϡϘʔυͰػೳӨڹ֬ೝ
    

    View Slide

24. ࠓޙʣϞχλϦϯάΛDataDogʹ౷Ұ
    ҎԼͷ؍఺͔ΒDataDogʹू໿͍ͯ͘͠
    
    
    ● ϞχλϦϯάͷू໿
    
    
    ● Ϋϥογϡͷػೳ͝ͱͷΞϥʔτ
    )LUHEDVH
    3HUIRUPDQFH
    )LUHEDVH
    &UDVKO\WLFV
    $XWR([SRUW
    %LJ4XHU\ 4XHU\ SXVK
    IRUZRUGWRGDWDGRJ
    &ORXG)XQFWLRQV
    'DWD'RJ
    &ORXG
    3XE6XE
    WULJJHU
    

    View Slide

25. ӡ༻ʹ͍ͭͯ
    

    View Slide

26. ӡ༻ͷมԽ
    Πϯγσϯτൃੜ࣌
    
    
    ● PMɺόοΫΤϯυɺΫϥΠΞϯτϝϯόʔશһ͕ϢʔβʔӨڹΛ൑அ͢ΔҰ࣌৘ใͱͯ͠׆༻
    
    
    ○ ΤϯδχΞͷௐࠪΛ଴ͨͣͯ͠ωΫετΞΫγϣϯΛಈ͖ग़ͤΔΑ͏ʹ
    

    ྫʣશϢʔβʔ΁ͷ͓͠ΒͤɺTwitterࠂ஌ɺ޿ࠂͷఀࢭ
    
    
    ● ِཅੑͷ௿Լ
    
    
    ○ ಛఆͷ৚݅ԼʢΩϟϯϖʔϯதͷΈ౳ʣʹൃੜ͢ΔύʔεΤϥʔΛ༏ઌ౓ߴ͘ΞϥʔτΛ͋͛
    ΒΕΔΑ͏ʹ
    

    View Slide

27. ӡ༻ͷมԽ
    ϦϦʔεޙͷϝτϦΫε֬ೝ࣌ʹόοΫΤϯυɺΫϥΠΞϯτͷڞ௨ݴޠʹ
    
    
    

    ● όοΫΤϯυɺΫϥΠΞϯτͦΕͧΕ͕׬શʹ෼͔Εͯ֬ೝ
    
    
    ● ֬ೝՄೳͳϝτϦΫεɺμογϡϘʔυΛҰ௨Γ໢ཏతʹ֬ೝ
    
    
    ● όοΫΤϯυɺΫϥΠΞϯτڞ௨ͷSLI / SLO μογϡϘʔυΛ֬ೝ
    
    
    ○ ௐࠪͷ৔߹͸SLIΛڞ௨ݴޠͱͯ͠ίϛϡχέʔγϣϯ
    
    
    ● ҧ൓͕͋ͬͨ৔߹ʹؔ࿈ͷϝτϦΫεɺμογϡϘʔυΛ֬ೝ
    
    
    ○ SLI / SLO΋ಉ࣌ʹߦ͏
    ݱࡏͷӡ༻
    ैདྷͷӡ༻
    

    View Slide

28. ࠓޙͷల๬
    ● ·ͩ·ͩӡ༻࢝ΊͨͯͳͷͰɺࠓޙ΋஌ݟΛஷΊ͍ͯ͘
    
    
    ● ࢦඪͦͷ΋ͷͷܧଓతͳϒϥογϡΞοϓ
    
    
    ○ ࢦඪͷ͖͍͠஋ɺλʔήοτʢ90% → 50%ʣ
    
    
    ○ ΤϥʔόδΣοτ
    
    
    ○ ػೳͷΩʔϝτϦΫεͷର৅
    
    
    ● DataDogͰҰؾ௨؏ͯ͠ϞχλϦϯάɺμογϡϘʔυΛ؅ཧͰ͖ΔΑ͏ʹ͢Δ
    
    
    ● ૊৫಺ͰͷSLI / SLOͷਁಁ
    
    
    ○ ݱঢ়·ͩ·ͩΤϯδχΞҎ֎ʹ͸ਁಁͰ͖ͯͳ͍
    

    View Slide

29. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
    

    View Slide