よりUXに近いSLI・SLOの運用による可用性の再設計

SRE NEXT 2022
https://sre-next.dev/2022/schedule/#jp47

Transcript

1. ΑΓUXʹ͍ۙSLIɾSLOͷӡ ༻ʹΑΔՄ༻ੑͷ࠶ઃܭ Ӭ໺ Ұഅ
2. ࣗݾ঺հ • 2019/4ʙ λοϓϧʹiOSΤϯδχΞͱͯ͠ δϣΠϯ • 2021೥ΑΓSREνʔϜʹϞόΠϧΤϯδχ Ξͱͯ͠ॴଐ • ݱࡏ͸αʔϏεͷ৴པੑΛߴΊΔϛογϣ ϯʹऔΓ૊ΜͰ͍·͢ Ӭ໺ Ұഅ
3. λοϓϧͷαʔϏεʹ͍ͭͯ https://tapple.me/ 2014೥5݄ʹϦϦʔεͨ͠ࠃ಺࠷େن໛ͷϚονϯάΞϓϦͰ͢ɻ1,000छҎ্ͷσʔτϓϥϯ͔Β૬ख Λืूͯ͠ϚονϯάͰ͖Δʮ͓Ͱ͔͚ػೳʯΛఏڙ͠ɺ೥ऩɾ৬ۀͳͲͷ৚͚݅ͩͰͳ͘ɺझຯΛ͖ͬ ͔͚ʹࣗ෼Β͘͠աͤ͝Δύʔτφʔͱग़ձ͑Δͷ͕ಛ௃Ͱɺ20୅ͷஉঁΛத৺ʹࢧ࣋Λड͚͍ͯ·͢ɻ
4. • SREνʔϜʹϞ バ ΠϧΤϯ ジ χΞ(iOSɺAndroid֤1ਓ) が ઐଐ で ॴଐ͢Δܗ • ΑΓϢʔβʔʹ͍ۙ෦෼ͰͷܭଌɺBFFͷਪਐͱ͍ͬͨόοΫΤϯυSRE͚ͩͰ͸खΛ৳͹ͮ͠Β ͍෦෼ʹ΋ΞδϦςΟߴ͘औΓ૊ΊΔΑ͏ʹ͢Δ໨త • Πϯϑϥ΍όοΫΤϯυͷઐ໳ੑͷ֫ಘ΋ SREνʔϜͷMobileΤϯδχΞʹ͍ͭͯ
5. ࠓճͷൃද಺༰ SLI / SLOΛϞόΠϧΞϓϦέʔγϣϯଆͰܭଌ͞ΕΔࢦඪΛ΋ͱʹ࠶ઃܭ͠ɺΑΓUXʹ͍ۙࢦඪΛ໨ࢦ ͢औΓ૊Έͷ঺հ • ৴པੑΛճ෮ͤ͞ΔͨΊͷΞΫγϣϯ΍ɺΠϯγσϯτ΁ͷରॲΛఆྔతʹߦ͑ΔΑ͏ʹ • ৴པੑΛఆྔతʹ೺ѲͰ͖Δ͜ͱͰͷ҆৺ײ • ࣄۀ൑அ΍༏ઌ౓ͷίϛϡχέʔγϣϯʹ͓͚Δڞ௨ݴޠʹ
6. ैདྷͷSLI/SLOͷӡ༻
7. όοΫΤϯυ • APIαʔόʔ, BFFαʔόʔͦΕͧΕͷϨΠςϯγΛSLIͱͯ͠ར༻ • ϞχλϦϯάɺΞϥʔτڞʹDataDogͰ؅ཧ • ҧ൓࣌΍ɺΤϥʔόδΣοτΛஶ͘͠ফඅ͢Δ৔߹͸ଈ࣌ো֐ରԠ ैདྷͷSLI/SLOͷӡ༻ — όοΫΤϯυ — https://developers.cyberagent.co.jp/blog/archives/20354/
8. ΫϥΠΞϯτ ΫϥΠΞϯτଆͰ͸ಠࣗͷSLIΛӡ༻ • Ϋϥογϡ཰ɺύʔεΤϥʔ਺ɺϑϨʔϜϨʔτ ैདྷͷSLI/SLOͷӡ༻ — ΫϥΠΞϯτ —
9. • Firebase Performance, CrashlyticsͷϩάΛར༻͠ɺϕϩγςΟΞϥʔτΛར༻ • Data PortalͰμογϡϘʔυ؅ཧ • ҧ൓࣌͸Өڹͷൣғɺػೳͷॏཁ౓ʹԠͯ͡ΞΫγϣϯΛܾఆ 
 ैདྷͷSLI/SLOͷӡ༻ — ΫϥΠΞϯτ — ΠϯύΫτ ΞΫγϣϯ High ࢪࡦΛͱΊͯͰ΋νʔϜશମͰՄೳͳݶΓϦιʔεΛͭ͗ࠐΈɺ ϗοτϑΟοΫε΋ݕ౼ Middle ୲౰ͷΤϯδχΞΛΞαΠϯ্ͨ͠Ͱɺ࣍ճϦϦʔεͰඞͣରԠɻ ඞཁͰ͋Ε͹ࢪࡦͷख΋ͱΊΔ Low ୲౰ͷΤϯδχΞΛΞαΠϯ্ͨ͠Ͱɺ ։ൃϦιʔε͕ۭ͖࣍ୈదٓରԠ͢Δ
10. SLO͕αʔϏεͷՄ༻ੑͷ൑அ࣠ʹͳ͍ͬͯͳ͍ 
 
 
 1. emergencyϨϕϧͷΞϥʔτ͕ൃՐ 2. ΤϯδχΞ͕֤ࣗӨڹൣғʢػೳɺϢʔβʔ਺ʣௐࠪ 3. ΞΫγϣϯܾఆʢ͓஌ΒͤɺTwitterࠂ஌ɺ޿ࠂఀࢭɺϝϯςΠϯʣ 
 
 
 1. emergencyϨϕϧͷΞϥʔτ͕ൃՐ 2. SLI/SLOμογϡϘʔυͰӨڹൣғ֬ೝ 3. ӨڹͷͰ͍ͯΔSLIɺΤϥʔόδΣοτͷফඅ͔Β࣍ͷΞΫγϣϯΛܾఆ ैདྷͷSLI/SLOͷӡ༻ — ՝୊ᶃ — ຊདྷͷཧ૝ ݱࡏͷӡ༻
11. όοΫΤϯυɺΫϥΠΞϯτʹดͨ͡ࢦඪͦΕͧΕͷΈͰ͸Մ༻ੑΛอূͰ͖ͳ͍ 
 
 αʔόʔͷSLO͕ຬͨ͞Ε͍ͯͯ΋ɺ • ૬खͷϝοηʔδ͕ݟΕ͍ͯͳ͍ 
 → ΫϥΠΞϯτଆͰͷύʔεΤϥʔ͕ݪҼͰϨεϙϯεΛड৴Ͱ͖ͯͳ͍ • ৽نొ࿥Ͱ͖ͳ͍ 
 → ֎෦SNSଆͰো֐ൃੜɺΦϯϘʔσΟϯάϑϩʔ͕ແݶϧʔϓ 
 
 
 • ϝοηʔδը໘Λ։͍ͨϢʔβʔͷ◦%͕ϝοηʔδͷӾཡʹ੒ޭ͍ͯ͠Δঢ়ଶ͔Θ͔Δ • ৽نొ࿥Λ։࢝ͨ͠Ϣʔβʔͷ◦%͕৽نొ࿥ʹ੒ޭ͍ͯ͠Δ͔Θ͔Δ ैདྷͷSLI/SLOͷӡ༻ — ՝୊ᶄ — ຊདྷͷཧ૝ ݱࡏͷӡ༻
12. ໨ࢦ͢ཧ૝ঢ়ଶ • Ϣʔβʔͷظ଴͢ΔUXΛΧόʔ͢Δࢦඪઃఆ 
 • SLI / SLOͷҧ൓ʹΑΔϢʔβʔӨڹ೺ѲΛ༰қʹ 
 • ϏδωεɺΫϥΠΞϯτɺαʔόʔؒͰαʔϏεঢ়ଶͷڞ௨ݴޠΛ࣋ͭ
13. ΑΓUXʹ͍ۙSLIɾSLOͷઃܭ
14. Ϣʔβʔͷظ଴͢ΔUXΛΧόʔ͢ΔSLIઃఆ ϝϯόʔ͔ΒʮαʔϏεͷՄ༻ੑʯͷٵ্͍͛ • औΓ͍ͨࢦඪɺఆٛɺܭଌํ๏ɺѱԽͨ͠৔߹ʹଛͳΘΕΔUXΛώΞϦϯά • ϑΥʔΧε͢ΔࢦඪɺίΞͳମݧΛ࢈ΜͰ͍ΔUXΛܾఆ͢Δ ϚΠΫϩαʔϏε౳ͷυϝΠϯϞσϧͱ߹கͤ͞Δ
15. ϓϥοτϑΥʔϜͷఏڙ͢Δࢦඪͷ࠾༻ Apple, Google͕σΟϕϩούʔʹఏڙ͢ΔSLI • ϓϥοτϑΥʔϜଆͷߟ͑Δྑ͍ΞϓϦͷఆٛ • ΞϓϦετΞ΍OSϨϕϧͰӨڹ͕͋Δ͜ͱ΋ ◦ ASOʢApp Store Optimizationʣ ◦ ༏ઌͯ͠ΞϯΠϯετʔϧΛଅ͞ΕΔ ◦ ΞϓϦͷόοΫάϥ΢ϯυͰͷੜଘ࣌ؒ • λοϓϧͰ͸ԼهͷࢦඪΛSLIͱͯ͠࠾༻ ◦ ΞϓϦαΠζ ◦ શମΫϥογϡ཰ʢANRʣ ◦ ىಈ࣌ؒ https://developers.cyberagent.co.jp/blog/archives/20354/
16. SLIҧ൓Λ࠶ݱͨ͠ΞϓϦΛ࢖ͬͨSLOܾΊ • աڈͷΠϯγσϯτ࣌ͷࢦඪ΍ɺࣄલώΞϦϯά͔Βɺ͖͍͠஋ʹ͋ͨΓΛ͚ͭΔ 
 • σόοάػೳͰৄࡉͳ͖͍͠஋Λௐ੔ՄೳͳΞϓϦΛ༻ҙ͠ɺ৮Γͳ͕Βٞ࿦
17. ࠶ఆٛͨ͠SLI όοΫΤϯυʢܧଓʣ • ϨΠςϯγʔ 
 ΫϥΠΞϯτʢʹϓϥοτϑΥʔϜࢦඪʣ • ىಈ࣌ؒɺશମΫϥογϡ཰ɺΞϓϦαΠζ 
 ίΞػೳʢʹػೳ͝ͱͷՄ༻ੑʣ • ϩάΠϯɺ৽نొ࿥ɺΧʔυϑϦοΫɺ 
 ϝοηʔδɺ͓Ͱ͔͚ɺWish 
 Մ༻ੑΛ΋ͱʹͨ͠SLIΛઃఆ 
 ڞ௨ Ϋϥογϡ / ANR ػೳ͝ͱͷΩʔϝτϦΫε ϝοηʔδ: ಡΈࠐΈ଎౓ ΧʔυϑϦοΫ: ࠶ϑϦοΫՄೳʹͳΔ·Ͱͷ࣌ؒ 
 ɾɾɾ
18. ΫϥΠΞϯτଆͰͷSLI / SLOͷܭଌ
19. ΫϥΠΞϯτଆͰͷSLI / SLOͷܭଌ 1. ·ͣ͸Firebase (+BigQuery) ʹ׬݁ͯ͠ϛχϚϜʹՔಇ • ػೳతʹे෼ͳ͜ͱΛݕূ 2. Monitoring, DashboardΛDataDogʹҠߦʢ͜Ε͔Βʣ • Ξϥʔτͷࣗ༝౓Λ্͛Δ • ଞͷϩάͰར༻ͯ͠ΔαʔϏεͱ߹ΘͤΔ
20. ΫϥΠΞϯτଆͰͷϞχλϦϯάπʔϧͷݕ౼ ͻͱ·ͣྉۚ໘Λߟྀ͠FirebaseΛར༻͠ɺʮӡ༻Λ։࢝ → ՝୊͕͋Ε͹ϒϥογϡΞοϓʯͷํ਑ʹ 
 1. Firebase Crashlytics / Performance + BigQuery • Firebaseࣗମ͸ແྉͷͨΊBigQueryͷQuery, Active StorageͷྉۚͷΈ • Crashlytics, Performance͸طଘͰ΋ར༻͍ͯ͠ΔͨΊಋೖίετ͕௿͍ • BigQueryʹ஝ੵ͢Δ͜ͱͰൺֱతࣗ༝౓͸ߴ͘ར༻Ͱ͖Δ 2. DataDog RUM • طଘͷϞχλϦϯάπʔϧͱͯ͠DataDogΛར༻͍ͯ͠ΔͨΊπʔϧΛ౷ҰͰ͖Δ • ηογϣϯ୯Ґͷ՝ۚମܥͷͨΊɺࠓճ͸ྉۚ໘Ͱஅ೦ 3. New Relic One • όοΫΤϯυͰDataDogΛར༻͍ͯ͠Δ͜ͱɺྉۚ໘Λഎܠʹஅ೦
21. FirebaseͰߏ੒͢ΔϛχϚϜͳܭଌϑϩʔ Firebase ( Crashlytics + Performance ) ͷΈͰϛχϚϜͳϑϩʔ͸࣮ݱՄೳ • BigQueryʹExportʢϦΞϧλΠϜ΋Մೳʣ • BigQuery্ʹμογϡϘʔυදࣔ༻ͷதؒςʔϒϧ࡞੒ • Data PortalͰμογϡϘʔυදࣔ )LUHEDVH 3HUIRUPDQFH )LUHEDVH &UDVKO\WLFV  8VHU &UDVK .H\0HWULFV *RRJOH'DWD3RUWDO $XWR([SRUW %LJ4XHU\ ,QWHUPHGLDWHWDEOH %LJ4XHU\
22. Ξϥʔτͷઃఆͷࣗ༝౓ Firebase Performance • ࡞੒ͨ͠ࢦඪ͝ͱʹΞϥʔτͷઃఆ͕Մೳ • ͖͍͠஋Λ50ʙ95ύʔηϯλΠϧͷࢦఆ͕Մೳ Firebase Crashlytics • ௚ۙ1࣌ؒͷηογϣϯʹର͢ΔΫϥογϡ཰ • ͖͍͠஋Λ0.1%ʙ1.0%ͰίϯτϩʔϧՄೳ
23. Ϋϥογϡϩάͷ෼ྨ https://firebase.google.com/docs/crashlytics/customize-crash-reports import UIKit import FirebaseCrashlytics final class LoginViewController: UIViewController { // ը໘Λදࣔ͢ΔλΠϛϯάͰCrashlyticsʹΧελϜΩʔΛઃఆ override func viewWillAppear(_ animated: Bool) { super.viewWillAppear(animated) Crashlytics.crashlytics().setCustomValue("login", forKey: "domain") } iOSΞϓϦͰͷϩάΠϯը໘ʹର͢Δ࣮૷ྫ ֤ػೳͷը໘දࣔ࣌ʹCrashlyticsʹΧελϜΩʔΛઃఆ͢Δ • ΧελϜΩʔΛ΋ͱʹBigQuery (Data Portal) ্ͰػೳυϝΠϯ͝ͱͷΫϥογϡΛूܭ • ػೳ͝ͱͷΞϥʔτ͸ઃఆͰ͖ͳ͍ͨΊɺVelocity Alert → μογϡϘʔυͰػೳӨڹ֬ೝ
24. ࠓޙʣϞχλϦϯάΛDataDogʹ౷Ұ ҎԼͷ؍఺͔ΒDataDogʹू໿͍ͯ͘͠ • ϞχλϦϯάͷू໿ • Ϋϥογϡͷػೳ͝ͱͷΞϥʔτ )LUHEDVH 3HUIRUPDQFH )LUHEDVH &UDVKO\WLFV $XWR([SRUW %LJ4XHU\ 4XHU\ SXVK IRUZRUGWRGDWDGRJ &ORXG)XQFWLRQV 'DWD'RJ &ORXG 3XE6XE WULJJHU
25. ӡ༻ʹ͍ͭͯ
26. ӡ༻ͷมԽ Πϯγσϯτൃੜ࣌ • PMɺόοΫΤϯυɺΫϥΠΞϯτϝϯόʔશһ͕ϢʔβʔӨڹΛ൑அ͢ΔҰ࣌৘ใͱͯ͠׆༻ ◦ ΤϯδχΞͷௐࠪΛ଴ͨͣͯ͠ωΫετΞΫγϣϯΛಈ͖ग़ͤΔΑ͏ʹ 
 ྫʣશϢʔβʔ΁ͷ͓͠ΒͤɺTwitterࠂ஌ɺ޿ࠂͷఀࢭ • ِཅੑͷ௿Լ ◦ ಛఆͷ৚݅ԼʢΩϟϯϖʔϯதͷΈ౳ʣʹൃੜ͢ΔύʔεΤϥʔΛ༏ઌ౓ߴ͘ΞϥʔτΛ͋͛ ΒΕΔΑ͏ʹ
27. ӡ༻ͷมԽ ϦϦʔεޙͷϝτϦΫε֬ೝ࣌ʹόοΫΤϯυɺΫϥΠΞϯτͷڞ௨ݴޠʹ 
 • όοΫΤϯυɺΫϥΠΞϯτͦΕͧΕ͕׬શʹ෼͔Εͯ֬ೝ • ֬ೝՄೳͳϝτϦΫεɺμογϡϘʔυΛҰ௨Γ໢ཏతʹ֬ೝ • όοΫΤϯυɺΫϥΠΞϯτڞ௨ͷSLI / SLO μογϡϘʔυΛ֬ೝ ◦ ௐࠪͷ৔߹͸SLIΛڞ௨ݴޠͱͯ͠ίϛϡχέʔγϣϯ • ҧ൓͕͋ͬͨ৔߹ʹؔ࿈ͷϝτϦΫεɺμογϡϘʔυΛ֬ೝ ◦ SLI / SLO΋ಉ࣌ʹߦ͏ ݱࡏͷӡ༻ ैདྷͷӡ༻
28. ࠓޙͷల๬ • ·ͩ·ͩӡ༻࢝ΊͨͯͳͷͰɺࠓޙ΋஌ݟΛஷΊ͍ͯ͘ • ࢦඪͦͷ΋ͷͷܧଓతͳϒϥογϡΞοϓ ◦ ࢦඪͷ͖͍͠஋ɺλʔήοτʢ90% → 50%ʣ ◦ ΤϥʔόδΣοτ ◦ ػೳͷΩʔϝτϦΫεͷର৅ • DataDogͰҰؾ௨؏ͯ͠ϞχλϦϯάɺμογϡϘʔυΛ؅ཧͰ͖ΔΑ͏ʹ͢Δ • ૊৫಺ͰͷSLI / SLOͷਁಁ ◦ ݱঢ়·ͩ·ͩΤϯδχΞҎ֎ʹ͸ਁಁͰ͖ͯͳ͍
29. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠