Upgrade to Pro — share decks privately, control downloads, hide ads and more …

よりUXに近いSLI・SLOの運用による可用性の再設計

953d2907b80dd5955c561d695ea9a494?s=47 nade
May 15, 2022

 よりUXに近いSLI・SLOの運用による可用性の再設計

953d2907b80dd5955c561d695ea9a494?s=128

nade

May 15, 2022
Tweet

More Decks by nade

Other Decks in Programming

Transcript

  1. ΑΓUXʹ͍ۙSLIɾSLOͷӡ ༻ʹΑΔՄ༻ੑͷ࠶ઃܭ Ӭ໺ Ұഅ

  2. ࣗݾ঺հ • 2019/4ʙ λοϓϧʹiOSΤϯδχΞͱͯ͠ δϣΠϯ • 2021೥ΑΓSREνʔϜʹϞόΠϧΤϯδχ Ξͱͯ͠ॴଐ • ݱࡏ͸αʔϏεͷ৴པੑΛߴΊΔϛογϣ

    ϯʹऔΓ૊ΜͰ͍·͢ Ӭ໺ Ұഅ
  3. λοϓϧͷαʔϏεʹ͍ͭͯ https://tapple.me/ 2014೥5݄ʹϦϦʔεͨ͠ࠃ಺࠷େن໛ͷϚονϯάΞϓϦͰ͢ɻ1,000छҎ্ͷσʔτϓϥϯ͔Β૬ख Λืूͯ͠ϚονϯάͰ͖Δʮ͓Ͱ͔͚ػೳʯΛఏڙ͠ɺ೥ऩɾ৬ۀͳͲͷ৚͚݅ͩͰͳ͘ɺझຯΛ͖ͬ ͔͚ʹࣗ෼Β͘͠աͤ͝Δύʔτφʔͱग़ձ͑Δͷ͕ಛ௃Ͱɺ20୅ͷஉঁΛத৺ʹࢧ࣋Λड͚͍ͯ·͢ɻ

  4. • SREνʔϜʹϞ バ ΠϧΤϯ ジ χΞ(iOSɺAndroid֤1ਓ) が ઐଐ で ॴଐ͢Δܗ

    • ΑΓϢʔβʔʹ͍ۙ෦෼ͰͷܭଌɺBFFͷਪਐͱ͍ͬͨόοΫΤϯυSRE͚ͩͰ͸खΛ৳͹ͮ͠Β ͍෦෼ʹ΋ΞδϦςΟߴ͘औΓ૊ΊΔΑ͏ʹ͢Δ໨త • Πϯϑϥ΍όοΫΤϯυͷઐ໳ੑͷ֫ಘ΋ SREνʔϜͷMobileΤϯδχΞʹ͍ͭͯ
  5. ࠓճͷൃද಺༰ SLI / SLOΛϞόΠϧΞϓϦέʔγϣϯଆͰܭଌ͞ΕΔࢦඪΛ΋ͱʹ࠶ઃܭ͠ɺΑΓUXʹ͍ۙࢦඪΛ໨ࢦ ͢औΓ૊Έͷ঺հ • ৴པੑΛճ෮ͤ͞ΔͨΊͷΞΫγϣϯ΍ɺΠϯγσϯτ΁ͷରॲΛఆྔతʹߦ͑ΔΑ͏ʹ • ৴པੑΛఆྔతʹ೺ѲͰ͖Δ͜ͱͰͷ҆৺ײ •

    ࣄۀ൑அ΍༏ઌ౓ͷίϛϡχέʔγϣϯʹ͓͚Δڞ௨ݴޠʹ
  6. ैདྷͷSLI/SLOͷӡ༻

  7. όοΫΤϯυ • APIαʔόʔ, BFFαʔόʔͦΕͧΕͷϨΠςϯγΛSLIͱͯ͠ར༻ • ϞχλϦϯάɺΞϥʔτڞʹDataDogͰ؅ཧ • ҧ൓࣌΍ɺΤϥʔόδΣοτΛஶ͘͠ফඅ͢Δ৔߹͸ଈ࣌ো֐ରԠ ैདྷͷSLI/SLOͷӡ༻ —

    όοΫΤϯυ — https://developers.cyberagent.co.jp/blog/archives/20354/
  8. ΫϥΠΞϯτ ΫϥΠΞϯτଆͰ͸ಠࣗͷSLIΛӡ༻ • Ϋϥογϡ཰ɺύʔεΤϥʔ਺ɺϑϨʔϜϨʔτ ैདྷͷSLI/SLOͷӡ༻ — ΫϥΠΞϯτ —

  9. • Firebase Performance, CrashlyticsͷϩάΛར༻͠ɺϕϩγςΟΞϥʔτΛར༻ • Data PortalͰμογϡϘʔυ؅ཧ • ҧ൓࣌͸Өڹͷൣғɺػೳͷॏཁ౓ʹԠͯ͡ΞΫγϣϯΛܾఆ 


    ैདྷͷSLI/SLOͷӡ༻ — ΫϥΠΞϯτ — ΠϯύΫτ ΞΫγϣϯ High ࢪࡦΛͱΊͯͰ΋νʔϜશମͰՄೳͳݶΓϦιʔεΛͭ͗ࠐΈɺ ϗοτϑΟοΫε΋ݕ౼ Middle ୲౰ͷΤϯδχΞΛΞαΠϯ্ͨ͠Ͱɺ࣍ճϦϦʔεͰඞͣରԠɻ ඞཁͰ͋Ε͹ࢪࡦͷख΋ͱΊΔ Low ୲౰ͷΤϯδχΞΛΞαΠϯ্ͨ͠Ͱɺ ։ൃϦιʔε͕ۭ͖࣍ୈదٓରԠ͢Δ
  10. SLO͕αʔϏεͷՄ༻ੑͷ൑அ࣠ʹͳ͍ͬͯͳ͍ 
 
 
 1. emergencyϨϕϧͷΞϥʔτ͕ൃՐ 2. ΤϯδχΞ͕֤ࣗӨڹൣғʢػೳɺϢʔβʔ਺ʣௐࠪ 3. ΞΫγϣϯܾఆʢ͓஌ΒͤɺTwitterࠂ஌ɺ޿ࠂఀࢭɺϝϯςΠϯʣ

    
 
 
 1. emergencyϨϕϧͷΞϥʔτ͕ൃՐ 2. SLI/SLOμογϡϘʔυͰӨڹൣғ֬ೝ 3. ӨڹͷͰ͍ͯΔSLIɺΤϥʔόδΣοτͷফඅ͔Β࣍ͷΞΫγϣϯΛܾఆ ैདྷͷSLI/SLOͷӡ༻ — ՝୊ᶃ — ຊདྷͷཧ૝ ݱࡏͷӡ༻
  11. όοΫΤϯυɺΫϥΠΞϯτʹดͨ͡ࢦඪͦΕͧΕͷΈͰ͸Մ༻ੑΛอূͰ͖ͳ͍ 
 
 αʔόʔͷSLO͕ຬͨ͞Ε͍ͯͯ΋ɺ • ૬खͷϝοηʔδ͕ݟΕ͍ͯͳ͍ 
 → ΫϥΠΞϯτଆͰͷύʔεΤϥʔ͕ݪҼͰϨεϙϯεΛड৴Ͱ͖ͯͳ͍ •

    ৽نొ࿥Ͱ͖ͳ͍ 
 → ֎෦SNSଆͰো֐ൃੜɺΦϯϘʔσΟϯάϑϩʔ͕ແݶϧʔϓ 
 
 
 • ϝοηʔδը໘Λ։͍ͨϢʔβʔͷ◦%͕ϝοηʔδͷӾཡʹ੒ޭ͍ͯ͠Δঢ়ଶ͔Θ͔Δ • ৽نొ࿥Λ։࢝ͨ͠Ϣʔβʔͷ◦%͕৽نొ࿥ʹ੒ޭ͍ͯ͠Δ͔Θ͔Δ ैདྷͷSLI/SLOͷӡ༻ — ՝୊ᶄ — ຊདྷͷཧ૝ ݱࡏͷӡ༻
  12. ໨ࢦ͢ཧ૝ঢ়ଶ • Ϣʔβʔͷظ଴͢ΔUXΛΧόʔ͢Δࢦඪઃఆ 
 • SLI / SLOͷҧ൓ʹΑΔϢʔβʔӨڹ೺ѲΛ༰қʹ 
 •

    ϏδωεɺΫϥΠΞϯτɺαʔόʔؒͰαʔϏεঢ়ଶͷڞ௨ݴޠΛ࣋ͭ
  13. ΑΓUXʹ͍ۙSLIɾSLOͷઃܭ

  14. Ϣʔβʔͷظ଴͢ΔUXΛΧόʔ͢ΔSLIઃఆ ϝϯόʔ͔ΒʮαʔϏεͷՄ༻ੑʯͷٵ্͍͛ • औΓ͍ͨࢦඪɺఆٛɺܭଌํ๏ɺѱԽͨ͠৔߹ʹଛͳΘΕΔUXΛώΞϦϯά • ϑΥʔΧε͢ΔࢦඪɺίΞͳମݧΛ࢈ΜͰ͍ΔUXΛܾఆ͢Δ ϚΠΫϩαʔϏε౳ͷυϝΠϯϞσϧͱ߹கͤ͞Δ

  15. ϓϥοτϑΥʔϜͷఏڙ͢Δࢦඪͷ࠾༻ Apple, Google͕σΟϕϩούʔʹఏڙ͢ΔSLI • ϓϥοτϑΥʔϜଆͷߟ͑Δྑ͍ΞϓϦͷఆٛ • ΞϓϦετΞ΍OSϨϕϧͰӨڹ͕͋Δ͜ͱ΋ ◦ ASOʢApp Store

    Optimizationʣ ◦ ༏ઌͯ͠ΞϯΠϯετʔϧΛଅ͞ΕΔ ◦ ΞϓϦͷόοΫάϥ΢ϯυͰͷੜଘ࣌ؒ • λοϓϧͰ͸ԼهͷࢦඪΛSLIͱͯ͠࠾༻ ◦ ΞϓϦαΠζ ◦ શମΫϥογϡ཰ʢANRʣ ◦ ىಈ࣌ؒ https://developers.cyberagent.co.jp/blog/archives/20354/
  16. SLIҧ൓Λ࠶ݱͨ͠ΞϓϦΛ࢖ͬͨSLOܾΊ • աڈͷΠϯγσϯτ࣌ͷࢦඪ΍ɺࣄલώΞϦϯά͔Βɺ͖͍͠஋ʹ͋ͨΓΛ͚ͭΔ 
 • σόοάػೳͰৄࡉͳ͖͍͠஋Λௐ੔ՄೳͳΞϓϦΛ༻ҙ͠ɺ৮Γͳ͕Βٞ࿦

  17. ࠶ఆٛͨ͠SLI όοΫΤϯυʢܧଓʣ • ϨΠςϯγʔ 
 ΫϥΠΞϯτʢʹϓϥοτϑΥʔϜࢦඪʣ • ىಈ࣌ؒɺશମΫϥογϡ཰ɺΞϓϦαΠζ 
 ίΞػೳʢʹػೳ͝ͱͷՄ༻ੑʣ

    • ϩάΠϯɺ৽نొ࿥ɺΧʔυϑϦοΫɺ 
 ϝοηʔδɺ͓Ͱ͔͚ɺWish 
 Մ༻ੑΛ΋ͱʹͨ͠SLIΛઃఆ 
 ڞ௨ Ϋϥογϡ / ANR ػೳ͝ͱͷΩʔϝτϦΫε ϝοηʔδ: ಡΈࠐΈ଎౓ ΧʔυϑϦοΫ: ࠶ϑϦοΫՄೳʹͳΔ·Ͱͷ࣌ؒ 
 ɾɾɾ
  18. ΫϥΠΞϯτଆͰͷSLI / SLOͷܭଌ

  19. ΫϥΠΞϯτଆͰͷSLI / SLOͷܭଌ 1. ·ͣ͸Firebase (+BigQuery) ʹ׬݁ͯ͠ϛχϚϜʹՔಇ • ػೳతʹे෼ͳ͜ͱΛݕূ 2.

    Monitoring, DashboardΛDataDogʹҠߦʢ͜Ε͔Βʣ • Ξϥʔτͷࣗ༝౓Λ্͛Δ • ଞͷϩάͰར༻ͯ͠ΔαʔϏεͱ߹ΘͤΔ
  20. ΫϥΠΞϯτଆͰͷϞχλϦϯάπʔϧͷݕ౼ ͻͱ·ͣྉۚ໘Λߟྀ͠FirebaseΛར༻͠ɺʮӡ༻Λ։࢝ → ՝୊͕͋Ε͹ϒϥογϡΞοϓʯͷํ਑ʹ 
 1. Firebase Crashlytics / Performance

    + BigQuery • Firebaseࣗମ͸ແྉͷͨΊBigQueryͷQuery, Active StorageͷྉۚͷΈ • Crashlytics, Performance͸طଘͰ΋ར༻͍ͯ͠ΔͨΊಋೖίετ͕௿͍ • BigQueryʹ஝ੵ͢Δ͜ͱͰൺֱతࣗ༝౓͸ߴ͘ར༻Ͱ͖Δ 2. DataDog RUM • طଘͷϞχλϦϯάπʔϧͱͯ͠DataDogΛར༻͍ͯ͠ΔͨΊπʔϧΛ౷ҰͰ͖Δ • ηογϣϯ୯Ґͷ՝ۚମܥͷͨΊɺࠓճ͸ྉۚ໘Ͱஅ೦ 3. New Relic One • όοΫΤϯυͰDataDogΛར༻͍ͯ͠Δ͜ͱɺྉۚ໘Λഎܠʹஅ೦
  21. FirebaseͰߏ੒͢ΔϛχϚϜͳܭଌϑϩʔ Firebase ( Crashlytics + Performance ) ͷΈͰϛχϚϜͳϑϩʔ͸࣮ݱՄೳ • BigQueryʹExportʢϦΞϧλΠϜ΋Մೳʣ

    • BigQuery্ʹμογϡϘʔυදࣔ༻ͷதؒςʔϒϧ࡞੒ • Data PortalͰμογϡϘʔυදࣔ )LUHEDVH 3HUIRUPDQFH )LUHEDVH &UDVKO\WLFV  8VHU &UDVK .H\0HWULFV *RRJOH'DWD3RUWDO $XWR([SRUW %LJ4XHU\ ,QWHUPHGLDWHWDEOH %LJ4XHU\
  22. Ξϥʔτͷઃఆͷࣗ༝౓ Firebase Performance • ࡞੒ͨ͠ࢦඪ͝ͱʹΞϥʔτͷઃఆ͕Մೳ • ͖͍͠஋Λ50ʙ95ύʔηϯλΠϧͷࢦఆ͕Մೳ Firebase Crashlytics •

    ௚ۙ1࣌ؒͷηογϣϯʹର͢ΔΫϥογϡ཰ • ͖͍͠஋Λ0.1%ʙ1.0%ͰίϯτϩʔϧՄೳ
  23. Ϋϥογϡϩάͷ෼ྨ https://firebase.google.com/docs/crashlytics/customize-crash-reports import UIKit import FirebaseCrashlytics final class LoginViewController: UIViewController

    { // ը໘Λදࣔ͢ΔλΠϛϯάͰCrashlyticsʹΧελϜΩʔΛઃఆ override func viewWillAppear(_ animated: Bool) { super.viewWillAppear(animated) Crashlytics.crashlytics().setCustomValue("login", forKey: "domain") } iOSΞϓϦͰͷϩάΠϯը໘ʹର͢Δ࣮૷ྫ ֤ػೳͷը໘දࣔ࣌ʹCrashlyticsʹΧελϜΩʔΛઃఆ͢Δ • ΧελϜΩʔΛ΋ͱʹBigQuery (Data Portal) ্ͰػೳυϝΠϯ͝ͱͷΫϥογϡΛूܭ • ػೳ͝ͱͷΞϥʔτ͸ઃఆͰ͖ͳ͍ͨΊɺVelocity Alert → μογϡϘʔυͰػೳӨڹ֬ೝ
  24. ࠓޙʣϞχλϦϯάΛDataDogʹ౷Ұ ҎԼͷ؍఺͔ΒDataDogʹू໿͍ͯ͘͠ • ϞχλϦϯάͷू໿ • Ϋϥογϡͷػೳ͝ͱͷΞϥʔτ )LUHEDVH 3HUIRUPDQFH )LUHEDVH &UDVKO\WLFV

    $XWR([SRUW %LJ4XHU\ 4XHU\ SXVK IRUZRUGWRGDWDGRJ &ORXG)XQFWLRQV 'DWD'RJ &ORXG 3XE6XE WULJJHU
  25. ӡ༻ʹ͍ͭͯ

  26. ӡ༻ͷมԽ Πϯγσϯτൃੜ࣌ • PMɺόοΫΤϯυɺΫϥΠΞϯτϝϯόʔશһ͕ϢʔβʔӨڹΛ൑அ͢ΔҰ࣌৘ใͱͯ͠׆༻ ◦ ΤϯδχΞͷௐࠪΛ଴ͨͣͯ͠ωΫετΞΫγϣϯΛಈ͖ग़ͤΔΑ͏ʹ 
 ྫʣશϢʔβʔ΁ͷ͓͠ΒͤɺTwitterࠂ஌ɺ޿ࠂͷఀࢭ • ِཅੑͷ௿Լ

    ◦ ಛఆͷ৚݅ԼʢΩϟϯϖʔϯதͷΈ౳ʣʹൃੜ͢ΔύʔεΤϥʔΛ༏ઌ౓ߴ͘ΞϥʔτΛ͋͛ ΒΕΔΑ͏ʹ
  27. ӡ༻ͷมԽ ϦϦʔεޙͷϝτϦΫε֬ೝ࣌ʹόοΫΤϯυɺΫϥΠΞϯτͷڞ௨ݴޠʹ 
 • όοΫΤϯυɺΫϥΠΞϯτͦΕͧΕ͕׬શʹ෼͔Εͯ֬ೝ • ֬ೝՄೳͳϝτϦΫεɺμογϡϘʔυΛҰ௨Γ໢ཏతʹ֬ೝ • όοΫΤϯυɺΫϥΠΞϯτڞ௨ͷSLI /

    SLO μογϡϘʔυΛ֬ೝ ◦ ௐࠪͷ৔߹͸SLIΛڞ௨ݴޠͱͯ͠ίϛϡχέʔγϣϯ • ҧ൓͕͋ͬͨ৔߹ʹؔ࿈ͷϝτϦΫεɺμογϡϘʔυΛ֬ೝ ◦ SLI / SLO΋ಉ࣌ʹߦ͏ ݱࡏͷӡ༻ ैདྷͷӡ༻
  28. ࠓޙͷల๬ • ·ͩ·ͩӡ༻࢝ΊͨͯͳͷͰɺࠓޙ΋஌ݟΛஷΊ͍ͯ͘ • ࢦඪͦͷ΋ͷͷܧଓతͳϒϥογϡΞοϓ ◦ ࢦඪͷ͖͍͠஋ɺλʔήοτʢ90% → 50%ʣ ◦

    ΤϥʔόδΣοτ ◦ ػೳͷΩʔϝτϦΫεͷର৅ • DataDogͰҰؾ௨؏ͯ͠ϞχλϦϯάɺμογϡϘʔυΛ؅ཧͰ͖ΔΑ͏ʹ͢Δ • ૊৫಺ͰͷSLI / SLOͷਁಁ ◦ ݱঢ়·ͩ·ͩΤϯδχΞҎ֎ʹ͸ਁಁͰ͖ͯͳ͍
  29. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠