Link
Embed
Share
Beginning
This slide
Copy link URL
Copy link URL
Copy iframe embed code
Copy iframe embed code
Copy javascript embed code
Copy javascript embed code
Share
Tweet
Share
Tweet
Slide 1
Slide 1 text
SLOࡦఆͱΞϥʔτઃఆ·Ͱͷ ͍ಓͷΓ αΠϘζגࣜձࣾ ্Ԭ ਅ (@ueokande) CloudNative Days Spring 2021 Online
Slide 2
Slide 2 text
Me • ্Ԭ ਅ (@ueokande) • αΠϘζͰDevOpsͯ͠·͢ • kintone.comͷ։ൃɾӡ༻ • ٳOSSΛ࡞ͬͨΓจॻΛॻ͍ͨΓ 2
Slide 3
Slide 3 text
kintone • ۀγεςϜΛ؆୯ʹ࡞ΕΔB2B͚αʔϏε • ࠃ֎18,000+ࣾͰಋೖ • USࢢ͚ʹʮkintone.comʯͱ͍͏ ϒϥϯυͰల։ͯ͠AWS্Ͱӡ༻ 3
Slide 4
Slide 4 text
ࠓ͢͜ͱ • US͚kintoneͷ SLO ͱ Ξϥʔτ ʹؔ͢ΔऔΓΈ • kintone.comΛAWS্Ͱϩʔϯν͔ͯ͠ΒݱࡏʹࢸΔ·Ͱɺ SLOɾΞϥʔτΛԿ͔ݟ͖ͯ͠·ͨ͠ • Ͳ͏͍ͬͨ؍Ͱվળͯ͠ɺͦͷ݁ՌͲ͏ͳ͔ͬͨΛ͓͠·͢ 4
Slide 5
Slide 5 text
ҎલͷSLOͱΞϥʔτ 5
Slide 6
Slide 6 text
SLO/SLI/SLA͓͞Β͍ 6 ݅Λຬͨͤͳ͍ͱ͖ ԿΛ͢Δ͔ͷ߹ҙ αʔϏεͷ৴པੑͷ ࢦ͖͢ඪ SLO Service Level Objective SLI Service Level Indicator SLA Service Level Agreement αʔϏεͷ৴པੑΛ ද͢ࢦඪ ྫ: 30ؒͷϦΫΤετޭ99.9% ྫ: 30ؒͷϦΫΤετޭ ྫ: ϦΫΤετޭ99%ΛԼճΕ શֹฦۚ
Slide 7
Slide 7 text
ҎલͷkintoneͷSLOͱΞϥʔτ • kintone͕ར༻Ͱ͖Δ͔൱͔ʢՔಇʣʹணͨ͠SLO • ੑೳɾεϧʔϓοτͳͲɺػೳ໘Ϣʔβʔࢹͷࢦඪͳ͔ͬͨ • ͦΕʹجͮ͘Ξϥʔτઃఆ͍ͯ͠ͳ͍ 7
Slide 8
Slide 8 text
༗༻ͳSLO͕ͳ͍ͱ… • దͳSLOࢦඪ͕ͳ͘Ξϥʔτֻ͚ʹ͍͘ • ۓٸରԠ͕ඞཁ͔ɺνϡʔχϯάඪɺλεΫͷ༏ઌ͕ܾΊʹ͍͘ 8 ὸʮͳΜ͔LJOUPOFͷද͕ࣔΊͬͪΌ͍Μ͚ͩͲʯ
Slide 9
Slide 9 text
SLOɾΞϥʔτΛݟ͖͔͚ͬ͢ • ࠃͷࣗࣾΠϯϑϥͰఏڙ͍ͯͨ͠US൛kintoneΛAWSʹ͓Ҿӽ͠ • ΞʔΩςΫνϟΛ৽͠ɺSLOɾΞϥʔτγεςϜݟ͢νϟϯε 9
Slide 10
Slide 10 text
SLOɾΞϥʔτ (v1) 10
Slide 11
Slide 11 text
SLOɾΞϥʔτ (v1) • AWS൛kintone.comͷϩʔϯνલʹSLOͱΞϥʔτΛઃఆ • ओཁͳػೳʢAPIʣʹର͢Δ͚ࣾͷSLO • ҎલͷSLOʹج͖ͮɺμϯλΠϜΛߟྀͨ͠SLOΛઃఆ 11 Ϩίʔυߋ৽ɺΞϓϦ࡞ɺ…
Slide 12
Slide 12 text
• μϯλΠϜʹجͮ͘SLOΛܾΊΔ SLO (v1) ͷྫ • API͝ͱͷμϯλΠϜΛఆٛ 12 ͋ΔAPI͕5ΟϯυͰ͍ͣΕ͔ͷج४Λ͑ΕμϯλΠϜ • ΟϯυͷΤϥʔ݅N݅Ҏ্ɺΤϥʔϨʔτ͕M%Ҏ্ • ΟϯυͷԠ࣌ؒXύʔηϯλΠϧ͕YඵҎ্ ͋ΔAPIͷ݄ؒՔಇ͕99.9%
Slide 13
Slide 13 text
μϯλΠϜͱSLO • APIͷ݄ؒՔಇ͕99.9% = ݄ؒμϯλϜ43 13 5min. 30day × × ××× ×× ××××××× ××× × ×× × 200500 ΤϥʔόδΣοτ
Slide 14
Slide 14 text
Ξϥʔτ (v1) • μϯλΠϜͷఆٛʹج͍ͮͨΞϥʔτΛͦΕͧΕͷAPIʹઃఆ 14 5ΟϯυͰ͍ͣΕ͔ͷᮢʹୡ͢ΔͱΞϥʔτ • ΟϯυͷΤϥʔ݅N݅Ҏ্ɺΤϥʔϨʔτ͕M%Ҏ্ • ΟϯυͷԠ࣌ؒXύʔηϯλΠϧ͕YඵҎ্
Slide 15
Slide 15 text
݁Ռ • SLOɾΞϥʔτ (v1) Λ͠Β͘ӡ༻ͯ͠՝͕͍͔ͭ͘ݟ͔ͭͬͨ • Ξϥʔτ͕աහ͗͢Δ • Ξϥʔτج४ͷఆ͕ٛᐆດʹͳΔ • γϯϓϧ͡Όͳ͍ 15
Slide 16
Slide 16 text
աහ͗͢ΔΞϥʔτʛΞϥʔτ (v1) ͷ݁Ռ • Ξϥʔτͷج४͕ඇৗʹաහͩͬͨ • ϦΫΤετͷগͳ͍ػೳεύΠΫͰݦஶ • աහա͗ΔΞϥʔτνʔϜϝϯόʔͷ݈߁ঢ়ଶʹӨڹ͢Δ 16
Slide 17
Slide 17 text
Ξϥʔτج४͕ᐆດʹʛΞϥʔτ (v1) ͷ݁Ռ • ʮաහ͗͢ΔΞϥʔτʯͷ࣍ͷΞΫγϣϯᮢͷ؇ • ϩʔϯνޙ͠Β͘ɺΞϥʔτͷௐΛ܁Γฦͨ͠ • ௐΛ܁Γฦ͍ͯ͠Δͱʮ͜ͷᮢͲ͏͍͏ࠜڌͰܾΊ͚ͨͬʁʯ ͱɺج४ͷҙਤ͕Θ͔Βͳ͘ͳΔ͜ͱ 17
Slide 18
Slide 18 text
γϯϓϧ͡Όͳ͍ʛΞϥʔτ (v1) ͷ݁Ռ • ՔಇܭࢉͷͨΊʹμϯλΠϜͱ͍͏֓೦Λಋೖ • μϯλΠϜɺͲΕ͘Β͍αʔϏεΛར༻Ͱ͖ͳ͘ͳ͔ͬͨͷ ίϛϡχέʔγϣϯʹ༗༻ • ҰํͰSLO࣮ͷूܭ͕ෳࡶͩͬͨΓɺΞϥʔτ͕ෳࡶʹ 18
Slide 19
Slide 19 text
SLOͱΞϥʔτͷݟ͠ • SLOɾΞϥʔτ͕ਖ਼͘͠ػೳͯ͠ͳ͍͜ͱʹؾ͘ • ਖ਼͘͠ͳ͍SLOϢʔβʔʹՁΛಧ͚ΒΕͳ͍ • ਖ਼͘͠ͳ͍ΞϥʔτϝϯόʔͷϞνϕʔγϣϯʹؔΘΔ • SLOɾΞϥʔτΛࠜຊతʹݟ͢͜ͱʹ (v2) 19
Slide 20
Slide 20 text
SLOɾΞϥʔτ (v2) 20
Slide 21
Slide 21 text
The Site Reliability Workbook • Site Reliability Engineeringͷଓฤ • ΑΓ࣮ફతͳӡ༻ํ๏ʹϑΥʔΧε • ͜ͷຊΑΓSLOͱΞϥʔτΛݟ͠ 21
Slide 22
Slide 22 text
ͳͥSLO͕ඞཁ͔ʁ • SLOΛܾΊΔ͜ͱͰσʔλυϦϒϯͳίϛϡχέʔγϣϯ͕Ͱ͖Δ • ͜Εো͔൱͔ • ύϑΥʔϚϯενϡʔχϯάͷج४ • λεΫͷ༏ઌ • ৴པੑػೳͰ͋Γސ٬ʹಧ͚ΒΕΔՁ 22 SLO: ։ൃɺSREɺεςʔΫϗϧμʔͰ߹ҙ͢ΔҰ؏ͨ͠ඪ
Slide 23
Slide 23 text
SLO (v2) • SLO (v1) ͷলΛͱʹSLO (v2) Λ࠶ઃఆ • ʮkintoneΛշదʹ͑Δ͔ʯߟ͑ͯج४Λߟ͑Δ • ΤϥʔϨʔτɾϨεϙϯελΠϜΛݸʑʹγϯϓϧͳSLOΛઃఆ 23 • ͋ΔAPIͷɺ1ϲ݄ͷϦΫΤετޭ͕M% • ͋ΔAPIͷɺ1ϲ݄ͷϦΫΤετͷX%͕Ԡ࣌ؒYඵະຬ
Slide 24
Slide 24 text
μογϡϘʔυʹΑΔSLOୡͷՄࢹԽ • Ҏલ (v1) ݄ʹSLOͷ࣮Λܭࢉ͢Δ͚ͩͩͬͨ • ݄ͷ్தͰΓͷΤϥʔόδΣοτ֬ೝͰ͖ͳ͔ͬͨ • ৽͍͠SLOݱࡏͷୡΛμογϡϘʔυ্ʹՄࢹԽ 24
Slide 25
Slide 25 text
SLOμογϡϘʔυͷڞ༗ • ΈΜͳ͕ݟ͑ΔॴʹݱࡏͷSLO࣮Λදࣔ 25
Slide 26
Slide 26 text
SLOͱΞϥʔτ • ͳͥγεςϜʹΞϥʔτ͕ඞཁ͔ʁ • োΛ͍ͪૣ͘ؾ͖͍ͨ = SLOΛҧ͢Δલʹम෮͍ͨ͠ • SRE߹ҙͨ͠ΤϥʔόδΣοτΛશྗͰकΔ͖ • ΤϥʔόδΣοτΛ͍ΔલʹোΛݕͰ͖ΔΞϥʔτ͕ඞཁ 26
Slide 27
Slide 27 text
όʔϯϨʔτʹجͮ͘ΞϥʔτʛΞϥʔτ (v2) • όʔϯϨʔτ1 = SLOͷظؒͰΤϥʔόδΣοτΛ͍Δ • ΤϥʔόδΣοτΛ͍ΔͷোΛؾ͘ΞϥʔτΛֻ͚Δ 27 ͨͱ͑30 όʔϯϨʔτ: ΤϥʔόδΣοτͷফඅΛද͢ࢦඪ
Slide 28
Slide 28 text
όʔϯϨʔτͱΤϥʔόδΣοτͷؔ 28 όʔϯϨʔτ2 15ͰΤϥʔόδΣοτΛ ͍Δ όʔϯϨʔτ10 3ͰΤϥʔόδΣοτΛ ͍Δ ΤϥʔόδΣοτফඅྔ 0% 20% 40% 60% 80% 100% ܦա 0 5 10 15 20 25 30 όʔϯϨʔτ1 30ͰΤϥʔόδΣοτΛ ͍Δ
Slide 29
Slide 29 text
• SRWBʹҎԼͷ2ͭͷόʔϯϨʔτ͔Βελʔτ͢ΔͱΑ͍ͱ͋Δ • SLO͕1ϲ݄ϦΫΤετޭ99.9%ͷͱ͖ͷΞϥʔτͷྫ όʔϯϨʔτʹجͮ͘Ξϥʔτͷᮢ 29 ΤϥʔόδΣοτ ফඅྔ Οϯυ όʔϯϨʔτ 2% 1H 14.4 5% 6H 6 APIͷ1࣌ؒͷΤϥʔϨʔτ > (14.4 * 0.001) APIͷ6࣌ؒͷΤϥʔϨʔτ > (6 * 0.001)
Slide 30
Slide 30 text
SLOɾΞϥʔτ (v2) ͷ݁Ռ • ΤϥʔόδΣοτΛ͍Βͣʹ͙͢ʹऩଋ͢ΔোʢGCʹΑΔ ԆͳͲʣͰؒى͜͞ΕΔ͜ͱ͕ݮͬͨ • μογϡϘʔυʹΑΓνʔϜͰSLOͷҙ͕ࣝߴ·ΓɺΞϥʔτ ͱͷؔੑΘ͔Γ͘͢ͳͬͨ • γϯϓϧͳSLOͰෳࡶͳ࣮ίϛϡχέʔγϣϯίετ͕ݮͬͨ 30
Slide 31
Slide 31 text
Lesson Learned SLOɾΞϥʔτվળ͔Βͷֶͼ
Slide 32
Slide 32 text
୭ͷͨΊͷSLO͔ • SLOΛୡ͢Δ͜ͱͰͤʹͳΔਓʢϢʔβʔʣ͕͍Δͣ • ӡ༻ͷಓ۩͚ͩͰͳ͘ɺͦͷഎܠʹαʔϏεͷར༻ऀ͕͍Δ • ϢʔβϏϦςΟ৴པੑΛߟྀͯ͠SLOΛݟͦ͏ 32
Slide 33
Slide 33 text
γϯϓϧͰ͋Ε • ෳࡶͳSLOίϛϡχέʔγϣϯͰোͱͳΔ • ޱ಄Ͱਓʹઆ໌Ͱ͖Δ͘Β͍ͷγϯϓϧ͕͞େࣄ • ෛ࠴ʹͳΔલʹSLOΛৼΓସ͑ΕΔΑ͏ʹ͠Α͏ 33
Slide 34
Slide 34 text
ΈΜͳ͕ݟ͑ΔॴʹՄࢹԽ • ݟ͑Δॴಋઢ͕ͳ͍ͱ୭ҙࣝ͠ͳ͍ • ՄࢹԽʹΑΓɺϝϯόʔશһ͕SLOΛҙࣝ͢Δ͜ͱ͕૿͑ͨ • ݱࡏνʔϜͰμογϡϘʔυΛݟΔձΛ2िؒʹ1ճ։࠵ 34
Slide 35
Slide 35 text
σʔλʹجͮ͘ίϛϡχέʔγϣϯ • λεΫͷ༏ઌSLOʹج͍ܾͮͯΊΒΕΔΑ͏ʹ • ։ൃɺSREɺεςʔΫϗϧμʔؒͰҰ؏ͨ͠ඪΛઃఆͰ͖Δ • νϡʔχϯάΛ͢Δͱ͖ɺצʹཔΒͣσʔλͰஅ͠Α͏ 35
Slide 36
Slide 36 text
͜Ε͔ΒͷνϟϨϯδ • kintoneͷ։ൃνʔϜͷSLOΛΊΔ׆ಈ • SLOະୡͳͷʹΞʔΩςΫνϟతʹνϡʔχϯά͕͍͠έʔε • Ϣʔβʔͷ͍ํ࣍ୈͰ༻ҙʹSLOΛഁΕΔέʔε 36
Slide 37
Slide 37 text
·ͱΊ • SLOʹΑͬͯҰ؏ͨ͠σʔλυϦϒϯͳίϛϡχέʔγϣϯ͕Ͱ͖Δ • ఆظతSLOɾΞϥʔτͷվળͰͷֶͼ͕͋ͬͨ • ୭ͷͨΊͷSLO͔ʁ • γϯϓϧͰ͋Ε • ΈΜͳ͕ݟ͑ΔॴͰՄࢹԽ • σʔλʹجͮ͘ίϛϡχέʔγϣϯ • ͜Ε͔ΒSLOΛ͍͍ͬͯͧ͘💪 37