Share
Embed
サービスレベル目標(SLO)を軸にした異常の原因探索のための研究の調査 / SLO-based Causality Discovery in Distributed Applications
Yuuki Tsubouchi (yuuk1)
July 07, 2020
Transcript
20 slides
-
લճͷ͓͞Β͍ 2 ࢄΞϓϦέʔγϣϯͷҟৗͷਝͳҼՌ୳ࡧख๏ ɾഎܠ: ࢄΞϓϦέʔγϣϯͷෳࡶԽʹΑΓɼҟৗͷݪҼಛఆ͕ࠔ ͱͳ͍ͬͯΔɽ ɾ: αʔϏεϨϕϧͷԼΛܖػͱͯ࣌͠ܥྻϝτϦοΫ͔ΒҼՌ άϥϑΛߏங͢Δख๏͕͋Δɽ͔͠͠ɼ୳ࡧ͢ΔϝτϦοΫͷछྨΛ ཧऀ͕༧Ίબ͢Δඞཁ͕͋Δɽ
ɾఏҊ: શͯͷϝτϦοΫܥྻΛ୳ࡧ͢ΔલఏͰɼʓʓͷख๏ʹΑΓߴ ʹ୳ࡧ͢Δɽ ɾGPGPUɼաڈͷ୳ࡧ݁Ռͷ࠶ར༻ɼγεςϜมߋཤྺͷ׆༻ɼෳ ϗετͷݕग़ͱআ֎ -
5 ઈରʹམͪͳ͍Α͏ʹ͢Δैདྷͷӡ༻ཧ *1 David Oppenheimer, Archana Ganapathi, and David A.
Patterson: “Why Do Internet Services Fail, and What Can Be Done About It?, USENIX Symposium on Internet Technologies and Systems (USITS), 2003. ɾैདྷͷӡ༻ཧͰɼγεςϜͷΤϥʔΛθϩʹ͢Δ͜ͱΛࢦ͢ ɾγεςϜোͷݪҼͱͯ͠࠷ଟ͍ͷཧऀʹΑΔઃఆΤϥʔͰ͋ ΓɼϋʔυΣΞͷϑΥʔϧτʹΑΔͷ10~25%ʹա͗ͳ͍*1 ɾཧऀγεςϜʹมߋΛՃ͑ͳ͍͜ͱʹΑΓɼΤϥʔΛൃੜͤ͞ ͳ͍Α͏ʹ͢Δ ɾͦͷ݁ՌɼϋʔυΣΞނো͕ߴ·ΔɼαʔϏεͷػೳՃ͕ Δɼ͋Δ͍ιϑτΣΞͷ੬ऑੑ͕Δͱ͍͕ͬͨى͖Δ -
৴པੑͱ γεςϜ͕ٻΊΒΕΔػೳΛɼఆΊΒΕͨ݅ͷԼͰఆ ΊΒΕͨظؒʹΘͨΓোΛى͜͢͜ͱͳ࣮͘ߦ͢Δ֬ ※2 6 ɾαʔϏεϨϕϧࢦඪʢService Level Indicator, SLIʣ ɾαʔϏεϨϕϧඪʢService
Level Objective, SLOʣ 100%ͷ৴པੑΛࢦ͞ͳ͍ *2 P. O’Connor, A. Kleyner. Practical Reliability Engineering, 5th edition, Wiley 2012. Ϋϥυ্Ͱల։͍ͯ͠ΔଟͷαʔϏεࣄۀऀ͕SLIɾSLOʹΑΓ ৴པੑΛఆྔతʹܭଌ͠ɼ݁ՌΛҙࢥܾఆʹར༻͍ͯ͠Δ ৴པੑͷࢦඪͱͦͷඪΛܾఆ͠ɼܭଌظؒதʹඪΛԼճΒͳ ͍ݶΓɼαʔϏεࣄۀऀੵۃతʹγεςϜΛมߋͰ͖Δ ※SLAʢService Level AgreementʣϏδωε্ͷܖͰ͋Γɺ Ϣʔβʔͷෆຬʹର͢ΔิঈͳͲؚ͕·ΕΔ -
7 γεςϜͷछྨ ϦΫΤετۦಈ ύΠϓϥΠϯ ετϨʔδ Ϣʔβʔ͕ԿΒ͔ͷछྨͷΠϕ ϯτΛ࡞͠ɺϨεϙϯε͕ฦ Δ͜ͱΛظ͢ΔγεςϜ *3 Betsy
Beyer et. al., The Site Reliability Workbook: Practical Ways to Implement SRE, O'Reilly Media, Inc. 2018. *2 ϨίʔυΛೖྗͱͯ͠औΓɺ ͦΕΒΛมԽͤ͞ɺͲ͔͜ผ ͷॴʹग़ྗ͢ΔγεςϜ σʔλ(όΠτྻɺϨίʔυɺ ϑΝΠϧͳͲ)Λड͚औΓɺͦ ΕΛޙʹऔΓग़ͤΔΑ͏ʹ͠ ͓ͯ͘γεςϜ ※2 Figure 2-1. Architecture for an example mobile phone game ΑΓҾ༻ αʔϏεྫ -
8 SLIͷఆٛͱछྨͷྫ *3 Table 2-1. Potential SLIs for different types
of components ΑΓҰ෦ൈਮ γεςϜͷछྨ SLIͷछྨ આ໌ ϦΫΤετۦಈ Մ༻ੑ Ϩεϙϯεʹޭͨ͠ϦΫΤετͷൺ ϦΫΤετۦಈ ϨΠςϯγ ᮢΑΓߴʹॲཧ͞ΕͨϦΫΤετͷൺ ύΠϓϥΠϯ σʔλͷ৽͞ ࣌ؒͷᮢΑΓ࠷ۙʹߋ৽͞Εͨσʔλͷൺ ύΠϓϥΠϯ σʔλͷਖ਼֬ੑ ਖ਼͍͠ͷग़ྗʹͭͳ͕ͬͨύΠϓϥΠϯͷೖ ྗϨίʔυͷൺ ετϨʔδ ٱੑ ॻ͖ࠐ·ΕͨϨίʔυͷ͏ͪɺਖ਼͘͠ಡΈग़ͤΔ ͷͷൺ SLI = [ good events / valid events × 100 ] *3 Betsy Beyer et. al., The Site Reliability Workbook: Practical Ways to Implement SRE, O'Reilly Media, Inc. 2018. -
9 SLIɾSLOͷઃఆྫ ྨ SLI SLO Մ༻ੑ ϩʔυόϥϯαͷϝτϦοΫ͔Βܭଌͨ͠ɼޭ ͨ͠ϦΫΤετͷൺ 97%ͷޭ ϨΠςϯγ
ϩʔυόϥϯαͷϝτϦοΫ͔Βܭଌͨ͠ɼे ʹߴͳϦΫΤετͷൺ 90%ͷϦΫΤετ͕400ms ҎԼ 99%ͷϦΫΤετ͕850ms ҎԼ σʔλͷ ৽͞ Ϧʔάςʔϒϧ͔ΒಡΈऔΒΕͨϨίʔυͷ͏ ͪɼۙʹߋ৽͞Εͨͷͷൺ ʮۙʯ1Ҏɼ͋Δ͍10Ҏͱఆٛ 90%ͷಡΈऔΓ͕1Ҏʹ ॻ͖ࠐ·ΕͨσʔλΛ༻ *3 Appendix A: Example of SLO DocumentΑΓҰ෦ൈਮ *3 Betsy Beyer et. al., The Site Reliability Workbook: Practical Ways to Implement SRE, O'Reilly Media, Inc. 2018. -
11 ͳ༧ࢉফඅͷ߹ SLOʹجͮ͘Ξϥʔτ ༧ࢉྔ(%) 0 25 50 75 100
0 2 4 6 8 10 ٸͳ༧ࢉফඅͷ߹ Τϥʔ༧ࢉ = 1 - SLOඪ ͱ͢Δ Τϥʔ༧ࢉফඅ͕ٸͳ߹ʹਝʹରԠ͕ඞཁͱͳΓɼͳ߹ ୯ҐͰରԠ͢Δ ༧ࢉྔ(%) 0 25 50 75 100 0 2 4 6 8 10 -
14 ঢ়ͱݪҼ ɾঢ়ɿԿ͕͜ΘΕͨͷ͔ ɾSLIঢ়Λදݱ͢Δ ɾݪҼɿͳͥͦΕ͕͜ΘΕͨͷ͔ ɾڀ໌࣌ʹҟৗʹؔ࿈ͷਂ͍ใΛ୳͢ɽϝτϦοΫɼϩάɼઃఆ ϑΝΠϧɼιʔείʔυͳͲɽ ঢ় ݪҼ HTTP
500͘͠400͕ฦ͞Ε͍ͯΔ σʔλϕʔεαʔό͕ଓΛڋ൱͍ͯ͠Δ ϨεϙϯεͷԼ bogosort Ͱ CPU ʹաେͳෛՙ͕͔͔͍ͬͯ Δɺ͋Δ͍Πʔαωοτέʔϒϧ͕ϥοΫͷԼ ʹڬ·͍ͬͯΔͳͲɻ *4 Table 6-1. Example symptoms and causes ΑΓҰ෦ൈਮ *4 Betsy Beyer et. al., Site Reliability Engineering: How Google Runs Production Systems, O'Reilly Media, Inc. 2016. -
15 ঢ়ͱݪҼͷ૬ରੑ ɾঢ়͔ΒࠜຊݪҼʹͨͲΓͭ͘·ͰʹɼෳͷදతͳݪҼ͕ଘࡏ ͢Δ͜ͱ͕͋Δ Ϩϕϧ ঢ় ݪҼ 1 HTTP 500͘͠400͕ฦ͞Ε͍ͯΔ
σʔλϕʔεαʔό͕ଓΛڋ൱͠ ͍ͯΔ 2 σʔλϕʔεαʔό͕ଓΛڋ൱͍ͯ͠Δ σʔλϕʔεαʔόͷ σΟεΫ༻ྔ͕ຬഋͱͳ͍ͬͯΔ 3 σʔλϕʔεαʔόͷ σΟεΫ༻ྔ͕ຬഋͱͳ͍ͬͯΔ ΫΤϦϩάͷϑΝΠϧαΠζ͕ٸ ʹ૿Ճ 4 ΫΤϦϩάͷϑΝΠϧαΠζ͕ٸʹ૿Ճ … -
17 ҼՌͷൖϞσϧ ωοτϫʔΫ ௨৴Ϟσϧ ΞϓϦέʔγϣϯɼτϥϯεϙʔτɼωο τϫʔΫͷ֤֊ʹ͓͚ΔωοτϫʔΫ௨৴ ʹΑΓҼՌ͕ൖ͢Δ Ϧιʔεಉډ Ϟσϧ ෳͷҟͳΔϓϩηε͕ಉҰͷϦιʔεΛڞ༗
͢Δͱ͖ʹϦιʔεܦ༝ͰҼՌ͕ൖ͢Δɽ αʔόԾԽίϯςφԾԽʹΑΓϗετΛ ಉډ͢Δͱ͖ʹൃੜ͢Δ -
18 ౷ܭతҼՌ୳ࡧʹண ɾҼՌͷൖϞσϧΛϕʔεʹɼٙࣅ૬ؔΛআ֎ͨ͠ҼՌάϥϑΛߏ ங͢Δ ɾ͜͜ͰϚΠΫϩαʔϏεͷจ຺Ͱෳͷؔ࿈ݚڀ*5,*6,*7͕ใࠂ ͞Ε͍ͯΔ ɾγεςϜͷछྨɼݪҼΛࣔ͢σʔλιʔεɼ૬ରੑɼڑɼҼՌ ൖϞσϧͷ֤؍ͱ࣮ߦɼϦιʔεফඅྔɼਫ਼ͳͲͷཁ݅Λ ౿·͑ͯɼະղܾͷΛ୳Δ *5
Ma, Meng, et al. AutoMAP: Diagnose Your Microservice-based Web Applications Automatically, Web Conference. pp. 246-258, 2020. *6 Lin, JinJin, Chen, Pengfei, Zheng, Zibin, Microscope: Pinpoint performance issues with causal graphs in micro-service environments, International Conference on Service-Oriented Computing, pp.3-20, 2018. *7 Qiu, Juan, et al, A Causality Mining and Knowledge Graph Based Method of Root Cause Diagnosis for Performance Anomaly in Cloud Applications, Applied Sciences, 10.6: 2166, 2020.