Share
Embed
Transcript
42 slides
-
2 ࣗݾհ Yuuki Tsubouchi / Ώ͏͏͖ https://yuuk.io/ ܦྺ גࣜձࣾͯͳ WebΦϖϨʔγϣϯΤϯδχΞɾSRE
͘͞ΒΠϯλʔωοτݚڀॴ ݚڀһ ژେֶใֶݚڀՊ ത࢜ޙظ՝ఔ WebαʔϏεͷ ։ൃɾӡ༻ SREͷݚڀ 5 ݱࡏ Site Reliability Engineering(SRE) Researcher @yuuk1t ใॲཧֶձ Πϯλʔωοτͱӡ༻ٕज़ݚڀձ ӡӦҕһ -
7 Ironies of Automation (1983) 1. ࣗಈԽʹΑΓਓؒͷ࡞ۀෛ୲ΛݮͰ͖Δ 2. ͔͠͠ɼࣗಈԽ͢Ε͢Δ΄Ͳਓؒͷೝෛՙ͕ߴ·Δ 3.
ೝෛՙʹ͑ΒΕΔΑ͏ʹߴͳ܇࿅͕ඞཁͱͳΔ • L. Bainbridge, "Ironies of automation,” Analysis, design and evaluation of man–machine systems, pp.129-135 1983. • G. Baxter, et al. "The ironies of automation: still going strong at 30?,” the 30th European Conference on Cognitive Ergonomics 2012. • B. Strauch, "Ironies of automation: Still unresolved after all these years," IEEE Transactions on Human-Machine Systems, vol. 48, no. 5, pp. 419-433 2017. • J. Paul Reed, “When /bin/sh Attacks: Revisiting "Automate All the Things”,” USENIX LISA19 2019. • Tanner Lund, “Ironies of Automation: A Comedy in Three Parts,”, USENIX SREcon19 Asia/Pacific 2019. શͯΛࣗಈԽ͠Α͏ͱͯ͠͏·͍͔͘ͳ͍ ෳࡶ͞Λࣺͯͯɺγϯϓϧʹͨ͠ͱͯ͠ɺͰ͖ͳ͍͜ͱͰ͖ͳ͍·· ෳࡶ͞Λड༰ͯ͠ɺ͍͔ʹਓؒͷೝෛՙΛԼͤ͞Δ͔ ࣗಈԽͷൽ -
8 Ironies Of Automationʹର͢ΔେલఏΞϓϩʔν Ironies of Automationʹର͢ΔΞϓϩʔν ɾ͍͔ʹෳࡶͳγεςϜʹରͯ͠ɺೝΛ֫ಘ͍͔ͯ͘͠ ɾೝෆͷঢ়ଶ =>
ࣦഊ͕ා͍ͷͰγεςϜมߋͷͨΊʹௐࠪʹ࣌ؒ Λ͔͚Δ => ෳࡶ͕͞૿͍ͯ͘͠΄Ͳ͕͔͔࣌ؒΔ => ೝ͕͍͔ͭ ͳ͍ Site Reliability Engineering (SRE) 1. ࣦഊΛڐ༰͢ΔલఏͰӡ༻ΛઃܭΛ͢Δ 2. ਓؒͷೝͦͷͷΛίϯϐϡʔλʹΦϑϩʔυ͢Δ AIٕज़ ※௶༎थ, ΫϥυܥͷࠃࡍձٞIEEE CLOUD 2020ࢀՃ, https://blog.yuuk.io/entry/2020/ieeecloud2020 ※ -
৴པੑͱ γεςϜ͕ٻΊΒΕΔػೳΛɼఆΊΒΕͨ݅ͷԼͰఆ ΊΒΕͨظؒʹΘͨΓোΛى͜͢͜ͱͳ࣮͘ߦ͢Δ֬ ※2 9 ɾαʔϏεϨϕϧࢦඪʢService Level Indicator, SLIʣ ɾαʔϏεϨϕϧඪʢService
Level Objective, SLOʣ SRE: 100%ͷ৴པੑΛࢦ͞ͳ͍ *2 P. O’Connor, A. Kleyner. Practical Reliability Engineering, 5th edition, Wiley 2012. Ϋϥυ্Ͱల։͍ͯ͠ΔଟͷαʔϏεࣄۀऀ͕SLIɾSLOʹΑΓ ৴པੑΛఆྔతʹܭଌ͠ɼ݁ՌΛҙࢥܾఆʹར༻͍ͯ͠Δ ৴པੑͷࢦඪͱͦͷඪΛܾఆ͠ɼܭଌظؒதʹඪΛԼճΒͳ ͍ݶΓɼαʔϏεࣄۀऀੵۃతʹγεςϜΛมߋͰ͖Δ ※SLAʢService Level AgreementʣϏδωε্ͷܖͰ͋Γɺ Ϣʔβʔͷෆຬʹର͢ΔิঈͳͲؚ͕·ΕΔ -
10 1. ࣦഊͷڐ༰ => ҙਤతʹނোΛೖ SREΛલఏʹAIΛద༻͢Δ Chaos Engineering 2. SLIɾSLOΛϕʔεʹAIͰো͔Βճ෮ΛࣗಈԽ
ɾChaosͷఆٛʮγεςϜͷऑΛ໌Β͔ʹ͢ΔͨΊͷ࣮ݧͷଅਐʯ ɾʮఆৗঢ়ଶʯΛܾΊͯɺԾઆݕূͷϑϨʔϜϫʔΫΛར༻ ɾγεςϜͷऑʹର͢ΔೝΛଅਐͤ͞Δ ɾ࠷ۙɺऔΓΜͰ͍Δͷͪ͜Β -
12 ɾػցֶशɺσʔλΛར༻ֶͯ͠श͢Δ ɾ͔͠͠ɺҟৗൃੜ࣌ͷσʔλݱ࣮ʹ͋·ΓೖखͰ͖ͳ͍… ɾChaos EngineeringʹΑΓɺҙਤతʹҟৗΛൃੜͤ͞ɺֶशʹར༻ ՄೳͳσʔλΛੜͤ͞Δ ͞ΒͳΔల: Chaos Engineering x
AI ɾ੍ޚཧͰಈతγεςϜϞσϧͷύϥϝʔλΛܾఆ͢ΔͨΊʹɺ ࣮ݧʹಈ࡞ͤ͞ΔϑΣʔζ͕͋ΔʢγεςϜಉఆʣ ɾΫϥυͷγεςϜͰ੍ޚϞσϧΛઃఆͯ͠ɺγεςϜಉఆ Chaos EngineneringͰߦ͏ -
18 ੑೳҟৗΛஅ͢ΔͨΊͷطଘͷΞϓϩʔν ϝτϦοΫ ςΩετϩά ࣮ߦτϨʔε ๛ͳใΛ͕ͭϩάʹग़ྗ͞Εͳ͍ͷ ͋Δ ॲཧܦ࿏ͷล୯Ґͷεϧʔϓοτ࣮ߦ࣌ؒΛ ѲͰ͖ΔɻΞϓϦέʔγϣϯʹܭଌॲཧΛઃ ఆ͢Δख͕ؒ͋Δ
ݸʑͷใྔগͳ͍͕ऩूɺอଘɺՄࢹԽ͠ ͍͢ɻ ࣮ڥͷద༻ੑΛ౿·͑ͯɺʮϝτϦοΫʯʹண -
19 ϝτϦοΫϕʔεΞϓϩʔν ֤αʔϏεͷܥྻάϥϑ͔Β૬ؔΛൃݟͰ͖Δ͕ɺݪҼՕॴ͕ෆ໌ ʮ౷ܭతҼՌ୳ࡧʯΛԠ༻ͨ͠Ξϓϩʔν͕ۙͰఏҊ͞Ε͍ͯΔ Service A response time Service D
response time Service E response time Service A response time Service D response time Service E response time Service F response time Service C response time ᶃܥྻؒͷҼՌൖάϥϑͷߏங ᶄҼՌͷܦ࿏ͷਪ Ma, M.,et al., AutoMAP: Diagnose Your Microservice-based Web Applications Automatically, WWW2020. Qiu, J.,et al., A Causality Mining and Knowledge Graph Based Method of Root Cause Diagnosis for Performance Anomaly in Cloud Applications, Applied Sciences, 2020. Lin, J.,et al., Microscope: Pinpoint Performance Issues with Causal Graphs in Micro-Service Environments, ICSOC2018. Service A response time Service F response time Service C response time Top-1 Top-2 Service B response time -
ɾஅʹར༻͢ΔϝτϦοΫͷछྨͷ߹ͤݻఆ ʢ1ʙ7ݸఔʣ ɾྫʣԠԆͷΈɺ{ԠԆ, CPUར༻, ϝϞϦ༻ྔ,…} ͳͲ ɾΑΓݪҼʹ͍ۙϝτϦοΫ͕݁Ռ͔Βআ֎͞ΕΔՄೳੑ͕͋Δ 20 ϝτϦοΫϕʔεΞϓϩʔνͷ՝ Ͱ͖ΔݶΓଟ͘ͷϝτϦοΫͷܥྻΛ୳ࡧ͢Δඞཁ͕͋Δ
TCPͷ࠶ૹΤϥʔ͕ൃੜ͍ͯ͠Δ͕ɺ ωοτϫʔΫଳҬͷมԽྔ͕খ͍͞ͳͲ -
21 ੑೳҟৗʹର͢ΔϝτϦοΫͷܥྻͷ࣍ݩݮͷఏҊ త: ϚΠΫϩαʔϏεʹͯҟৗͷܦ࿏ΛࣗಈͰਪ͢ΔͨΊͷج൫ ఏҊ: ҟৗͷݕʹԠͯ͠ɺʮҰ࣌తʹʯஅʹ༗༻ͳܥྻΛશܥྻ͔ Βߴʹநग़͢Δ࣍ݩݮख๏ “TSifter” (Time series
Sifter) ɾᶃਖ਼֬ੑ :அʹ༗༻ͳܥྻ͕ݮ͞Ε͍ͯͳ͍ ɾᶄ࣍ݩݮ: ແ༻ͳܥྻΛͳΔ͘ଟ͘ݮ͍ͨ͠ ɾᶅߴੑ : ਝʹݪҼΛΈ͚͍ͭͨ (ཧ1ఔ) ܥྻʢ=࣍ݩʣ͕૿Ճ͢ΔͱҼՌൖάϥϑ͕ڊେԽ͢Δ 3ͭͷཁ݅ -
22 ࠷ऴతʹ࣮ݱ͍ͨ͠ݪҼஅγεςϜͷશମ૾ શܥྻ औಘ ܥྻͷ ࣍ݩݮ ݪҼஅ ࣌ܥྻ σʔλϕʔε ܥྻͷऩू
ҟৗݕ ఏҊख๏ͷείʔϓ YES Service A/ req_errors Service D/ connections Service E/ ܥྻؒͷҼՌͷܦ࿏ ᶃ ᶄ ᶅ ᶆ ᶇ ʢҼՌάϥϑߏஙʣ αʔϏε୯ҐͰ ࣍ݩݮ -
24 ఏҊख๏ TSifter ͷཁ݅ͱղܾ ᶃਖ਼֬ੑ ᶄ࣍ݩݮ ᶅߴੑ ಎ1 ಎ2 ҟৗൃੜલޙͰ࣌ܥྻͷ͕มԽ͠ͳ͍
ܥྻஅ࣌ʹෆཁ → ࣌ܥྻσʔλͷఆৗੑΛͭܥྻΛআ֎ ࣌ܥྻάϥϑͷܗঢ়͕ࣅ͍ͯΔܥྻ܈ҟ ৗͷஅ࣌ʹ → αʔϏε୯ҐͰ࣌ܥྻͷΫϥελϦϯά ܥྻnʹରͯ͠ΫϥελϦϯάॲཧ , ... → ಎ1ͷআ֎ॲཧ Λઌʹ࣮ߦ͢Δ O(kn) O(n2) O(n) -
25 TSifter: 2ஈ֊ͷ࣍ݩݮख๏ ɾɾɾ ɾɾɾ ɾɾɾ εςοϓ1 ఆৗੑΛͭ ܥྻΛআڈ ੜͷܥྻ
ඇఆৗͳܥྻ ΫϥελԽ͞Εͨܥྻ දܥྻ ҟৗظؒ ΫϥελϦϯάޙʹΫϥελ ͷදܥྻΛબ εςοϓ2 ྨࣅͷܗঢ়Λ ͱΔܥྻΛ ΫϥελϦϯά ҟৗൃੜલnͷ ݻఆͷΟϯυ෯ -
27 εςοϓ2: ܥྻؒͷܗঢ়ྨࣅੑʹண αʔϏεͷܥྻ܈ ܗঢ়ͷྨࣅੑΛද͢ڑई shape-based distance (SBD) Λ࠾༻ ʢ࣌ؒ࣠ํʹγϑτɺॎ࣠ʹ৳ॖ͍ͯͯ͠ྨࣅͱΈͳ͢ʣ
Paparrizos, J. and Gravano, L., k-Shape: Efficient and Accurate Clustering of Time Series,(SIGMOD2015) ߴԽͷͨΊɺ1ճͷॲཧͰΫϥελΛܾఆՄೳͳ֊తΫϥελϦϯάΛ࠾༻ ʢ ͕ͩɺ1αʔϏε͋ͨΓͷܥྻ͕খ͍ͨ͞ΊʹͳΒͳ͍ʣ O(n2) αʔϏεͷදܥྻ܈ Ϋϥελ දܥྻͷબ ଞͷܥྻͱͷڑͷ૯͕࠷খͷܥྻ -
29 ࣮ݧڥ ੍ޚαʔό Locust Kubernetes CPUෛՙೖ ωοτϫʔΫԆೖ ϚΠΫϩαʔϏεΫϥελ Front-End Catalogue
Orders Payment Shipping User Carts ղੳαʔό Prometheus ֎෦ෛՙͷ ੜ ܥྻऔಘϞδϡʔϧ stress-ng tc ղੳϞδϡʔϧ ܥྻऩूִؒ: 5ඵ ܥྻͷΟϯυ෯: 30 Intel Xeon 3.10GHz, 8core,32GB ܥྻͷऩूɾอଘ Sock Shop -
30 ϕʔεϥΠϯख๏: Sieve ɾεςοϓ1: ࢄͷখ͍͞ϝτϦοΫΛऔΓআ͘ ɾεςοϓ2: k-ShapeʹΑΔΫϥελϦϯά Thalheim, J., et
al., Sieve: Actionable Insights from Monitored Metrics in Distributed Systems, (Middleware 2017) ߃ৗతʹར༻ՄೳͳγεςϜͷಛΛநग़͢Δ͜ͱ͕తͰ͋Γɺຊ ݚڀͱత͕ҟͳΔ͕ɺҟͳΔతʹԠ༻Ͱ͖ΔՄೳੑ͕͋Δ ࣌ܥྻσʔλͷ࣍ݩݮख๏ Paparrizos, J. and Gravano, L., k-Shape: Efficient and Accurate Clustering of Time Series,(SIGMOD2015) -
33 ᶅߴੑͷධՁ: ֤ॲཧεςοϓͷ࣮ߦ࣌ؒ ɾCPUίΞ4ɺϝτϦοΫ100kͷڥ ɾTSifterϕʔεϥΠϯʹରͯ͠ɺ311ഒߴͱͳͬͨ ɾʢޙड़ͷՃ࣮ݧͰɺ࠷Ͱ270ഒߴʣ εςοϓ1 (sec) ࣄલআڈ εςοϓ2
(sec) ΫϥελϦϯά ߹ܭ࣮ߦ࣌ؒ (sec) TSifter 54.41 8.68 63.09 ϕʔεϥΠϯ 32.33 19590.83 19623.16 -
34 ɾ྆ख๏ͱʹɺCPUίΞ·ͨܥྻʹରͯ͠ɺઢܗʹεέʔϧ ᶅߴੑͷධՁ: εέʔϥϏϦςΟ TSifter ϕʔεϥΠϯ 0 20 40 60
20000 40000 60000 80000 100000 Execution time (sec) Number of metrics Clustering 1.21 2.43 3.81 5.72 8.68 Filtering 10.24 20.28 31.05 42.14 54.41 Total 11.45 22.71 34.86 47.86 63.09 0 5000 10000 15000 20000 20000 40000 60000 80000 100000 Execution time (sec) Number of metrics Clustering 3908.10 7773.00 11710.26 15670.81 19590.83 Filtering 2.88 7.63 13.54 22.91 32.33 Total 3910.98 7780.63 11723.80 15693.72 19623.16 0 200 400 600 800 1000 1200 1400 1 2 3 4 Execution time (sec) Number of CPU cores Clustering 1224.87 613.31 416.55 317.65 Filtering 0.17 0.17 0.17 0.17 Total 1225.04 613.48 416.72 317.82 0 1 2 3 4 1 2 3 4 Execution time (sec) Number of CPU cores Clustering 0.37 0.21 0.20 0.15 Filtering 3.57 1.81 1.26 0.99 Total 3.93 2.02 1.46 1.14 TSifter ϕʔεϥΠϯ -
35 ࣮ߦ࣌ؒ1Ҏ͕ཧͰ͋ΓɺϕʔεϥΠϯख๏ͷ࣮ߦ ࣌ؒ1225ඵʢ20ʣͰ͋ΓɺݱͰͷཁ݅Λຬͨͤͳ͍ ֤ཁ݅ʹର͢ΔධՁͷ·ͱΊ ᶃਖ਼֬ੑ ᶄ࣍ݩ ݮ ᶅߴੑ ࣮ݧͰɺαʔϏεͷछྨނোέʔε͕ݶఆతͳͨΊɺ ՃͷධՁ͕ඞཁ
࣍ݩݮϕʔεϥΠϯख๏͕Θ͔ͣʹ্ճΔ ࠷ऴతʹཁٻ͞ΕΔ࣍ݩݮͷఔࠓޙͷ՝ CPUίΞͱܥྻ͕มԽͯ͠ɺ྆ख๏ͷ࣮ߦ࣌ؒൺಉ -
40 ɾෳࡶԽ͢ΔγεςϜʹରͯ͠ɺIronies of Automationͷڭ܇Λൽ ΓʹɺೝෛՙΛݮͤ͞Δ͜ͱ͕ॏཁͱͳΔ ɾΤϥʔΛڐ༰ => SRE => Chaos
Engineering ɾೝͷࣗಈԽ => AI => ҟৗൃੜ͔Βճ෮·ͰΛAIͰղܾ ɾ࠷ॳͷऔΓΈͱͯ͠ɺ࠷৽ͷݚڀՌͰ͋Δ࣌ܥྻσʔλͷ࣍ݩ ݮख๏Λհͨ͠ ·ͱΊ