AIOps研究録―SREのための システム障害の自動原因診断 / SRE NEXT 2022

SRE NEXT 2022講演。 https://sre-next.dev/2022/schedule/#jp37

Transcript

1. AIOpsݚڀ࿥ʕSREͷͨΊͷ
   

   γεςϜো֐ͷࣗಈݪҼ਍அ
   yuuk1 @͘͞ΒΠϯλʔωοτݚڀॴ
   

   2022/05/15
   SRE NEXT 2022 ONLINE
   

   View Slide

2. 2
   ϓϩϑΟʔϧ
   yuuk1 (Yuuki Tsubouchi)
   ͘͞ΒΠϯλʔωοτݚڀॴɹݚڀһ
   
   
   ژ౎େֶେֶӃ৘ใֶݚڀՊɹത࢜ޙظ՝ఔ3೥
   
   
   TopotalɹςΫϊϩδΞυόΠβʔ
   ৽ଔ͔Β5೥ؒWebΦϖϨʔγϣϯɾSREͷΤϯδχΞ
   https://yuuk.io/
   3೥લΑΓ͘͞ΒΠϯλʔωοτʹస৬͠ɺݚڀ։ൃͷੈք΁
   2೥લʹେֶӃത࢜ޙظ՝ఔʹೖֶ
   SRE NEXT 2020 IN TOKYO جௐߨԋ
   @yuuk1t
   

   View Slide

3. AIOpsͷݚڀ
   ΢ΣϒΦϖϨʔγϣϯɾSRE
   2013
   2018
   2020ʙ
   ӡ༻σʔλͷޮ཰తͳ؍ଌ๏ͷݚڀ
   SRE DIVERSITY
   [Y.Tsubouchi 2021], [Y. Tsubouchi 2022]
   [Y. Tsubouchi 2021]: ௶಺༎थ, ࿬ࡔேਓ, ᖛా݈, দ໦խ޾, খྛོߒ, Ѩ෦ത, দຊ྄հ, HeteroTSDB: ҟछ෼ࢄKVSؒͷࣗಈ֊૚ԽʹΑΔߴੑೳͳ࣌ܥྻσʔ
   λϕʔε, ৘ใॲཧֶձ࿦จࢽ, Vol.62, No.3, pp.818-828, 2021೥3݄.
   [Y. Tsubouchi 2022]: Yuuki Tsubouchi, Masayoshi Furukawa, Ryosuke Matsumoto, Low Overhead TCP/UDP Socket-based Tracing for Discovering Network
   Services Dependencies, Journal of Information Processing, Vol.30, pp.260-268, 2022.
   

   View Slide

4. AIOpsͷ֤ݚڀͷମܥత੔ཧΑΓ͸Ή͠Ζɺσʔλα
   ΠΤϯεͷܦݧ͕ͳ͍தɺݚڀͷݱ৔Ͱૺ۰͖ͯͨ͠
   ໰୊΁ͷࢼߦࡨޡΛ͓࿩͠·͢ɻ
   ιϑτ΢ΣΞ։ൃɾӡ༻ͷݱ৔Ͱ໌೔͔Β࢖͑Δ஌ࣝ΍
   ςΫχοΫ͸ఏڙͰ͖·ͤΜ͕ɺ΄Μͷগ͠ઌͷະདྷΛ
   ײ͡ΔΑ͏ͳͳʹ͔Λ࣋ͪؼ͍͚ͬͯͨͩΕ͹޾͍Ͱ͢ɻ
   

   View Slide

5. 1.
   SREͱAI
   

   View Slide

6. 6
   HAL 9000 ʰ2001 ೥Ӊ஦ͷཱྀʱ
   18ষ SREͷͨΊͷػցֶशೖ໳ ͔ΒͷҾ༻
   ” ͨͬͨࠓɺAE35Ϣχοτͷো֐Λݕग़͠·ͨ͠ɻ
   ࢲ͸72࣌ؒҎ಺ʹ100%ͷ֬཰Ͱػೳఀࢭ͠·͢ɻ”
   ― HAL 9000ɺʰ2001 ೥Ӊ஦ͷཱྀʱ
   “͜ͷөը͕ඳ͘ະདྷΛઌݟͷ໌Λ΋ͬͯߏ૝ͨ͠ͷ͸Ξʔ
   αʔɾCɾΫϥʔΫ(Arthur C. Clarke)ͰɺγεςϜͱϋʔυ΢Σ
   Ξͷো֐ൃੜΛԿ࣌ؒ΋લʹ༧ଌͰ͖Δ׬શࣗಈԽαʔϏεͱ
   AI Λ૊Έ߹Θͤ·ͨ͠ɻHAL 9000 ͸ɺཱࣗͨࣗ͠ݾௐ੔ܕͷ
   ܽ఺͕ͳ͍ػցͱ͍͏ਓྨͷເ(͋Δ͍͸ѱເ)Ͱ͋Γɺਓؒʹ
   Αͬͯఆٛ͞Εͨ໨ඪΛୡ੒͢ΔͨΊʹɺӉ஦ધͷ৐һͱϛο
   γϣϯͷ྆ํʹไ࢓͠·͢ɻ”
   

   View Slide

7. 7
   ɾ1980೥୅ʹ͸ɺωοτϫʔΫ؅ཧʹɺ஌ࣝϕʔεAI΍χϡʔϥϧωοτ
   ϕʔεAIΛԠ༻͢ΔՄೳੑ͕ٞ࿦͞Ε͍ͯΔ
   ৘ใγεςϜͷӡ༻ʹAIΛԠ༻͢ΔىݯΛ୳Δ
   [Cebulka 1989]: Cebulka KD, et al., Applications of arti
   fi
   cial intelligence for meeting network management challenges in the 1990s, IEEE GLOBECOM 1989.
   ɾಛఆͷαʔϏεΛαϙʔτ͢ΔͨΊͷωοτϫʔΫͷॳظઃܭ
   
   
   ɾηϯτϥϧΦϑΟεؒͷઓज़తͳઃඋܭը
   
   
   ɾεΠον͔Βͷϝοηʔδͷ؂ࢹͱ਍அ
   [Notaro 2021]: Notaro P, et al., A Survey of AIOps Methods for Failure Management. ACM TIST, 2021.
   ɾ1990೥୅ॳ಄͔ΒΦϯϥΠϯͷιϑτ΢ΣΞ΍ϋʔυ΢ΣΞͷނো༧஌
   Ϟσϧ͕͍͔ͭ͘ఏҊ͞Ε͍ͯΔɽͦͷଞͷނো๷ࢭํ๏ͳͲ΋ಉ࣌ظ
   [Cebulka 1989]
   [Notaro 2021]
   

   View Slide

8. 8
   ݱ୅ʹ͓͚ΔAIOpsͷߩݙྖҬ
   [Notaro ’20]: Notaro, P, Jorge C, and Michael G. "A Systematic Mapping Study in AIOps.” ICSOC. Springer, Cham, 2020.
   [Notaro ’20]: Fig.2 Taxonomy of AIOps as observed in the identified contributions
   

   ΑΓసࡌ
   ো֐؅ཧʹؔ͢Δݚڀ
   Ϧιʔεͷׂ౰ͳͲͷ
   

   ࠷దԽʹؔ͢Δݚڀ
   

   View Slide

9. 9
   AIOpsͷݚڀྖҬ͝ͱͷ࿦จ਺
   [Notaro ’20]: Notaro, P, Jorge C, and Michael G. "A Systematic Mapping Study in AIOps.” ICSOC. Springer, Cham, 2020.
   ɾAIOpsؔ࿈ͷ࿦จ਺ɿ670
   
   
   ɾ670݅ͷ62.1%͕Failure Managementʢো֐؅ཧʣʹؔ࿈͍ͯ͠Δ
   
   
   ɾো֐༧ଌʢ26.4ˋʣো֐ݕग़ʢ33.7ˋʣݪҼ෼ੳʢ26.7ˋʣ
   ࿦จ਺͸૿Ճ܏޲
   

   View Slide

10. 10
    γεςϜ؂ࢹͷ੒ख़͍ͯ͠Ε͹ɺো֐ͷൃੜ
    ʹؾ͔ͮͳ͍͜ͱ͸ͳ͍ͷͰ͸ͳ͍͔ʁ
    AIOpsͷͲͷྖҬʹऔΓ૊Ή͔ʁ
    ো֐ͷݪҼ͸ͳʹ͔ʁʹ౴͑Δ΄͏͕೉͍͠
    ༧ଌ/༧๷
    ݪҼ਍அ
    ؇࿨
    ࠜຊݪҼ෼ੳ
    ݕ஌
    Failure Managementʢো֐؅ཧʣͷ΄͏͕৴པੑʹ௚݁
    म෮
    AIOpsͰ
    

    ࠷ॳʹ࿈૝
    ?
    

    View Slide

11. 2.
    γεςϜো֐ͷݪҼ਍அΛ
    

    ࣗಈԽ͢ΔͨΊͷߏ૝
    

    View Slide

12. 12
    ΞϥʔτετʔϜ
    γεςϜো֐ݕ஌ޙͷ՝୊
    େྔͷϝτϦΫεͷӾཡ
    CRITICAL: front-end - http_request latency_95
    CRITICAL: user - latency_95
    CRITICAL: user-db_memory_usage
    CRITICAL: user-db_cpu_user_usage
    CRITICAL: orders - jvm_heap_memory_usage
    CRITICAL: orders-db - network_transmit_bytes
    CRITICAL: payment - http_request_error_5xx
    CRITICAL: front-end - http_request_latency_50
    CRITICAL: front-end - http_request_latency_90
    CRITICAL: user-db - cpu_system_usage
    ೝ஌ෛՙ
    

    ૿େ
    

    View Slide

13. 13
    ঱ঢ়ɿͳʹ͕յΕͨͷ͔ʁ
    
    
    ݪҼɿͳͥյΕͨͷ͔ʁ
    ͳͥΞϥʔτετʔϜ͕ൃੜ͢Δͷ͔ʁ
    ঱ঢ়ͱݪҼʹ͸૬ରؔ܎͕͋ΔͨΊɺ۠ผ͢Δ͜ͱ͕࣮͸೉͍͠
    ঱ঢ়ͱݪҼ͕ಉ࣌ʹΞϥʔτ͞ΕΔͨΊ
    }
    ঱ঢ় ݪҼ
    1 HTTP 500΋͘͠͸400͕ฦ͞Ε͍ͯΔ σʔλϕʔεαʔό͕઀ଓΛڋ൱͍ͯ͠Δ
    2 σʔλϕʔεαʔό͕઀ଓΛڋ൱͍ͯ͠Δ
    σʔλϕʔεαʔόͷ
    
    
    σΟεΫ࢖༻ྔ͕ຬഋͱͳ͍ͬͯΔ
    3
    σʔλϕʔεαʔόͷ
    
    
    σΟεΫ࢖༻ྔ͕ຬഋͱͳ͍ͬͯΔ
    ΫΤϦϩάͷϑΝΠϧαΠζ͕ٸ଎ʹ૿Ճ
    4 ΫΤϦϩάͷϑΝΠϧαΠζ͕ٸ଎ʹ૿Ճ …
    ঱ঢ়ͷΈΛΞϥʔτ͢Ε͹Α͍ͷͰ͸ʁ
    

    View Slide

14. 14
    “Alert symptoms, diagnose causes”
    SLOʹجͮ͘ΞϥʔςΟϯά
    [SRWbook 18] Chapter 5 "Alerting on SLOs", Beyer B, et al., The Site Reliability Workbook: Practical ways to implement SRE. O'Reilly Media, Inc."; 2018.
    ※ [SRWbook 18]
    Four Golden Signals / RED
    Latency, Traffic, Errors, Saturation
    Rate, Errors, Duration
    αʔϏεશମͷ঱ঢ়ʹର͢ΔΞϥʔτ AIʹΑΔݪҼ਍அ
    τϦΨʔ
    ӡ༻σʔλͷࣗಈղੳ
    ӡ༻σʔλ
    ϝτϦΫε
    
    
    ϩά
    
    
    τϨʔε
    
    
    Πϕϯτ
    SREs
    

    View Slide

15. 15
    ྨࣅͷண૝Λ΋ͭઌਓ͸͍Δ
    Chen P, et al., Causeinfer: Automatic and distributed performance diagnosis with hierarchical causality graph in large distributed systems. IEEE INFOCOM 2014.
    CauseInfer (2014)
    ɾίϯϙʔωϯτ͝ͱʹͷύέοτͷண৴࣌
    ࠁ͔Βਪఆ͢ΔTCP஗ԆΛܭଌ
    
    
    ɾ౷ܭతҼՌ୳ࡧ෼໺ͷPCΞϧΰϦζϜʢޙ
    ड़ʣʹΑΓҼՌάϥϑੜ੒
    SLOϝτϦΫεͷҧ൓ݕ஌ΛτϦΨʔͱͯ͠ҼՌਪ࿦
    CauseInfer (2014) Fig. 2.ΑΓҰ෦సࡌ
    Lin J, et al., Microscope: Pinpoint performance issues with causal graphs in micro-service environments. ICSOC, 2018.
    Microscope (2018) CauseInferʹରͯ͠ɺඇ௨৴ؔ܎ͷґଘΛߟྀ
    ɾCaudeInferಉ༷ʹҼՌάϥϑΛੜ੒
    ɾϚΠΫϩαʔϏεͰ͸ɺ֤αʔϏεͰ
    Ԡ౴͕࣌ؒܭଌ͞ΕΔ
    Microscope (2018) Fig. 2. ΑΓసࡌ
    

    View Slide

16. ҼՌάϥϑͷܦ࿏ͷϥϯΩϯάग़ྗ
    
    
    ɾAutoMAP (2020): ϚΠΫϩαʔϏε୯ҐͰԠ౴͚࣌ؒͩͰͳ͘7छྨͷϝ
    τϦΫε͔ΒಘͨݸผͷҼՌάϥϑΛ߹੒
    
    
    ɾFluxInfer (2020): PCΞϧΰϦζϜͰ͸ͳ͘ɺॏΈ෇͖ແ޲ґଘάϥϑ +
    ϖʔδϥϯΫ
    
    
    ɾMicroCause (2020): PCΞϧΰϦζϜͷ࣌ܥྻͷϥάΛߟྀ͢Δվྑ
    16
    ྨࣅͷண૝Λ΋ͭઌਓ͸͍Δ ʔ ൃలฤ
    ݪҼϝτϦΫεͷϥϯΩϯάग़ྗ
    
    
    ɾPatternMatcher (2021): ϝτϦΫεͷҟৗύλʔϯΛCNNͰ෼ྨ
    
    
    ɾFluxRank (2019): ྨࣅͷ࣌ܥྻΛΫϥελϦϯάޙʹϩδεςΟοΫճؼ
    ͰϥϯΩϯά
    

    View Slide

17. 17
    ݪҼ਍அ࿦จͷϦετ https://github.com/dreamhomes/RCAPapers
    ͞ΒͳΔઌߦݚڀͨͪ
    Notaro P, Cardoso J, Gerndt M. A Survey of AIOps Methods for Failure Management. ACM
    Transactions on Intelligent Systems and Technology (TIST). 2021 Nov 30;12(6):1-45.
    Lyu Y, Rajbahadur GK, Lin D, Chen B, Jiang ZM. Towards a Consistent Interpretation of AIOps
    Models. ACM Transactions on Software Engineering and Methodology (TOSEM). 2021 Nov
    15;31(1):1-38.
    Soldani J, Brogi A. Anomaly detection and failure root cause analysis in (micro) service-based
    cloud applications: A survey. ACM Computing Surveys (CSUR). 2022 Feb 3;55(3):1-39.
    https://blog.yuuk.io/entry/2020/ieeecloud2020
    https://netman.aiops.org/publications/
    ਗ਼՚େֶ NETMAN LAB
    ࠃࡍձٞ IEEE CLOUD
    AIOpsؔ࿈ͷαʔϕΠ࿦จ
    

    View Slide

18. 18
    ɾ՝୊̍ ೖྗͱͳΔϝτϦΫεͷछྨ
    ΍ݸ਺Λ༧Ίࢦఆ͢Δඞཁ͕͋Δɻ
    
    
    ɾ՝୊̎ ࢦఆͨ͠ม਺ʹରͯ͠ݸผʹ
    ద੾ͳؔ਺ΛબΜͩΓɺνϡʔχϯ
    ά͢Δඞཁ͕͋Δɻ
    
    
    ɾ՝୊̏ ਪఆ݁Ռͷઆ໌ੑͷͨΊʹ͸
    ϝτϦΫεͷΈͰ͸ෆ଍͢Δɻ
    ઌߦݚڀͰͷະղܾͳྖҬΛ୳Δ
    ɾେྔͷϝτϦΫεΛऩूͰ͖Δ
    Α͏ʹͳͬͨ
    
    
    ɾʹ΋ؔΘΒͣɺͦΕΒͷϝτϦ
    ΫεΛ׆༻Ͱ͖͍ͯΔ͔ʁ
    જࡏత໰୊ҙࣝ
    γεςϜͰ؍ଌ͞Εͨશͯͷ
    

    ϝτϦΫεΛೖྗͱͯ͠
    

    ҼՌάϥϑΛੜ੒
    

    View Slide

19. ҟछࠞ߹ϝτϦΫεʹରͯ͠ݸผʹԾఆΛஔ͔ͳ͍
    19
    શϝτϦΫε͔ΒݪҼ਍அʹ͔͚Δͱॲཧ͕࣌ؒ௕͘ͳΔ
    ݚڀͷ໰୊ઃఆɿݪҼ਍அͷલॲཧ
    ো֐
    

    ݕ஌
    ݪҼ਍அ
    ఏҊ
    ࣌ܥྻͷ
    

    ݸ਺Λ࡟ݮ
    ௶಺༎थ΄͔, TSifter: ϚΠΫϩαʔϏεʹ͓͚Δੑೳҟৗͷਝ଎ͳ਍அʹ޲͍ͨ࣌ܥྻσʔλͷ࣍ݩ࡟ݮख๏, Πϯλʔωοτͱӡ༻ٕज़γϯϙδ΢Ϝ࿦จू, 2020೥.
    ௚ۙͷݻఆ෯ͷ
    

    ϝτϦΫεΛऔಘ
    ਺෼୯Ґͷ࣮ߦ࣌ؒ
    ΦϑϥΠϯղੳ
    ϝτϦΫεΛࣄલࢦ
    ఆͤͣʹɺߴ଎ͳݪ
    Ҽ਍அ͕Մೳʹ
    ҼՌάϥϑ
    

    ͷੜ੒
    

    View Slide

20. 3.
    ࣌ܥྻղੳɾҼՌάϥϑੜ੒
    

    ͷͨΊͷࢼߦࡨޡ
    

    View Slide

21. 21
    ΦϖϨʔλʔͷೝ஌ॲཧͷྲྀΕΛτϨʔε͢Δ
    ᶃظؒ಺ͰҟৗΛؚΉ
    

    ɹ࣌ܥྻΛൃݟ
    ᶄ࣌ܥྻάϥϑͷܗঢ়
    ͕ࣅ͍ͯΔ΋ͷΛ
    

    άϧʔϓԽ
    ΫϥελϦϯά
    ΦϑϥΠϯ
    

    ҟৗݕ஌
    ϑΣʔζ̍
    ϑΣʔζ̎
    

    View Slide

22. ϑΣʔζ̍
    ΦϑϥΠϯ
    
    
    ҟৗݕ஌
    ϑΣʔζ2
    ܗঢ়
    

    ΫϥελϦϯά
    ҼՌάϥϑͷ
    

    ੜ੒
    લॲཧ ݪҼ਍அ
    ҼՌάϥϑͷ
    

    

    View Slide

23. 23
    ϑΣʔζ̍ɿ୯มྔ࣌ܥྻͷҟৗੑʹண໨
    ࣌ܥྻͷҟৗύλʔϯΛ13ύλʔϯʹ෼ྨͨ͠ྫ
    [PatternMatcher 2021]: Wu C, Zhao N, Wang L, Yang X, Li S, Zhang M, Jin X, Wen X, Nie X, Zhang W, Sui K. Identifying Root-Cause Metrics for Incident Diagnosis in Online
    Service Systems.
    [PatternMatcher 2021]
    30෼ఔ౓ͷ୹ظؒͷղੳͰΑ͍ͨΊɺقઅੑ΍ϦϦʔεʹΑΔਖ਼ৗ
    ϞʔυͷมԽΛߟྀ͠ͳͯ͘Α͍ɻ
    ͢Ͱʹো֐ݕग़͞ΕͨޙͳͷͰɺΦϑϥΠϯҟৗݕ஌ͰΑ͍
    

    View Slide

24. [PatternMatcher 2021]: Wu C, Zhao N, Wang L, Yang X, Li S, Zhang M, Jin X, Wen X, Nie X, Zhang W, Sui K. Identifying Root-Cause Metrics for Incident Diagnosis
    in Online Service Systems. 24
    ඪຊXͱඪຊY͕ಉҰͷ฼ूஂͷ෼෍ΑΓੜ͍ͯ͡
    Δ͔Λݕఆ͢Δ
    2ඪຊؒͷ෼෍ͷࠩΛΈΔݕఆɿK-Sݕఆ
    ͏·͍͔͘ͳ͔ͬͨέʔε
    ɾγϣʔτεύΠΫͷΑ͏ͳݦஶͳ֎Ε஋ΛؚΉ
    ࣌ܥྻ
    p஋: 0.11
    p஋: 0.51
    ɾগ਺ͷ֎Ε஋Ͱ͸ɺ෼෍͕ҧ͏ͱΈͳ͞ΕΔ΄
    ͲͰ͸ͳ͍
    ࣌ܥྻΛ௨ৗظؒͱςετظؒʹ2෼ׂ͠ɺظؒؒ
    ͷ෼෍ࠩҟΛݕఆ
    ʢ[PatternMatcher 2021]Ͱ࠾༻͞Ε͍ͯΔʣ
    

    View Slide

25. 25
    ɾ௚؍ɿ࣌ܥྻͷಛੑ͕࣌ؒͷܦաͱͱ΋ʹมԽ͠ͳ͍
    
    
    ɾఆٛɿܥྻͷฏۉ͓Αͼ෼ࢄ͕࣌ؒʹΑΒͣҰఆɼ͔ͭࣗݾڞ෼ࢄ͕࣌ؒ
    ࠩͷΈʹґଘ͢Δੑ࣭
    ࣌ܥྻͷੑ࣭ʮఆৗੑʯͷ͋ͯ͸Ί
    ɾఆৗੑʹ͋ͯ͸·Βͳ͍ܥྻ͸ҟৗͱΈͳ͢
    γϣʔτεύΠΫ
    1֊
    

    ࠩ෼
    ͏·͍͔͘ͳ͔ͬͨέʔε
    ADFݕఆͰݕఆՄೳ
    ɾγϣʔτεύΠΫ͕
    

    ఆৗͱ൑ఆ͞ΕΔ
    ɾADFݕఆ͸ʮࠩ෼ܥྻʯ
    ͕ఆৗੑͷੑ࣭Λຬͨ͢
    ͔ݕఆ
    εύΠΫ͸ฏۉ΁ͱ
    

    ճؼͯ͠͠·͏
    ߨԋͰ͸εΩοϓ
    

    View Slide

26. 26
    ҟৗݕ஌ख๏ͱͯ͠޿͘஌ΒΕ͍ͯΔख๏Λ࢖͏
    ౷ܭతҟৗݕ஌ͷجຊͷεςοϓ
    ֬཰෼෍Ͱਖ਼ৗύλʔϯΛදݱͰ͖ΔͱԾఆ
    1. ෼෍ͷਪఆ
    2. ҟৗ౓ͷࢉग़
    3. ᮢ஋ͷઃఆ
    ҟৗ౓ʹର͢Δᮢ஋ͷઃఆʹΑΓҟৗΛ൑ఆ
    ະ஌ύϥϝʔλΛؚΉ֬཰෼෍Λਖ਼ৗϞσϧͱͯ͠Ծఆ
    
    
    σʔλ͔Βະ஌ύϥϝʔλΛਪఆ
    ਖ਼ৗ͔ΒͷͣΕͷ౓߹͍Λ༧Ίఆٛͯ͠ࢉग़
    ࢀߟɿҪख߶, ೖ໳ ػցֶशʹΑΔҟৗ
    ݕ஌, 7.3અ ίϩφࣾ, 2015..
    

    View Slide

27. 27
    ࣌ܥྻͷ֤఺ͷ෼෍͕ਖ਼ن෼෍ʹै͏ͱԾఆ
    ֎Ε஋ݕग़ɿϗςϦϯάͷ ๏
    T2
    ࢀߟɿҪख߶, ೖ໳ ػցֶशʹΑΔҟৗݕ஌, 7.3અ ίϩφࣾ, 2015..
    ౓਺෼෍΁
    1. ෼෍ਪఆ
    f(x) =
    1
    2πσ2
    exp
    (
    −
    (x − μ)2
    2σ2 )
    ωοτϫʔΫૹ৴όΠτ਺ͷϝτϦΫε
    μ =
    1
    N
    N
    ∑
    i=1
    xi σ2 =
    1
    N
    N
    ∑
    i=1
    (xi
    − μ)2
    2. ҟৗ౓ͷܭࢉ
    ඪຊฏۉ ඪຊ෼ࢄ
    a(x) = (
    x − μ
    σ )
    2
    3. ᮢ஋൑ఆ
    ҟৗ౓ͷ෼෍͸ࣗ༝౓1ͷΧΠೋ৐෼෍ʹै͏
    ᮢ஋Λ֬཰ͰܾఆͰ͖Δʢ0.01, 0.05ͳͲʣ
    ϊΠζʹऑ͍
    ະ஌
    

    ύϥϝʔλ
    

    View Slide

28. 28
    ɾࣗݾճؼϞσϧɿ͋Δ࣌ࠁ t ͷ஋Λɺ࣌ࠁ t Ҏલͷ஋Λ࢖ͬͯճؼ͢ΔϞσϧ
    
    
    ɾ௚؍ɿະདྷ͸աڈͷ஋͔Β༧ଌͰ͖ΔͱԾఆ
    ࣗݾճؼϞσϧʢARϞσϧʣʹΑΔҟৗݕ஌
    ҟৗ౓
    ࣮ଌ஋ʢ੨ʣ༧ଌ஋ʢᒵ৭ʣ
    yn
    =
    r
    ∑
    t=1
    at
    yn−t
    + vn
    ʢ༧ଌ஋ − ؍ଌ஋ = ༧ଌޡࠩʣ= ҟৗ౓
    ࣌ܥྻ ͕༩͑ΒΕͨͱ͖
    y1
    , . . . yN
    : ϥά࣍਺
    r
    : ࣗݾճؼ܎਺
    at
    : ฏۉ0ɺ෼ࢄ ͷ
    

    ਖ਼ن෼෍ʹै͏ϗϫΠτ
    ϊΠζ
    vn
    σ2
    ɾ Λܾఆ͠ɺ࣮ଌ஋͔Β܎਺ ͱ෼ࢄ Λਪఆ͢Δ
    
    
    ɾ ͸੺஑৘ใྔج४ʢAICʣΑΓܾఆ͢Δͷ͕Ұൠత
    
    
    ɾ ͱ ͷۙࣅ஋Λ࠷খೋ৐๏ʹΑΓ࠷໬ਪఆ
    r at
    σ2
    r
    at
    σ2
    ࢀߟɿҪख߶, ೖ໳ ػցֶशʹΑΔҟৗݕ஌, 7.3અ ίϩφࣾ, 2015..
    

    View Slide

29. 29
    ະ஌ύϥϝʔλʢ܎਺ ʣΛ؍ଌσʔλ
    ͢΂ͯʹରͯ͠ਪఆ͢Δ
    at
    ࣗݾճؼϞσϧɿαϯϓϧ಺༧ଌͱαϯϓϧ֎༧ଌ
    αϯϓϧ಺༧ଌͷޡࠩ αϯϓϧ֎༧ଌͷޡࠩ
    ֶशظؒͱݕূظؒʹ෼ׂ͢ΔʢਤͰ͸1:1ʣ
    ̋ݦஶͳ
    

    ֎Ε஋
    ະ஌ύϥϝʔλΛֶशσʔλ͔Βਪఆ
    ͠ɺݕূσʔλͱͷ༧ଌޡࠩΛࢉग़͢Δ
    ̋ϗϫΠτ
    

    ϊΠζ
    ✗Ϩϕϧγϑτʹ
    ա৒ద߹ͯ͠ޡ
    ͕ࠩখ͍͞
    ✗ϊΠζͷ൓ԠʹΑΔ
    

    ޡݕ஌
    
    
    ϗςϦϯάͱಉ͡…?
    ̋Ϩϕϧγϑτ
    

    View Slide

30. 30
    ɾ఺୯Ґͷ֎Ε஋ΛΈΔΑΓɺMSEʢฏۉ༧ଌޡࠩʣͷΑ͏ͳྦྷੵͷޡࠩΛ
    ΈΔҟৗ౓Λ࠾༻͢Δ
    
    
    ɾෳ਺ͷख๏ͷ૊Έ߹Θͤ
    ͜Ε͔ΒͲ͏͢Δ͔ ࿥ը࣌఺Ͱະղܾ
    ୹࣌ؒͷҟৗͱɺͦΕҎ֎ͷҟৗͷ྆ํΛ͏·͘ͱΒ͑Δͷ͕೉͍͠
    [PatternMatcher 2021] Fig. 2ΑΓసࡌ
    

    View Slide

31. ϑΣʔζ̍
    ΦϑϥΠϯ
    
    
    ҟৗݕ஌
    ϑΣʔζ2
    ܗঢ়
    

    ΫϥελϦϯά
    ҼՌάϥϑͷ
    

    ੜ੒
    લॲཧ ݪҼ਍அ
    ҼՌάϥϑͷ
    

    ߨԋͰ͸εΩοϓ
    ϑΣʔζ̎ ܗঢ়ΫϥελϦϯά
    ߨԋͰ͸εΩοϓ
    

    View Slide

32. 32
    ɾػցֶशʹ͓͚ΔλεΫͷ໊শ͸ʮΫϥελϦϯάʯ
    
    
    ɾݸʑͷσʔλ͕Ͳͷఔ౓ྨࣅ͍ͯ͠Δ͔ ʢྨࣅ౓ɾڑ཭ई౓ʣ
    
    
    ɾΫϥελΛͲͷΑ͏ͳखॱͰൃݟ͍͔ͯ͘͠
    ϑΣʔζ̎ɿܗঢ়͕ࣅ͍ͯΔ࣌ܥྻΛάϧʔϓԽ
    ωοτϫʔΫͷૹड৴ଳҬͱύέοτ਺ͷϝτϦΫε͕ΫϥελԽ͞Εͨྫ
    Ϋϥελ୅දΛબ୒
    ॏ৺͔Βͷ࠷ۙ๣ͷ࣌ܥྻ
    ੵ෼஋͕࠷େͷ࣌ܥྻ
    ߨԋͰ͸εΩοϓ
    

    View Slide

33. 33
    ΫϥελϦϯάͷൣғ
    Proxy App DB
    Cluster 1 Cluster 2 ಉҰίϯϙʔωϯτ಺Ͱ
    

    ΫϥελϦϯά
    ɾίϯϙʔωϯτΛ·͍ͨͩϝτϦΫεΛΫϥελϦϯά͢Δͱ
    

    ҼՌάϥϑʹඞཁͳϊʔυ͕ࣦΘΕΔ
    
    
    ɾͨ·ͨ·঱ঢ়ϝτϦΫεͱݪҼϝτϦΫεͷܗঢ়͕ྨࣅ͢ΔέʔεͳͲ
    ߨԋͰ͸εΩοϓ
    

    View Slide

34. 34
    ࣌ܥྻΫϥελϦϯάͷੈք
    [Paparrizos 15]: Paparrizos J, Gravano L. k-Shape: Ef
    fi
    cient and accurate clustering of time series. SIGMOD 2015.
    [Paparrizos 15] Fig. 1. Time-series clustering taxonomy. ΑΓసࡌ
    [Paparrizos 15] Fig. 2. The time-series clustering approaches. ΑΓసࡌ
    ࣌ܥྻશମͷ
    

    ΫϥελϦϯά
    ܗঢ়ʹجͮ͘
    

    Ξϓϩʔν
    ਓ͕ؒ࣌ܥྻάϥϑΛΈͯࣅ͍ͯΔͱ
    

    ൑அ͢Δͷ͸ɺܗঢ়ʹج͍͍ͮͯΔ͸ͣ
    ߨԋͰ͸εΩοϓ
    

    View Slide

35. 35
    ࣌ܥྻΫϥελϦϯάͷܗঢ়ྨࣅੑ
    Paparrizos J, Gravano L. k-Shape: Ef
    fi
    cient and accurate clustering of time series. SIGMOD 2015.
    [k-Shape 2015] Figure 2: Similarity computation ΑΓ
    

    Ұ෦ൈਮͯ͠సࡌ
    ED: ϢʔΫϦουڑ཭
    [k-Shape 2015] Figure1: ΑΓҰ෦ൈਮͯ͠సࡌ
    ॎ࣠ʹରͯ͠৳ॖͨ͠ͱ͖ʹࣅ͍ͯΔ͔ʁ
    ԣ࣠ʹରͯ͠γϑτͨ͠ͱ͖ʹࣅ͍ͯΔ͔ʁ
    
    
    z-scoreม׵ (ฏۉ0,ඪ४ภࠩ1ͱ͢Δม׵)Ͱୡ੒
    ԣ࣠γϑτʹର͢Δߟྀ͕ͳ͍
    Scalingෆมੑ
    Shifttingෆมੑ
    DTW: ಈతۭؒ৳ॖ๏ʢओྲྀʣ
    2ຊͷ࣌ܥྻͷ֤఺ಉ࢜ͷڑ཭Λશͯܭࢉ͢Δͨ
    Ίɺܭࢉྔ͕େ͖͍
    ߨԋͰ͸εΩοϓ
    

    View Slide

36. 36
    SBD (Shape-Based Distance)
    Paparrizos J, Gravano L. k-Shape: Ef
    fi
    cient and accurate clustering of time series. SIGMOD 2015.
    ૬ޓ૬ؔ
    ϕΫτϧͷ಺ੵɿϕΫτϧಉ͕࢜֯౓͕͍ۙͱ͖ʢ ͕0ʹ͍ۙͱ͖ʣେ͖ͳ஋ΛͱΔ
    θ
    x ⋅ y = |x||y|cos θ
    1ຊͷ࣌ܥྻΛ1ݸͷϕΫτϧͱΈͳ͢
    ϕΫτϧΛͣΒ͠ͳ͕Β಺ੵΛܭࢉ͍͖ͯ͠ɺ಺ੵ͕࠷େͱͳΔγϑτ Λൃݟ͢Δ
    w
    CCw
    (x, y) = Rw−m
    (x, y)
    Rk
    (x, y) =
    {
    ∑m−k
    l=1
    xl+k
    ⋅ yl
    R−k
    (y, x)
    x = (x1
    , . . . , xm
    ) y = (y1
    , . . . , ym
    )
    ճ ΛӈʹฏߦҠಈͤͨ͞
    

    ͱ͖ͷ ͱͷྨࣅ౓Λ ͱ͢Δ
    k = w − m y
    x CCw
    (x, y)
    Λ
    

    ʹରͯ͠ܭࢉ
    CCw
    (x, y) w ∈ 1,2,...,2m − 1
    ͷܭࢉྔΛ
    

    ߴ଎ϑʔϦΤม׵Ͱ
    

    ΁
    O(m2)
    O(mlogm)
    ߨԋͰ͸εΩοϓ
    

    View Slide

37. 37
    ΫϥελԽͷखॱͷݕ౼
    ΫϥελϦϯά
    ֊૚తΫϥελϦϯά
    ෼ׂ࠷దԽతΫϥελϦϯά
    ࢀߟɿਆቇ හ߂, ΫϥελϦϯάʢClusteringʣ https://www.kamishima.net/archive/clustering.pdf
    Ϋϥελͷྑ͞Λදؔ͢਺Λఆٛ͠ɼ
    

    ͦͷؔ਺Λ࠷దԽ͢ΔΑ͏ͳ
    

    ΫϥελΛൃݟ
    ڽूܕ ෼ׂܕ
    σʔλҰ͕ͭݸʑͷΫϥελͷ
    

    ঢ়ଶ͔Βɼॱ࣍ΫϥελΛซ߹
    σʔλू߹શମ͕ҰͭͷΫϥελ
    ͷঢ়ଶ͔Βɼॱ࣍ΫϥελΛ෼ׂ
    ܭࢉྔ͕ଟ͍
    ܭࢉྔ͕ଟ͍
    ࣄલʹΫϥελ਺Λ
    

    ܾΊͳ͚Ε͹ͳΒͳ͍
    ༗໊ͳk-means͸
    

    ͪ͜Β
    ܭࢉྔ͕গͳ͍
    ߨԋͰ͸εΩοϓ
    

    View Slide

38. 38
    ֊૚తΫϥελϦϯάͷߴ଎ੑ
    ࠷దͳΫϥελ਺Λܾఆ͢ΔͨΊʹ
    ܁Γฦ࣮͠ߦ
    ֊૚తΫϥελϦϯάʢ࠷୹ڑ཭๏ʣ
    ڑ཭ͷᮢ஋Λࣄલઃఆ
    ͯ͠Ϋϥελ਺Λܾఆ
    ෼ׂ࠷దԽΫϥελϦϯά
    ߨԋͰ͸εΩοϓ
    

    View Slide

39. 39
    ɾܗঢ়͕ࣅ͍ͯΔ࣌ܥྻΛάϧʔϓԽ
    
    
    ɾڑ཭ई౓ͱͯ͠ɺ૬ޓ૬ؔϕʔεͷSBDΛ࠾༻
    
    
    ɾߴ଎ੑͷͨΊɺ֊૚తΫϥελϦϯά+࠷୹ڑ཭๏ͷ࠾༻
    ϑΣʔζ̎ɿΫϥελϦϯά·ͱΊ
    [FluxRank 2019]͸ɺີ౓४ڌΫϥελϦϯά + ϐΞιϯ૬ؔʹΑΔ࣌ܥྻ
    ΫϥελϦϯά͕࠾༻͞Ε͍ͯͨ͜ͱʹؾ͍ͮͨͷͰɺൺֱ͢Δඞཁ͕͋Δ
    ɾݱ࣌఺Ͱ͸՝୊ͳ͠ͷ͸͕ͣ…
    ߨԋͰ͸εΩοϓ
    

    View Slide

40. ϑΣʔζ̍
    ΦϑϥΠϯ
    
    
    ҟৗݕ஌
    ϑΣʔζ2
    ܗঢ়
    

    ΫϥελϦϯά
    લॲཧ ݪҼ਍அ
    ҼՌάϥϑͷ
    

    ੜ੒ͯ֬͠ೝ͍ͨ͠
    ݪҼ਍அɿҼՌάϥϑੜ੒
    ҼՌάϥϑͷ
    

    ੜ੒
    
    

    View Slide

41. 41
    ɾMݸͷ֬཰ม਺
    
    
    ɾ֬཰ม਺͝ͱʹNݸͷඪຊ
    ҼՌ୳ࡧΛϝτϦΫεͷݪҼ਍அʹԠ༻
    DAGʢ༗޲ඇ८ճάϥϑʣ
    ग़ྗ
    front-end:latency user:latency
    ɾ
    

    ɾ
    

    ɾ
    Mݸͷ
    

    ࣌ܥྻ
    Nݸͷඪຊ
    orders:latency
    user:cpu_usage
    user-db:cpu_usage
    orders:network_transmit_bytes
    ঱ঢ়ϝτϦΫε
    orders-db:network_receive_bytes
    

    View Slide

42. 42
    ౷ܭతҼՌ୳ࡧͷੈք
    ؍ଌσʔλ͔ΒҼՌؔ܎Λਪఆ͢Δ
    ੍໿ϕʔε είΞϕʔε
    Glymour C, et al., Review of causal discovery methods based on
    

    graphical models. Frontiers in genetics. 2019
    ߏ଄ํఔࣜϕʔε
    ϊϯύϥϝτϦοΫʢԾఆΛ͓͔ͳ͍ʣ
    ύϥϝτϦοΫɾ
    

    ηϛύϥϝτϦοΫ
    PCΞϧΰϦζϜ (1991)
    ม਺ؒͷ৚݅෇͖ಠཱੑΛ੍໿
    

    ͱͯ͠ҼՌάϥϑΛߏங͢Δ
    FCIΞϧΰϦζϜ (1995)
    ະ؍ଌڞ௨ݪҼΛߟྀ
    ಉ͡৚݅෇͖ಠཱੑΛ༩
    ͑ΔҼՌάϥϑͷू߹Ͱ
    ͋ΔϚϧίϑಉ஋ྨ͝ͱ
    ʹɼϞσϧͷΑ͞ΛධՁ
    GESΞϧΰϦζϜ (2003) LiNGAM (2006)
    ݪҼΛXΛ݁ՌΛYͱͯ͠
    ߏ଄ΛํఔࣜͰදݱ
    ؔ਺ܥ΍ޡࠩม਺ͷ෼෍
    ʹԾఆΛஔ͘
    

    View Slide

43. 43
    1. ॳظԽɿ Mݸͷ֬཰ม਺ͷ׬શ࿈݁άϥϑΛߏங͢Δ
    PCΞϧΰϦζϜ
    2. Τοδ࡟আɿ ྡ઀͢Δ֤ม਺ʹ͍ͭͯɺ৚݅෇͖ಠཱੑ͕ଘࡏ͢Ε͹ɺ2
    ม਺ؒͷΤοδΛ࡟আɻ
    3. ϧʔϧϕʔεͷํ޲ܾఆʢv-structureɺΦϦΤϯςʔγϣϯϧʔϧʣ
    ಠཱੑɿม਺ؒͷ૬ؔΛΈΔͳͲͷख๏Ͱಠཱ͔Ͳ͏͔Λਪఆɻ
    
    
    ৚݅෇͖ಠཱੑɿ͋Δม਺Λ৚݅ͱͨ͠ͱ͖ͷଞͷ2ม਺ͷಠཱੑ
    

    View Slide

44. 44
    1. ॳظԽ
    PCΞϧΰϦζϜͷࣄલ஌ࣝʹΑΔಠ֦ࣗு
    2. Τοδ࡟আ
    3. ϧʔϧϕʔεͷํ޲ܾఆ
    ํ޲ܾఆ͸ؒҧ͍͑ͯΔ͜ͱ͕͋Δ
    ᶃ ௚઀ωοτϫʔΫ௨৴Λ͍ͯ͠ͳ͍ม਺ؒͷΤοδΛແ৚݅Ͱ࡟আ
    ᶄ ௨৴ͷऩूσʔλΛجʹɺํ޲Λܾఆɾमਖ਼͢Δ
    

    ɹL4௨৴ɿTCP/UDPͷ઀ଓ։࢝ํ޲ɺύέοτͷϥά૬ؔ
    
    
    ɹL7௨৴ɿHTTPͷϦΫΤετͷ޲͖ …
    ׬શ࿈݁άϥϑͰ͸Τοδ͕ଟ͍ͨΊɺΤοδ࡟আͷॲཧ͕࣌ؒ௕͍
    ࡟আͨ݁͠Ռ
    ߨԋͰ͸εΩοϓ
    

    View Slide

45. ϝοηʔδ
    ੜ੒͞ΕͨҼՌάϥϑͷྫ
    

    View Slide

46. 46
    ݪҼͱ঱ঢ়ͷϝτϦΫεؒͷ
    

    ܦ࿏͕੾ΒΕΔ
    PCΞϧΰϦζϜͷ৚݅෇͖ಠཱੑݕఆͷ՝୊
    front-end:latency user:latency
    orders:latency
    user:memory_usage
    user-db:cpu_usage
    orders:network_receive_bytes
    ঱ঢ়ϝτϦΫε
    orders-db:network_transmit_bytes
    ݪҼ
    

    ϝτϦΫε
    ௐࠪ
    ৚݅෇͖ಠཱੑݕఆͰ͸ɺ৚݅ม਺ͷӨڹΛআڈ͠
    ্ͨͰɺ2ม਺ͷ૬ؔΛΈΔʢภ૬ؔʣ
    
    
    ো֐ൃੜ࣌ͷࣅͨมಈͷܥྻ͕ଟ͍ͱ͖ʹޡͬͯ
    Τοδ͕੾ΒΕ΍͍͢Α͏ʹΈ͑Δ
    front-end:
    

    latency
    orders-db:
    

    network_transmit
    

    _bytes
    user:
    

    memory_
    

    usage
    user:
    

    latency
    ࣅͨมಈΛࣔ͢3ม਺ؒͷ
    Τοδ͕ͳ͔ͥ੾ΒΕ͍ͯΔ
    ✗
    

    View Slide

47. 47
    2ม਺ͷಠཱੑݕఆͷΈͰΤοδ࡟আ
    
    
    ɾ૬ؔͷڧ͍ม਺͕ଟ͍ͨΊɺҼՌάϥϑ͕ڊେʹͳΔ
    
    
    ɾʮมԽͷ։࢝఺ʯʹண໨͠ɺ։࢝఺ͷҐஔͰΫϥελϦϯά
    

    Ϋϥελ͝ͱʹҼՌάϥϑߏங
    PCΞϧΰϦζϜͷ՝୊ΛͲͷΑ͏ʹղܾ͢Δ͔ʁ
    ࿥ը࣌఺Ͱະղܾ
    ৚݅ม਺ͷબ୒Λߟྀ͢Δ
    
    
    ɾมԽ܏޲͕͍ۙม਺Λબ୒ީิ͔Β֎͢
    

    View Slide

48. 48
    ɾ֤࿦Λ௨ఈ͢ΔΑ͏ͳʮ஌ʯ͸·ͩݟ͍͍ͩͤͯͳ͍
    
    
    ɾݪҼ਍அʹ޲͍ͨલॲཧϑϨʔϜϫʔΫͷఏҊΛ໨ࢦ͢
    3. ·ͱΊɿ֤࿦ͷ੔ཧͱֶज़తߩݙ
    ϑΣʔζ̍
    ΦϑϥΠϯ
    
    
    ҟৗݕ஌
    ϑΣʔζ2
    ܗঢ়
    

    ΫϥελϦϯά
    ҼՌάϥϑͷ
    

    ੜ੒
    લॲཧ ݪҼ਍அ
    ҼՌάϥϑͷ
    

    PCΞϧΰϦζϜʹ͸੍໿͕͋Δ
    

    View Slide

49. 4.
    ·ͱΊ
    

    View Slide

50. 50
    1. AIOpsݚڀ͸ɺো֐؅ཧͱϦιʔεϓϩϏδϣχϯάͷ2ͭ
    
    
    2. “Alert symptoms, diagnose causes”ͷண૝ͱઌߦݚڀ
    
    
    3. ݪҼ਍அʹ޲͚ͨલॲཧϑϨʔϜϫʔΫͷߏ੒Λ໨ࢦͨ͠ࢼߦࡨޡ
    
    
    4. ҼՌάϥϑͷੜ੒ͱ఻౷తͳख๏ͷ՝୊
    ·ͱΊ
    

    View Slide

51. 51
    ɾσʔλαΠΤϯεͷಛఆྖҬͷ஌͚ࣝͩͰ͸׬݁͠ͳ͍ͨΊɺֶͿ͜ͱ
    ͕େม
    
    
    ɾ AIOps෼໺ͷॻ੶͕ӳޠݍ΋ؚΊͯଘࡏ͠ͳ͍ͨΊɺ෼໺ͷ೴಺Ϛοϓ
    Λߏங͢Δ͜ͱ͕೉͍͠
    
    
    ɾዞҙతͳύϥϝʔλઃఆͳ͠ͰϞσϧͷੑೳΛߴΊΔ͜ͱ͕೉͍͠
    
    
    ɾՄࢹԽ͠ͳ͍ͱਖ਼͍݁͠ՌΛฦ͍ͯ͠Δ͔͕Θ͔Βͳ͍
    
    
    ɾ࣮ݧʹ࢖͑ΔΑ͏ͳ࣮؀ڥσʔλ͕ೖखͮ͠Β͍
    
    
    ɾࠃ಺Ͱ͸ɺAIOpsʹऔΓ૊ΜͰ͍Δਓ͕গͳ͍ͨΊɺίϛϡχςΟͰ৘
    ใΛڞ༗͠ͳ͕ΒΈΜͳͰࢁΛొΔ͜ͱ͕೉͍͠
    ΍ͬͯΈͯ೉͔ͬͨ͜͠ͱ
    

    View Slide

52. 52
    σʔληοτͷಈతੜ੒γεςϜɿMeltria
    ࿩͞ͳ͔ͬͨ͜ͱ
    ௶಺༎थ, ੨ࢁਅ໵, MeltriaɿϚΠΫϩαʔϏεʹ͓͚Δҟৗݕ஌ɾݪҼ෼ੳͷͨΊͷσʔληοτͷಈతੜ੒γεςϜ, Πϯλʔωοτͱӡ༻ٕज़
    γϯϙδ΢Ϝ࿦จू, 2021, 63-70 (2021-11-18), 2021೥11݄.
    https://github.com/ai4sre/meltria
    Kubernets
    
    
    Sock Shop: ϚΠΫϩαʔϏεΞϓϦ
    
    
    LitmusChaos: ނো஫ೖ
    
    
    Argo Workflows: εέδϡʔϥ
    

    View Slide

53. 53
    ɾΠϯγσϯτͱϙετϞʔςϜͷڞ༗
    
    
    ɾݪҼϝτϦΫε΍ϩά͕ͳΜͰ͔͋ͬͨ
    
    
    ɾো֐ͷ࢑ఆճ෮ʹࢸΔ·ͰʹσʔλΛͲͷखॱͰΈ͔ͨ
    কདྷͷల๬
    ো֐؅ཧʹؔ͢ΔAIOpsͷίϛϡχςΟΛৢ੒͍͖͍ͯͨ͠
    ɾϝτϦΫεͳͲͷӡ༻σʔλͷڞ༗
    
    
    ɾશσʔλͷڞ༗͸೉͘͠ͱ΋ɺಛఆͷϝτϦΫεͷσʔλ͚ͩͰ΋
    ɾো֐஫ೖ& ෛՙࢼݧ
    

    View Slide

54. ෳࡶͳιϑτ΢ΣΞͷӡ༻ʹదԠ͢ΔͨΊʹɺɹɹɹɹɹ
    "*ʹΑΓਓؒͷೝ஌ॲཧ͕ࣗಈԽ͞ΕΔɻ
    

    ͦͷҰํͰɺ"*ͱ͍͏ผछͷෳࡶ͞Λ΋ͭιϑτ΢ΣΞΛ
    ৽ͨʹӡ༻͢Δ͜ͱʹͳΔɻ
    

    

    ͜ͷ૖େͳ*SPOJFTPG"VUPNBUJPOʢࣗಈԽͷൽ೑ʣΛ
    

    ਓྨ͸ͲͷΑ͏ʹղܾ͍ͯ͘͠ͷ͔͸Θ͔Βͳ͍ɻ
    

    View Slide