Japan Corporation, 2) Japan Advanced Institute of Science and Technology (JAIST) 3) School of Computing, Tokyo Institute of Technology Approximate QoS Rule Derivation Based on Root Cause Analysis for Cloud Computing PRDC 2019 December 1-3, 2019, Kyoto, Japan Satoshi Konno 1) 2) and Xavier Defago 3)
of Yahoo! JAPAN 3 3 Media US Search Video Answer Mail JP US JP Membership C2C Payment C2C EC B2C EC Local Search Knowledge search Mail News Yahoo Auction Premium Loco
OSS Database Platforms 4 300+ Systems 200+ Systems MySQL 2000+ DBs 100+ Systems Cassandra 30 70 60 40 Yahoo Japan NoSQL Team RDB Team • Demand on developing autonomous recovery systems • The number of nodes is increasing year by year. • The human resources are limited. 4000+ Nodes X : Autonomous Recovery X : Autonomous Recovery
Contents 5 • Background and Related Work • Proposal Autonomous Recovery Methods (μQoS and Shape-Root) • Evaluation Result • Conclusion and Future Plans
Systems Traditional (Storage) Monitoring Systems Monitoring Studies for Cloud Computing 7 X : Analysis O : Aggregation X : Root Cause O : Analysis Replacing Tech Giant Public System OSS Type Capacity Period Legacy 2010 DataGarage × Distributed 4,000 nodes - TableStore 2014 Atlas + Winston △ Centralized 2 billion records 6 h Epic 2015 Gorilla △ Centralized 10 billion records 26 h HBase 2016 Borgmon × Distributed + Hierarchical 10,000 nodes 12 h
for Cloud Computing 8 [1] Abdelzahir Abdelmaboud, Dayang NA Jawawi, Imran Ghani, Abubakar Elsafi, and Barbara Kitchenham. Quality of service approaches in cloud computing: A systematic mapping study. Journal of Systems and Software, 101:159–179, 2015 Map of focus areas in research on QoS approaches in cloud computing [1] Distribution of primary studies by contribution type [1] Models: Discusses concepts, makes comparisons, explores relationships, identifies challenges, or makes classifications. Tools: Supports various aspects of QoS approaches in cloud computing. Methods: Presents a model, algorithm or approaches describing the rules.
for Cloud Computing 9 [1] S Anithakumari and K Chandrasekaran. Monitoring and management of service level agreements in cloud computing. In Cloud and Autonomic Computing (ICCAC), 2015 International Conference on, pages 204–207. IEEE, 2015. X : Resource expansion based on system failure without finding the root cause O : Resource extension based on increased demand
μQoS Concept with Root Cause 13 Consumer Provider Root Cause Metrics Service QoS Rule (with No Action) Unsatisfied Metrics Operation QoS Rule (with Recovery Action) STEP1 STEP2 STEP3 Execute the μQoS Rule STEP4 Generating a μQoS Rule • Reliable Root Cause • Fast Root Cause Analysis Mandatory Requirements for Fast Autonomous Recovery
Analysis Methods for Time-Series Data 15 • Correlation Analysis: Traditional parametric statistics method • Clustering Analysis: Grouping a set of metrics in the same group • Recent Studies: BigRoot, Gorilla, etc.
: Correlation Analysis 16 PPMCC [1] Karl Pearson. Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58:240–242, 1895. [2] Charles Loboz, Slawek Smyl, and Suman Nath. Datagarage: Warehousing massive performance data on commodity servers. Proceedings of the VLDB Endowment, 3(1-2):1447–1458, 2010. [3] AbdullahMueen,SumanNath,andJieLiu.Fastapproximatecorrelation for massive time-series data. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, pages 171–182. ACM, 2010. [4] Tuomas Pelkonen, Scott Franklin, Justin Teller, Paul Cavallaro, Qi Huang, Justin Meza, and Kaushik Veeraraghavan. Gorilla: A fast, scalable, in-memory time series database. Proceedings of the VLDB Endowment, 8(12):1816–1827, 2015. • Pearson product-moment correlation coefficient • Traditional Parametric Statistics Method • Some monitoring studies on Cloud computing [2][3][4] denoted using the general correlation algorithm, but these studies did not reveal how to find the root causes using PPMCC in more detail.
: Clustering Analysis 17 k-Shape [1] [1] John Paparrizos and Luis Gravano. k-Shape: Efficient and accurate clustering of time series. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pages 1855–1870. ACM, 2015. • Scalable shape-based clustering algorithm for time-series data based on k-means clustering • Normalized version of the cross- correlation is used for measuring distances between metrics
: Recent Studies 18 BigRoots [1] [1] Honggang Zhou, Yunchun Li, Hailong Yang, Jie Jia, and Wei Li. Bigroots: An effective approach for root-cause analysis of stragglers in big data system. IEEE Access, 6:41966–41977, 2018. • Root-cause analysis for the underlying reasons for stragglers. • Analyzing the stragglers using general metrics such as shuffle read/write bytes and JVM garbage collection time, CPU, I/O, and network.
: Shape-Root for μQoS 19 Shape-Root • Developed with an emphasis on precision and analysis speed to identify reliable root causes dynamically for μQoS • Based on a shape based algorithm, Dynamic Time Warping (DTW), to measure the metrics correlation distance • Root-cause analysis for all time-series metrics for unsatisfied QoS metrics unlike BigRoot • Excluding descendant metrics and confounding metric based on the timestamps unlike PMCC
• Q1: Effectiveness of μQoS and Shape- Root in detecting candidate root causes • Q2: Correlation between analysis span and precision for the time-series data • Q3: Effectiveness of μQoS and Shape- Root for autonomous recovery to real services?
Methods for μQoS 23 • Shape-Root: Our proposal root cause method for μQoS • PPMCC [1]: Standard correlation analysis • k-Shape [2]: General time-series clustering analysis algorithm • BigRoot [3]: Root cause algorithm for stragglers [1] Karl Pearson. Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58:240–242, 1895. [2] John Paparrizos and Luis Gravano. k-Shape: Efficient and accurate clustering of time series. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pages 1855–1870. ACM, 2015. [3] Honggang Zhou, Yunchun Li, Hailong Yang, Jie Jia, and Wei Li. Bigroots: An effective approach for root-cause analysis of stragglers in big data system. IEEE Access, 6:41966–41977, 2018.
24 • Evaluating the following metrics based on leave-one- out cross-validation • TP: Number of correct potential root causes • FP: Number of wrong potential root causes • FN: Number of not detected root causes
Detecting root causes for injected faults 25 QoS Rules Consumer Bad ? CPU Stress (30min cycle) YCSB Read Heavy Workload Rule11 was unsatisfied with CPU load (30min cycle) Best Good
Autonomous Recovery Effectiveness for a real system 28 Consumer (Rule41 ) Initial QoS Rules 5. Execute Rule43 3. Derivate Rule43 2. Root Cause Analysis for Derivate for Rule41 1. Rule41 is Unsatisfied 4. Add Rule43 and Execute Rule43 YCSB Anti-Pattern Workload Provider (Rule42 )
Future Plans μQoS : Event-driven monitoring rule derivation method based on case-based reasoning and root cause analysis for autonomous recovery • Good: μQoS have demonstrated the effectiveness in the real system with high precision and real-time root cause algorithm called Shape-Root. • Bad: Oversimplified the acausal root cause exclusion algorithm based on only the metric timestamp. In complex real systems which has many QoS rules, acausal μQoS may be executed. The study focused only on root cause analysis for past failures. As the next step, we currently plan to expand the μQoS framework for future failure based on model-based reasoning or anomaly detection for preventing potential failures. 30
cybergarage
trycycle
shucho0103
institutdelelevage
konakalab
kiyoshi
rajeshrinet
sshimizu2006
.jpeg){kind=avatar}
arthur1
masakiokuda
thekraken
tapps
deanohume
aleyda
rabernat
afnizarnur
oripsolob
.png){kind=avatar}
helenjbeal
khoart
techseoconnect
codingconduct
baasie
