FChain: toward black-box online fault localization for cloud systems

Presentation of the paper "FChain: toward black-box online fault localization for cloud systems" by Hiep Nguyen, Zhiming Shen, Yongmin Tan and Xiaohui Gu, presented at ICDCS 2013

Transcript

1. FChain Hiep Nguyen, Zhiming Shen, Yongmin Tan, Xiaohui Gu Toward Black-box Online Fault Localization for Cloud Systems Presented by Stefano Zanella
2. Agenda Why ? What ? (and what not!) How ? Experimental evaluation Results Open questions Paper evaluation
3. Why?
4. Text IaaS is complex no matter whether public or private http://harish11g.blogspot.it/2012/07/amazon-availability-zones-aws-az.html
5. Text Distributed Software is complex and it’ll be even more so in the future http://www.mimul.com/pebble/default/2012/09/06/1346890689112.html
6. Failures are unavoidable …so they must be considered from day 0
7. Failures are unavoidable …so they must be considered from day 0 resource contentions
8. Failures are unavoidable …so they must be considered from day 0 resource contentions software bugs
9. Failures are unavoidable …so they must be considered from day 0 resource contentions software bugs hardware failures
10. Failures are unavoidable …so they must be considered from day 0 resource contentions software bugs hardware failures …
11. Text http://www.n2growth.com/blog/leadership-and-blame/
12. The Challenge:
13. localize faulty component(s)
14. localize faulty component(s) (as quickly as possible)
15. Text Available tools aren’t enough http://blogs.urbancode.com/agile/maslows-hammer-the-curse-of-tool-blindness/
16. White/Grey box techniques
17. White/Grey box techniques intrusive
18. White/Grey box techniques intrusive difficult to deploy
19. White/Grey box techniques intrusive difficult to deploy significant runtime overhead
20. White/Grey box techniques intrusive difficult to deploy significant runtime overhead Impractical for online fault localisation
21. Black box techniques
22. Black box techniques only for anomaly detection…
23. Black box techniques only for anomaly detection… …or system metric attribution
24. Black box techniques only for anomaly detection… …or system metric attribution some are application-specific or fault-specific
25. Black box techniques only for anomaly detection… …or system metric attribution some are application-specific or fault-specific Still not enough
26. What?
27. FChain
29. black-box
30. black-box online
31. black-box online fault localization
32. black-box online fault localization system
33. «black box» vs
34. «black box» application instrumentation vs
35. «black box» application instrumentation low-level system metrics vs
36. «online» vs
37. «online» offline algorithm vs
38. «online» offline algorithm online algorithm vs
39. «fault localization» vs
40. «fault localization» anomaly detection vs
41. «fault localization» anomaly detection fault pinpointing vs
42. «system» vs
43. «system» single algorithm vs
44. «system» single algorithm set of cooperative algorithms vs
45. What not:
47. no prior application knowledge
48. no prior application knowledge (suitable for IaaS clouds)
49. no prior application knowledge (suitable for IaaS clouds) no requirement for training data
50. no prior application knowledge (suitable for IaaS clouds) no requirement for training data (detects known and new anomalies)
51. How?
52. Key observations
53. Key observations Performance anomalies often manifest as abnormal system metric fluctuations that are distinctive from the normal fluctuation patterns
54. Key observations Performance anomalies often manifest as abnormal system metric fluctuations that are distinctive from the normal fluctuation patterns The abnormal system metric changes often start from the faulty components and then propagate to other non-faulty components via inter-component interactions
55. Key questions
56. Key questions How to distinguish between normal & abnormal fluctuations?
57. Key questions How to distinguish between normal & abnormal fluctuations? How to correctly determine the root cause?
58. Key ideas
59. Key ideas Prediction models + predictability metric
60. Key ideas Prediction models + predictability metric Dependency relationships + fault propagation model
61. Text The mechanics http://21stcenturydamocles.deviantart.com/art/Chronometer-as-study-in-the-mechanics-of-time-336999135
62. Text Architecture
63. Text Architecture
64. Text Architecture
65. Text Architecture
66. Text Architecture
67. Text Architecture
68. Text Architecture
69. Text Architecture
70. Text Architecture
71. Text Architecture
72. Text Architecture
73. Core modules Normal fluctuation modeling Abnormal change point selection Integrated fault diagnosis Online pinpointing validation
74. Abnormal change point selection
75. Abnormal change point selection metrics fluctuate under normal workload
76. Abnormal change point selection metrics fluctuate under normal workload (standard algorithms detect random points)
77. Abnormal change point selection bursty and spiky might be normal
78. Abnormal change point selection bursty and spiky might be normal (smoothing + outlier detection isn’t enough)
79. Abnormal change point selection So what?
80. Abnormal change point selection So what? Predict changes!
81. Normal fluctuation modeling
82. Normal fluctuation modeling PRESS (PRedictive Elastic reSource Scaling)
83. Normal fluctuation modeling PRESS (PRedictive Elastic reSource Scaling) “when and how will a metric change?“
84. Normal fluctuation modeling PRESS (PRedictive Elastic reSource Scaling) “when and how will a metric change?“ prediction == error
85. Abnormal change point selection Observation: normal == low prediction error
86. Abnormal change point selection Ergo: abnormal == (prediction error > threshold)
87. Abnormal change point selection Q: how high is “high“?
88. Abnormal change point selection Q: how high is “high“? (tip: it isn’t fixed)
89. Abnormal change point selection A: dynamic threshold computation
90. Abnormal change point selection
91. Abnormal change point selection given change point xt
92. Abnormal change point selection given change point xt X = [ xt - 20s, xt + 20s ]
93. Abnormal change point selection given change point xt X = [ xt - 20s, xt + 20s ] FFT(X)
94. Abnormal change point selection given change point xt X = [ xt - 20s, xt + 20s ] top(90%, FFT(X))
95. Abnormal change point selection given change point xt X = [ xt - 20s, xt + 20s ] FFT-1(top(90%, FFT(X)))
96. Abnormal change point selection given change point xt X = [ xt - 20s, xt + 20s ] percentileOf(FFT-1(top(90%, FFT(X))),90)
97. Abnormal change point selection given change point xt X = [ xt - 20s, xt + 20s ] threshold = percentileOf(FFT-1(top(90%, FFT(X))),90)
98. Abnormal change point selection
99. Abnormal change point selection Q: when does the anomaly start?
100. Abnormal change point selection Q: when does the anomaly start? (tip: might be earlier than noticed)
101. Abnormal change point selection A: tangent-based rollback
102. Abnormal change point selection f’(xt)
103. Abnormal change point selection f’(xt) - f’(xt-1)
104. Abnormal change point selection f’(xt) - f’(xt-1) < 0.1
105. Abnormal change point selection f’(xt) - f’(xt-1) < 0.1 xt = xt-1
106. Abnormal change point selection while f’(xt) - f’(xt-1) < 0.1 xt = xt-1
107. Integrated fault diagnosis
108. Integrated fault diagnosis sort all components by fault manifestation time
109. Integrated fault diagnosis sort all components by fault manifestation time earliest == faulty
110. Integrated fault diagnosis sort all components by fault manifestation time earliest == faulty time difference < 2s == concurrent fault
111. Integrated fault diagnosis sort all components by fault manifestation time earliest == faulty time difference < 2s == concurrent fault all faulty, same trend == external fault
112. Online pinpointing validation
113. Online pinpointing validation «inter-component dependency discovery»
114. Online pinpointing validation (no dependency b/w components == all faulty) «inter-component dependency discovery»
115. Online pinpointing validation (no dependency b/w components == all faulty) (offline use, black-box) «inter-component dependency discovery»
116. Caveats
117. Caveats back-pressure effect
118. Caveats back-pressure effect data stream processing
119. Caveats back-pressure effect data stream processing
120. Caveats back-pressure effect data stream processing FCHAIN STILL WORKS
121. http://www.thedirtfloor.com/2009/12/23/bansky-elephant-in-the-room/
122. …and it’s called «time» http://www.thedirtfloor.com/2009/12/23/bansky-elephant-in-the-room/
123. NTP
124. NTP error < 0.1ms (LAN), 5 ms (WAN)
125. NTP error < 0.1ms (LAN), 5 ms (WAN) anomaly propagation takes several seconds
126. NTP error < 0.1ms (LAN), 5 ms (WAN) anomaly propagation takes several seconds small time skews are tolerable
127. Experimental evaluation
128. Platform Linux + Xen NCSU’s Virtual Computing Lab
129. Metrics CPU usage RAM usage Network in Network out Disk read Disk write 1s sampling interval
130. Benchmarks RUBiS online auction benchmark Hadoop IBM System S
131. SLO Violations RUBiS: avg response time > 100ms Hadoop: no progress for > 30s IBM System S: avg processing time > 20ms
132. Fault Injection memory leak cpu hog network hog disk hog bottleneck (single component)
133. Fault Injection offload bug load balancing bug concurrent cpu hog concurrent disk hog concurrent memory leak (multi component)
134. Competitors Histogram NetMedic Topology Dependency PAL Fixed filtering (existing black-box localization schemes)
135. Scores precision = recall = Ntp Ntp + Nfp Ntp Ntp + Nfn
136. Config params look back window = 100s (500s for disk hog) concurrent faults threshold = 2s burst extraction window = 20s burst spectrum = 90% top frequencies expected prediction error = 90th percentile
137. Single-component RUBiS
138. Single-component System S
139. Multi-component RUBiS
140. Multi-component System S
141. Multi-component Hadoop
142. Online validation effect
143. Results
144. Summary
145. Summary Up to 90% higher precision
146. Summary Up to 90% higher precision Up to 20% higher recall
147. Summary Up to 90% higher precision Up to 20% higher recall Slave modules demands < 1% CPU, ~3MB RAM per host
148. Summary Up to 90% higher precision Up to 20% higher recall Slave modules demands < 1% CPU, ~3MB RAM per host High parallelism, high scalability
149. Summary Up to 90% higher precision Up to 20% higher recall Slave modules demands < 1% CPU, ~3MB RAM per host High parallelism, high scalability Still room for improvement (e.g. adaptive look-back window)
150. Open questions
151. Open questions
152. Open questions How quickly is a failure detected?
153. Open questions How quickly is a failure detected? More predictive / less reactive approach?
154. Open questions How quickly is a failure detected? More predictive / less reactive approach? Data retention?
155. Paper evaluation
156. originality
157. technical meaning
158. clarity
159. fitness
160. Thanks! Stefano Zanella ! zanella.stefano@gmail.com @stefano_zanella https://github.com/stefanozanella