DevOps: Why Analytics Fail

Are you plagued by false alerts? Is your monitoring system asleep at the switch when real “situations” occur? Do you feel like your analytics are letting your down? This presentation explores why analytics fail; but also how you can implement monitoring strategies where they succeed.

Transcript

1. Why Analytics Fail Betsy Nichols, Ph.D. Chief Data Scientist DevOpsDays Austin May 2016
2. www.netuitive.com 2 The stories you are about to hear are true. The names have been changed to protect the innocent. @eanTweet
3. www.netuitive.com 3 Where is DevOps?
4. www.netuitive.com 4 Market Visibility Time Hype Cycle: Analytics for DevOps Analytics for DevOps Hyperbole Index 2016 @eanTweet
5. www.netuitive.com 5 = Applied Analytics engine- ering math hacking computer science
6. www.netuitive.com 6 Engineering Computer Science Math Hacking
7. www.netuitive.com 7 Math Hacking Engineering Computer Science
8. www.netuitive.com 8 Math Hacking Engineering Computer Science
9. www.netuitive.com 9 Math Hacking Engineering Computer Science
10. www.netuitive.com 10 Types of Analytics • Off-line analytics (“Reporting”) o  Trends over hours, weeks, or months o  Optimization strategies o  Recommendations o  Business intelligence • Hybrid • Near real-time analytics (“Monitoring”) o  Detection o  Troubleshooting o  Remediation @eanTweet
11. Reporting @eanTweet
12. www.netuitive.com 12 Report: ASG Capacity vs Utilization ASG Group: # Nodes Provisioned 15 ASG Group: 95% Percentile CPU Utilization Time
13. www.netuitive.com 13 Report: ASG Capacity vs Utilization ASG Group: # Nodes Provisioned 15 ASG Group: 95% Percentile CPU Utilization Time $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $
14. Monitoring @eanTweet
15. www.netuitive.com 15 Alarm No Alarm Problem No Problem TP TN FP Good, Bad, @eanTweet
16. www.netuitive.com 16 = Wanted _ Alarms Unwanted _ Alarms SNR = TP FP Success (model) @eanTweet
17. www.netuitive.com 17 Goal: @eanTweet
18. www.netuitive.com 18 Analytics DeterministicwStatistical @eanTweet
19. www.netuitive.com 19 Deterministic Analytics* * Current DevOps state of the art @eanTweet
20. www.netuitive.com 20 + Static Thresholds + Counting & Transformations + Eyeballing Dashboards @eanTweet Common Deterministic Analytics
21. www.netuitive.com 21 Dashboards: Visualization @eanTweet
22. www.netuitive.com 22 Dashboards: Do Not Scale Pr(Missed Alarm) # Metrics @eanTweet
23. www.netuitive.com 23 False Negative: EC2 Cost Explosion Duration: 13 hours Cost: $thousands Cost Per Hour Hour Anomalies unnoticed due to lack of automation @eanTweet Credentials revoked Credentials stolen
24. www.netuitive.com 24 Static Thresholds: Intuitive Cost Per Hour Hour @eanTweet
25. www.netuitive.com 25 Cost Per Hour Hour Static Thresholds: Hard To Set @eanTweet
26. www.netuitive.com 26 Static Thresholds: Monotonic Metrics Anomalies Hidden in Cumulative Data @eanTweet
27. www.netuitive.com 27 Static Thresholds: Monotonic Metrics Anomalies Hidden in Cumulative Data @eanTweet
28. www.netuitive.com 28 Counting + Transformations (Uni-Variate) •  Delta: raw[n] à (raw[n] – raw[n-1]) •  Rate: raw[n] à (raw[n] / time) •  Scale: raw[n] à (raw[n] * constant) •  Min: raw[n] à min(raw[…]) •  Max: raw[n] à max(raw[…]) •  RHMAX: raw[n] à (raw[n] / max(raw)) •  Frequency: range(x) à # observations @eanTweet
29. www.netuitive.com 29 Transformation: Monotonic Metrics Sum Delta m[n] = x[n] - x[n-1] @eanTweet
30. www.netuitive.com 30 Transformation: Frequency Possible alternative to static threshold F(range) = count @eanTweet
31. www.netuitive.com 31 Transforms: Seasonal Anomaly @eanTweet
32. www.netuitive.com 32 Statistical Analytics* *Where the state of the art is heading @eanTweet
33. www.netuitive.com 33 Statistical Analytics Assumption: *The Tempest by William Shakespeare, Act II, Scene I “What’s past is prologue*” @eanTweet
34. www.netuitive.com 34 + Correlation Models 1 0.8 0.4 0 -0.4 -0.8 -1 + Machine Learning @eanTweet
35. www.netuitive.com 35 Correlation t1 t3 t1 t3 t4 t5 t1 t2 t3 t4 t5 t7 t6 t1 t2 t3 t4 t5 t7 t6 Confidence Interval = x% Revenues/sec Requests/sec @eanTweet
36. www.netuitive.com 36 Correlation Analytic r = ((x i −µx )(y i −µy )) i=1 n ∑ (x i −µx )2 i=1 n ∑ (y i −µy )2 i=1 n ∑ Pearson Product Moment Coefficient of Correlation for two metrics X and Y @eanTweet
37. www.netuitive.com 37 2D: Correlated Metrics (r~0.91) Requests/sec Revenue/sec @eanTweet
38. www.netuitive.com 38 2D: Correlation Anomaly (r~0.51) Requests/sec Revenue/sec
39. www.netuitive.com 39 High Correlation (r~0.99) Requests/sec Revenue/sec @eanTweet
40. www.netuitive.com 40 Correlation: Non-Linear Relationships For each pattern r = 0.00
41. www.netuitive.com 41 Statistical Machine Learning @eanTweet
42. www.netuitive.com 42 Metrics Model Phase I: Learning Learning Engine @eanTweet
43. www.netuitive.com 43 Metrics Model Phase II: Detection Anomalies Learning Engine @eanTweet
44. www.netuitive.com 44 Metrics New Model Phase III: Adaptive Learning Anomalies Adaptations Learning Engine @eanTweet
45. www.netuitive.com 45 Learned Bands of Normalcy Time 05/01 05/02 05/03 05/04 05/06 05/07 05/08 200 150 100 50 0 Values = Raw Metric Observations Xi ! σn (Xi | X j ) ! σn (X W i ) @eanTweet multi-variate uni-variate “Bands of Normalcy”
46. www.netuitive.com 46 Learning: Web Page Views @eanTweet multi-variate uni-variate observed values Time
47. www.netuitive.com 47 Learning @ Work Deviation from Norm @eanTweet observed values uni-variate band of normalcy (narrowing)
48. www.netuitive.com 48 Credit Card Authorizations Deviations @eanTweet observed values multi-variate uni-variate
49. www.netuitive.com 49 Nuance: New Normal t2.small (1 vcpu) m4.large (2 vcpu) CPU Utilization Save $$$ @eanTweet multi-variate uni-variate observed values
50. www.netuitive.com 50 Nuance: Bad Change Bad Good Response Time SLA Not Met multi-variate uni-variate observed values
51. www.netuitive.com 51 How do you tell the difference ? New Normal vs Long Running Problem︎
52. www.netuitive.com 52 Nuance: Memory Leak Algorithm: Only if no anomalies: 1. Take sequence of relative minimum values 2. Fit a linear regression model. Test goodness of fit. 3. Use model to predict future relative min values 4. Alarm two hours relative min of heap used = 90% % Heap Used Garbage collection Garbage collection Garbage collection Garbage collection
53. www.netuitive.com 53 What metrics are relevant? A very specific failure mode︎ @eanTweet
54. www.netuitive.com 54 Math is the Queen of the Sciences …. Impressive, but not enough. @eanTweet
55. www.netuitive.com 55 Context is Important @eanTweet
56. Adding Context
57. www.netuitive.com 57 Selection @eanTweet
58. www.netuitive.com 58 Workflow Bakes in Context Context Results Data Analytics @eanTweet Context Actions
59. www.netuitive.com 59 Take Aways
60. Best Analytics = Math +
61. www.netuitive.com 61 @eanTweet
62. www.netuitive.com 62 Contact Info Elizabeth Nichols, Ph.D. Chief Data Scientist bnichols@netuitive.com @eantweet www.netuitive.com @netuitive (703) 464-1500 @eanTweet