Elizabeth (Betsy) Nichols

Detecting anomalies is easy. What’s hard is deciding what to do when you find one. A good decision takes a lot more information than what pure anomaly detection can provide.

This talk is about making your analytics accountable and provides practical steps to ensure that anomaly detection is helping, instead of pestering you with noise. Additionally, we cover strategies to ensure that mistakes, which are inevitable, aren’t repeated.

This talk provides a framework, lessons learned, and specific techniques for driving constant improvement in your monitoring’s decisions about what to do and when.

Transcript

1. Anomalies ≠ Alerts Elizabeth (Betsy) Nichols, PhD May 2017
2. www.netuitive.com 2 The next 30- min • Now • Pain • Relief • Bonus: Use Case @eanTweet
3. Now @eanTweet
4. www.netuitive.com 4 Monitoring Now Context Engine @eanTweet Data Alert Anomaly Detected Alert Issued
5. Anomalies = Alerts @eanTweet
6. @eanTweet
7. Pain @eanTweet
8. 1. Alert Fatigue @eanTweet
9. www.netuitive.com 9 Alert No Alert Problem No Problem TP TN FP Good, Bad, @eanTweet
10. www.netuitive.com 10 1.  Alerts à Alert Fatigue 2.  Alerts = Anomalies 3.  Anomalies à Alert Fatigue 4.  How many anomalies can we reasonably expect? First Question: @eanTweet
11. www.netuitive.com 11 Anomalies Live Here Mean Lower Upper Metric Possible Observed Values Pr(Observation) @eanTweet
12. www.netuitive.com 12 Step 1: Population Distribution Probability Observed Value (units = StdDev from the Mean) @eanTweet
13. www.netuitive.com 13 Step 2: Population Size •  Consider one metric (time series) •  1 minute observations •  That amounts to: 365days 1year × 24hours day × 60min hour = 525600 obs / year / metric 525600 365 =1440 obs / day / metric •  # AWS CloudWatch metrics/EC2 = 14 •  # statsd metrics per server ~ 125 @eanTweet
14. www.netuitive.com 14 Step 3: Compute Anomalies / Day # Metrics: 50,000 11.4M 1.6M 2280 1.6M 1.6M 11.4M 2280 µ ±4σ -4σ -3σ -2σ -1σ µ +1σ +2σ +3σ 4σ 9.7K 9.7K @eanTweet Samples: 1440/d
15. www.netuitive.com 15 #anomalies∈ (−∞,−4σ ]= 2×50,000×1440× P(X ≤ −4) where P(X ≤ −4) = 1 σ 2π e − (x−µ)2 2σ 2 −∞ −4 ∫ dx = 3.167124e-05 @eanTweet
16. At scale, there are lots and lots of anomalies @eanTweet
17. www.netuitive.com 17 1.  Decision required for each anomaly 2.  Anomalies = 3.  Likely: #FP >> #TP 4.  Decision not to alert à 5.  Potential à FN 6.  FP is bad; FN is ugly To Alert … or … Not to Alert @eanTweet {TP}∪{FP}
18. 2. Seeking Needles(TP) @eanTweet
19. www.netuitive.com 19 Pr(Prob) Pr(!Prob) Observed values Probability
20. www.netuitive.com 20 Pr(Prob) Pr(!Prob) Observed values Probability TN TP ?
21. www.netuitive.com 21 Pr(Prob) Pr(!Prob) Observed values Probability TN TP ?
22. www.netuitive.com 22 TN TP
23. www.netuitive.com 23 TN TP
24. www.netuitive.com 24 Pr(Prob) Pr(!Prob) Observed values Probability TN TP
25. @eanTweet Pr(FP) == Noise Pr(FN) == Missed Alert As Pr(FN) Pr(FP)
26. Relief @eanTweet
27. www.netuitive.com 27 Context @eanTweet
28. www.netuitive.com 28 Without Context @eanTweet
29. www.netuitive.com 29 With Context @eanTweet
30. www.netuitive.com 30 Without Context @eanTweet
31. www.netuitive.com 31 With Context @eanTweet
32. www.netuitive.com 32 Nice Forecasting with Context @eanTweet
33. www.netuitive.com 33 Raw Metric Values Bands of Normalcy Anomalies Time Without Context @eanTweet
34. www.netuitive.com 34 With Context (Case 1) t2.small (1 vcpu) m4.large (2 vcpu) EC2 CPU Utilization @eanTweet FP
35. www.netuitive.com 35 With Context (Case 2) @eanTweet FN Response Time Good SLA not met
36. www.netuitive.com 36 Context Adding @eanTweet
37. www.netuitive.com 37 Basic Monitoring Pattern @eanTweet Context Data Engine Alerts
38. www.netuitive.com 38 Basic Context Cont ext Data Value Semantic Engine Rule •  Streaming •  Asynchronous •  Synchronous Tag Server Metric Alerts Alert @eanTweet
39. www.netuitive.com 39 Context: Semantic Model @eanTweet Think of analytics as attribute discovery Context Data Semantic Model Analytics
40. www.netuitive.com 40 clean profile model aggregate classify detect anomalies @eanTweet Attribute Discovery
41. www.netuitive.com 41 EC2 EC2 EC2 ASG Service Model Arrival Rate •  Queue Length •  Response Time Completion Rate •  # EC2’s •  Mean CPU Utilization Queueing Recommendation •  # EC2’s •  Mean CPU Utilization Latency Quant Models Instance Instance Instance Auto Scaling Group Elastic Load Balancer @eanTweet
42. www.netuitive.com 42 Engine with Context Analytics Context @eanTweet Data Semantic Model Action Policy
43. www.netuitive.com 43 Engine = Analytics + Alert Policies Semantic Model Scope Conditions Action(s) @eanTweet
44. www.netuitive.com 44 Example: Policy for Action For all EC2’s in US West Region with tag=“PROD” in ASGx … If AvgCpuUtil(EC2) deviating & RunQ > 2*(# CPUs) & ReqRate(ELBx ) !deviating & AvgLatency(ELBx ) > 2 sec For duration >=10 minutes On a week day Send critical alert Invoke autoscaleUp on ASGx @eanTweet
45. www.netuitive.com 45 Alerts With Context @eanTweet •  Annotations •  Conditions Results •  Links to runbook •  Affected services •  Timestamp •  Severity •  Duration •  Images Context Analytics+Policy + Semantic Model Data Alerts
46. www.netuitive.com 46 Without Context Context Engine @eanTweet Data Alert Anomaly Detected Alert Issued
47. www.netuitive.com 47 With Context Context @eanTweet Data Semantic Model ENGINE
48. www.netuitive.com 48 Take Aways @eanTweet
49. www.netuitive.com 49 Anomalies ≠ Alerts •  Anomalies = Alerts: Not a good idea •  Preferred strategy: anomalies + context = alertsthe good kind •  To add context •  Organize all that you know into a semantic model that can be populated in real time or not •  Think of analytics as a way to discover key attributes that enrich your semantic model •  Decide on appropriate action based upon the context held in the semantic model and issue enriched alerts @eanTweet
50. Best Monitoring = Math + @eanTweet Context
51. Use Case @eanTweet
52. Cost Optimization @eanTweet
53. www.netuitive.com 53 Report: ASG Capacity vs Utilization ASG Group: # Nodes Provisioned 11 ASG Group: 95% Percentile CPU Utilization Time $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ @eanTweet
54. www.netuitive.com 54 Make It Work Max Workload Time in Hours à # Nodes = 11 Response Time < 3 sec ASG Nodes Workload Max Requests/sec Requests/sec @eanTweet
55. www.netuitive.com 55 Regular Shapes Actual Requests/sec # Nodes = 11 Response Time < 3 sec @eanTweet
56. www.netuitive.com 56 EC2 EC2 EC2 ASG Service Model Arrival Rate •  Queue Length •  Response Time Completion Rate •  # EC2’s •  Mean CPU Utilization Queueing Recommendation •  # EC2’s •  Mean CPU Utilization Latency Quant Models Instance Instance Instance Auto Scaling Group Elastic Load Balancer @eanTweet
57. www.netuitive.com 57 Queueing Model queue queue queue EC2 EC2 EC2 Arrivals Completions Arrival Rate •  Queue Length •  Service Time Completion Rate •  # EC2’s •  Mean CPU Utilization Latency @eanTweet
58. M/M/c Queuing Model •  λ = Arrival rate with Poisson distribution •  µ = Average service time with exponential distribution •  c = # servers •  Servers serve from the front of the queue (FCFS) •  If less than c jobs, some servers will be idle •  If greater than c jobs, some will queue in a buffer •  The buffer is of infinite size @eanTweet
59. Equations Response time = intensity = Ref: https://en.wikipedia.org/wiki/M/M/c_queue @eanTweet The probability that an arriving job is forced to queue is given by Erlange’s C formula:
60. M/M/c Model Results 2 4 6 8 10 µ = 1.0s Response Time @eanTweet
61. M/M/c Model Application INPUT: • Service Time (µ) • Arrival rate of requests (λ) • Response Time goal (3sec) OUTPUT: • Actual Response Time • Mapping: arrival rate à # nodes with constraint: response time < 3sec @eanTweet
62. www.netuitive.com 62 Scaling Model Reserve OnDemand Spot Response Time < 3 sec Arrival Rate @eanTweet Recommendation
63. www.netuitive.com 63 Risk Mitigation Factors Max scale-in rate: Max scale-out rate: Min instance counts: Zero thresholds: Target response time: Workload metric: Update frequency: 3 sec aws.elb.elb-c.arrivalrate hourly 1% 2 3 unlimited @eanTweet
64. www.netuitive.com 64 With Context Context @eanTweet Data Semantic Model ENGINE •  ASG model •  Real-time workload •  Real-time performance •  Weekly ASG profile •  M/M/c model results •  Client risk parameters •  EC2 instance types •  Hourly costs •  AWS ASG scale actuators
65. www.netuitive.com 65 Context à Navigable Landscape @eanTweet
66. Best Monitoring = Math + @eanTweet Context
67. www.netuitive.com 67 Contact Info Elizabeth(Betsy) Nichols, Ph.D. bnichols@netuitive.com @eanTweet #talk-betsy-nichols www.netuitive.com @netuitive (703) 464-1500 @eanTweet