Monitoring and Alerting: Knowing the Unknown

Transcript

1. Monitoring and Alerting @amandasopkin
2. Where are we headed?
3. Introduction/Disclaimer @amandasopkin
4. Best Practices Overview
5. Principles of a Solid Monitoring/Alerting System No single point of failure in monitoring system Alerts are clear and actionable If everything is quiet, something is wrong
6. Rule 0: Set priorities
7. Set priorities “99.99% uptime for the zoombinator responses to emails” Minimize latency to X service from Z. Prioritize traffic on Fridays in November.
8. A note on 99% uptime
9. Rule 1: Alert on symptoms, not causes
10. Alert on symptoms, not causes The danger of false positives
11. Alert on symptoms, not causes Is this indicative of a direct impact on user work? Are there various causes possible? Could this NOT have an impact?
12. Alert on symptoms, not causes Server load is high This server is down for maintenance Slow site responses
13. Rule 2: Keep it simple
14. Keep it simple “The zoombinator received userid 10202 and returned availability X, Y, Z” “After zoomification response 123 the zoombinator received 39393, proceed to create string akjkjd, and { data: [29, 393, x99] } and returned 6, 7, 8, { data: 3939 }.”
15. Rule 3: Consoles are
16. Consoles are
17. Consoles are
18. Consoles are Have no more than 5 graphs on a console. Have no more than 5 plots (lines) on each graph. You can get away with more if it is a stacked/area graph.
19. Monster effectiveness
20. Having good consoles: monster effectiveness?
21. Having good consoles: monster effectiveness?* *Still very important!!
22. Rule 4: Make it easy to figure out which component is at fault
23. Make it easy to figure out which component is at fault “The zoombinator received userid 10202 and threw exception X on line 29” “Received userid 10202 and failed to return availability.
24. Rule 5: Create a process for addressing and resolving alerts
25. Create a process for addressing and resolving alerts 1. Alerts/pages will go to the team alias. 2. Current “team guardian” will address all alerts and provide an update to the team. 3. Alerts will be diagrammed to show their trends.
26. Discovering points of failure Escalation procedures
27. Discovering points of failure Escalation procedures What will happen if no one responds? What is the appropriate chain? After hours vs. on the job?
28. Discovering points of failure Discovering points of failure
29. Discovering points of failure: Latency
30. Discovering points of failure: Latency
31. Discovering points of failure: Latency
32. Discovering points of failure: Latency
33. Discovering points of failure: Latency the 99th percentile latency is the worst latency that was observed by 99% of all requests. it is the maximum value if you ignore the top 1%. a common notation for 99th percentile is "p99"
34. Discovering points of failure: Latency
35. Discovering points of failure: Latency but p99 is still ignoring the worst 1% of requests. if you want to see the worst latencies, you can take a look at the p100 latency, often called the maxiumum or the max
36. Discovering points of failure: Latency
37. Discovering points of failure: Latency
38. Analyzing latency
39. Discovering points of failure: Latency Distinguish between latency of successful responses vs. failed responses
40. Discovering points of failure: Latency you can define a cut-off latency of (for example) 500ms, and then measure how many requests exceeded that value. you can think of this as a "latency error budget". 1% of your (projected) traffic is the amount of "slow" requests that you can spend.
41. Discovering points of failure: Latency "the p99 latency, aggregated over 10 second intervals, should not go above 500ms for longer than 20 seconds."
42. Discovering points of failure: Error rates
43. Discovering points of failure: Error rates
44. Analyzing error rate
45. Discovering points of failure: Error rates What you might miss...
46. (Sidenote) Discovering points of failure: Error rates
47. Discovering points of failure: Traffic A measure of how much demand is placed on the system
48. Discovering points of failure: Traffic
49. Analyzing traffic
50. Discovering points of failure: Saturation A measure of the “fullness” of the service.
51. Discovering points of failure: Saturation
52. Analyzing saturation
53. Discovering points of failure: More generally 1. Evaluate trends in alerts. 2. For significant problems, consider a larger evaluation/retrospective. 3. Consider GA and consumer care input in conjunction with data.
54. Discovering points of failure: Meta-monitoring
55. Meta-monitoring
56. Discovering points of failure: Regular batch jobs
57. Discovering points of failure: Conclusion The 4 signals: latency, error rates, saturation, traffic Meta-monitoring is important Don’t forget batch jobs!
58. Best practices for logs
59. Framework for log levels FATAL ERROR WARN INFO DEBUG TRACE
60. Framework for log levels FATAL: a “shutdown” level problem ERROR: an error that is fatal to what is being attempted WARN: anything that can cause oddities INFO: generally useful DEBUG: diagnostic information TRACE: one specific part of a function
61. What is the goal for each log?
62. Logging: dos and don’ts: Avoid side effects try { log.trace("Id=" + request.getUser().getId() + " accesses " + manager.getPage().getUrl().toString()) } catch(NullPointerException e) { // do something }
63. Logging: dos and don’ts DO Be concise and descriptive “Availability for userid 3939 at 10-19-2018T05:07 returned successfully in AvailabilityByDay.” “Availability returned successfully”
64. Logging: dos and don’ts DO Log method arguments and return values Parameters: zpid: 39393, listingId: 3939 Return value: “true” “true”
65. Logging: dos and don’ts Consider logging ALL external input if(log.isDebugEnabled()) { log.debug("Processing ids: {}", requestIds); } else { log.info("Processing ids size: {}", requestIds.size()); }
66. Logging: dos and don’ts Log exceptions the right way (Java) try { Integer x = null; ++x; } catch (Exception e) { log.error(e); }
67. Logging: dos and don’ts Log exceptions the right way try { Integer x = null; ++x; } catch (Exception e) { log.error(e.toString()); log.error(e.getMessage()); }
68. Logging: dos and don’ts Log exceptions the right way try { Integer x = null; ++x; } catch (Exception e) { log.error(e); log.error(e.toString()); log.error(e.getMessage()); log.error("Error reading configuration file: " + e); }
69. Logging: dos and don’ts Log exceptions the right way try { Integer x = null; ++x; } catch (Exception e) { log.error("", e); log.error("Error reading configuration file", e); }
70. Logging: dos and don’ts Provide context “listingId 399393 returned for call to get tour availability from BDP” “listingId 39939”
71. Logging: dos and don’ts Use unique identifiers “tourId: 39393” “id: output 39393”
72. In conclusion... Make it easy to fix!
73. Now that we’ve got the main bases down...
74. A note on testing.... Code Review Integration Testing Unit Testing Manual Testing
75. 30% 45% A note on testing 55% Code Review Integration Testing Unit Testing Manual Testing 5-10%
76. 30% 45% A note on testing 55% Code Review Integration Testing Unit Testing Manual Testing 5-10%
77. Finding the weird s***... Load test your system
78. Finding the weird s***... Load test your system Do a bug bash
79. Finding the weird s***... Load test your system Do a bug bash Always be alert
80. Things you might STILL miss Highly segmented failures Small buckets of users “Blips” on the radar
81. Things you might miss Highly segmented failures alerts by segment Small buckets of users consumer care “Blips” on the radar threshold alerts
82. Things you might miss Highly segmented failures alerts by segment Small buckets of users consumer care “Blips” on the radar threshold alerts
83. When all else fails...
84. Why do we care?
85. Why do we care? • Early signs of bigger issues • Potentially really bad for a small vocal number (bad for ratings) • Pie in the sky
86. Anomaly detection
87. Kinds of anomalies: • Point anomalies: a single blip • Contextual anomalies: bad within context • Collective anomalies: these shouldn’t happen together
88. Anomaly detection: Thresholds
89. Threshold anomaly detection
90. Multivariate correlation vs.
91. Multivariate anomaly detection
92. Cyclical thresholds (days of week)
93. Cyclical thresholds
94. Learning patterns of traffic
95. Pattern anomaly detection
96. Key challenges
97. Server data not normally distributed
98. Overfitting based on complex model
99. Quasi-seasonal Traffic November 2017 Traffic Thanksgiving 2017
100. Many missed anomalies
101. Different bounds for different metrics
102. Changing environment
103. “Normal” changes
104. Case Study 1: Basic Anomaly Detection
105. Case Study 2: Lots of metrics combined
106. Case Study 3: Beyond
107. Case Study 3: Beyond
108. Case Study 3: Beyond
109. Case Study 3: Beyond
110. Case Study 3: Beyond
111. Case Study 3: Beyond
112. Case Study 3: Beyond
113. Case Study 3: Beyond
114. Case Study 3: Beyond
115. Case Study 3: Beyond
116. Case Study 3: Beyond
117. Case Study 3: Beyond
118. Case Study 3: Beyond
119. Case Study 3: Beyond
120. Case Study 3: Beyond
121. Case Study 3: Beyond
122. Case Study 3: Beyond
123. Case Study 3: Beyond
124. Case Study 3: Beyond
125. Case Study 3: Beyond
126. Case Study 3: Beyond
127. Kibana Flexible Easy anomaly detection Open source
128. So why do we do this again?
129. Anomaly detection... Gives us early signals Keeps us safe Is computationally hard, but possible
130. How effective is it against monsters?
133. Open Source Recommendations
134. Kibana (ELK) by Elastic
135. Prometheus
136. Prometheus example... https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
137. In conclusion...
138. Conclusion... Cover your bases with 4 golden signals Be proactive about your system Consider anomaly detection for your use case
139. Thank you! @amandasopkin
140. Sources • Flaticon.com • https://igor.io/latency/ • https://prometheus.io • https://landing.google.com/sre/sre-book/chapters/ monitoring-distributed-systems/ • Stack overflow