Efficient Monitoring in modern environments

Container environments make it easy to deploy hundreds of microservices in today’s infrastructures. Monitoring thousands of metrics efficiently introduces new challenges to not lose insight, avoid alert fatigue and maintain a high development velocity. In this talk I’ll present an overview of important metrics including the 4 golden signals, discuss strategies to organize alerting efficiently, give insight into SoundCloud’s monitoring history and highlight a few success and failure stories.

Transcript

1. Efficient monitoring in modern environments Tobias Schmidt - ContainerDays Hamburg 2016 @dagrobie - github.com/grobie
2. Introduction About myself Production Engineer for 5+ years Container orchestration (in-house, Kubernetes) Service discovery Monitoring (Prometheus) Production readiness
3. Monitoring
4. Collecting, processing, aggregating, and displaying real- time quantitative data about a system, such as query counts and types, processing times, and server lifetimes. Site Reliability Engineering - O’Reilly 2016 Monitoring
5. Monitoring
6. Monitoring Why monitor? Enable automatic alerting Analysis of long-term trends Validate new features/experiments/implementations Debugging
7. Monitoring Blackbox vs. Whitebox Blackbox: Externally observed What the user sees Whitebox: Data exposed by the system Allows to act on imminent issues
8. Metrics
9. Metrics Instrument everything Host (CPU, memory, I/O, network, filesystem, …) Container (CPU, memory, restarts, OOM, throttling, …) Applications (throughput, latency, queues, …)
10. Metrics Export detailed metrics Attach all relevant information Use aggregations later in alerts and dashboards
11. Metrics Four golden signals Minimum set of metrics every service should have Coined by Google SRE
12. Four golden signals Latency Time to serve user requests Median doesn’t reflect user experience
13. Four golden signals Traffic Demand placed on a system (HTTP requests, network throughput, transactions, …)
14. Four golden signals Errors Failure responses to user requests
15. Four golden signals Saturation & Utilization Consumption of constrained resources (Memory, I/O, CPU slices, …)
16. Alerting
17. Alerting Use symptom based alerting Monitor for your users Four golden signals (traffic is tricky) Only page if something needs immediate human intervention
18. Alerting Prevent alert fatigue Alert grouping Provide easy silencing Dependencies Avoid static thresholds
19. Alerting Use ticketing system Avoid email spam Warnings are tasks like new features
20. Alerting Provide runbooks (playbooks) Keep them concise Explanation, hints, links Dynamic - include recent observations Discuss with non-experts
21. Alerting Practice outages “Game days” Repeat regularly
22. Matt T. Proud, Julius Volz, Björn Rabenstein, Matthias Rampke Philosophy on Alerting - Rob Ewaschuk Acknowledgements
23. Thank you May the queries flow, and your pagers be quiet. Tobias Schmidt - ContainerDays Hamburg 2016 @dagrobie - github.com/grobie