Monitoring for reliability

Running a production operation without monitoring is like driving a car without looking out at the road. This presentation aims to convince that monitoring is essential, show how it can be added to any application, and give some examples of what you can achieve with a well reasoned monitoring set-up.

Transcript

1. Monitoring for Reliability Michele Pittoni – SpazioDati Speck&Tech Retreat ’18, Dimaro
2. Hi! I’m Michele DevOps engineer @ SpazioDati Secretary @ Speck&Tech Board game player @ Ludimus
3. Once upon a time… Ghost by Cristiano Zoucas from the Noun Project
4. Load balancer AAA Proxy AAA DB API API DB Elasticsearch 3rd party HAproxy HAproxy HAproxy Consul
6. A very successful deploy…
7. I. Fundamentals of monitoring
8. Why do we monitor? • Alerts • Dashboards • Debugging • Experiments • Long-term trends
9. The “Four Golden Signals” • Latency • Traffic • Errors • Saturation
10. Rate Errors Duration Focussed on services Useful proxy of user happiness Good indication of service level Weak for predicting problems
11. Utilisation Saturation Errors Focussed on resources Good for: • predicting problems • capacity planning • debugging outages Weak correlation with user experience
12. II. How-to
13. Prometheus
14. Prometheus
15. Prometheus
16. Prometheus
17. Prometheus
18. Prometheus
19. Prometheus
20. III. What is monitoring good for?
21. Service Level * Service Level Indicators a.k.a. Metrics Service Level Objectives • Approved by stakeholders • Realistic (achievable) • Have consequences • Are continuously refined Service Level Agreement contractual form of SLOs
22. 100% is always a bad SLO
23. Error budget & policy Error budget = 100% - SLO Error budget policy defines what to do when the budget is exhausted, e.g. • Stopping new features • Shifting priority to bug fixing • Assigning more resources • Relaxing SLO
24. Alerting rules Only one alert per problem Only if it’s urgent Only if it’s actionable Only if it requires human intelligence
25. Alerting rules - alert: DiskIsAlmostFull expr: node_filesystem_avail / node_filesystem_size < 0.2 for: 10m labels: severity: normal annotations: description: | '{{ $labels.instance }} free disk space is < 0.2 (current value: {{ $value }})'
26. Alerting rules - alert: CpuCreditsWillFinish expr: predict_linear(cpu_credits[1h], 2 * 3600) < 0 for: 5m labels: severity: medium annotations: description: | Credits for instance {{ $labels.instance }} are predicted to finish in 2 hours.
27. Alerting rules - alert: HighErrorRate expr: slo_errors_per_request:ratio_rate10m >= 0.001 - record: slo_errors_per_request:ratio_rate10m expr: sum(rate(slo_errors[10m])) / sum(rate(slo_requests[10m]))
28. Long alerting window Graph from The Site Reliability Workbook, http://landing.google.com/sre/book.html
29. Long alerting delay Graph from The Site Reliability Workbook, http://landing.google.com/sre/book.html
30. Multi-window alerting Graph from The Site Reliability Workbook, http://landing.google.com/sre/book.html
31. Self–healing & automation
32. Alert + webhook = awesome routes: - match: alertname: CeleryStuck receiver: restart_celery receivers: - name: restart_celery url: https://your.alert.receiver/endpoint
33. Self–healing & automation
34. Now go, and monitor ALL THE THINGS! @mpitt pittoni@spaziodati.eu michele.pittoni.org