A practical introduction to observability

Good observability is essential for modern software. It gives us confidence that our systems are working properly. And it also allows us to debug issues efficiently. In this talk, we’ll explore everything you need to know to start applying good observability to your projects. And we’ll see the most common pitfalls you need to be aware of. We will start with the tools and basic concepts in monitoring. And we’ll go over the 3 most common mistakes people make with it. Then we’ll see how to have automatic alerts to detect issues. And, we’ll touch on the principles for setting up good alerts. As a final step, we’ll see how to build our logging system and how to apply it in the most efficient way to debug issues easily.

Transcript

1. A practical introduction to observability Nikolay Stoitsev Engineering Manager @ Halo DX
4. Monitoring
5. Monitoring Logging
6. Monitoring Logging Distributed Tracing
7. Monitoring
8. Monitoring system components Application Application Application Monitoring System Time Series Database Dashboard
9. Monitoring system components Application Application Application Monitoring System Time Series Database Dashboard Prometheus, Graphite, m3db
10. Monitoring system components Application Application Application Monitoring System Time Series Database Dashboard Prometheus UI, Grafana
11. Counter
12. Counter increase
13. Timer
14. Labels
15. What to watch out for?
16. Cardinality
17. Cardinality • search.success, app_version=1, type=Patient • search.success, app_version=1, type=Exam • search.success, app_version=2, type=Patient • search.success, app_version=2, type=Exam
18. #1. Don’t add high cardinality tags
19. Metrics are not accurate • DB engine optimizes for faster operations • When performing some operations for a different time resolution • When archiving metrics for long term storage
20. #2. Don’t rely on metrics infrastructure for BI
21. Don’t use average values • Averages hide the outliers • Doesn’t represent typical behavior
22. Use percentiles • Represents the worst experience in 90% of the time • Can measure p90, p95, p99 p90
23. Histograms • Shows the whole distribution • Configurable buckets
24. #3. Use percentiles or histograms
26. Example alert
27. Alert Levels Send Slack/Teams Message
28. Alert Levels Send alert to oncall
29. Alerting tool is usually built into the metrics system
30. Alerts should be • urgent • important • actionable • real
31. Should represent either ongoing or imminent problems
32. What to watch out for?
33. 1. Better to remove an alert when it’s noisy
35. #2. Use success rate
36. Symptom-based monitoring • Number of 5xx HTTP response codes • Response time • Email sending is not working • Users can’t log in
37. Cause-based monitoring • Free disk space on database server • Memory utilisation • Free file descriptors
38. Many causes may trigger a symptom
39. User impact is most important
40. #3. Focus on symptom-based alerts
41. Cause-based alerts are also necessary
42. Picking alerts to start with Front-end Load Balancer Back-end DB Count rate of successful log-in Count request success rate
44. Logging
45. Logging system Application Application Application Log Aggregation Database Dashboard Log Collector Log Collector Log Collector Logstash, Fluentd
46. Logging system Application Application Application Log Aggregation Database Dashboard Log Collector Log Collector Log Collector Elasticsearch, Loki
47. Logging system Application Application Application Log Aggregation Database Dashboard Log Collector Log Collector Log Collector Kibana
48. Log messages
49. Finding logs Can search by: • content of log message message : *notification* • all logs from a service kubernetes.labels.app/name.keyword : "api-gateway" • many more thanks to flexible query schema
50. What to watch out for?
51. #1. Use appropriate log level - info, warn, error
52. Structured logging • Append useful key=value pairs • Can group (aggregate) by the keys • Can sort by aggregations
53. #2. Use structured logging
54. Too many logs Application Application Application Log Aggregation Real Time Search Engine Log Scraper Log Scraper Log Scraper Dashboard
55. Too many logs Application Application Application Log Aggregation Real Time Search Engine Log Scraper Log Scraper Log Scraper Dashboard Reduce log retention period
56. Too many logs Application Application Application Log Aggregation Real Time Search Engine Log Scraper Log Scraper Log Scraper Dashboard Cold Storage Query UI
57. #3. Use proper retention period or cold storage
59. Distributed tracing https://www.youtube.com/watch?v=rM1z7Q1TxR0
60. End-to-end summary 1. Configure automated alerts
61. End-to-end summary 1. Configure automated alerts 2. Use metrics and tracing to pinpoint the problem
62. End-to-end summary 1. Configure automated alerts 2. Use metrics and tracing to pinpoint the problem 3. Use structured logging to find the root cause of the problem easily
63. End-to-end summary 1. Configure automated alerts 2. Use metrics and tracing to pinpoint the problem 3. Use structured logging to find the root cause of the problem easily 4. Fix problems and make sure all metrics are always back to normal
64. Thank you! Q&A Nikolay Stoitsev Engineering Manager at Halo DX Photo by Pixabay, Şahin Sezer Dinçer, Andrea Piacquadio, Ian Beckley from Pexels