Monitoring vs. Debugging - SRE BLR Meetup

Related blog posts

https://last9.io/blog/understanding-metrics-events-logs-traces-key-pillars-of-observability/

https://last9.io/blog/observability-oss-vs-paid-vs-managed-oss/

https://last9.io/blog/taking-back-control-of-your-monitoring/

Join my Discord community to mingle with like-minded SRE/DevOps folks - https://discord.com/invite/Q3p2EEucx9

Transcript

1. SRE BLR Meetup 29th July Monitoring vs. Debugging Prathamesh Sonpatki, last9.io 1
2. SRE BLR Meetup 29th July You -> Me-> 🌮 🍱 Prathamesh Sonpatki, last9.io 2
4. M.E.L.T. 4
6. Pillars of Obs Monitoring vs. Debugging 6
8. Why this matters today? • Workloads have changed • Infra is cattle - ephemeral • Services are dynamic • Push to Cloud A 3 node cluster running 10 namespaces with 5 deployments with a replica set of ~3-5 with 10 config maps emits whooping 16566 time series per minute using the popular kube-state-metrics library 8
9. Why this matters today? • Workloads have changed • Infra is cattle - ephemeral • Services are dynamic • Push to Cloud • Pod Metrics • Deployment Metrics • ReplicaSet Metrics • StatefulSet Metrics • DaemonSet Metrics • Job Metrics • Service Metrics • Namespace Metrics • Node Metrics 9
10. Why this matters today? • Volume • Velocity • Variety • Complexity 10
11. Why this matters today? • Volume • Velocity • Variety • Complexity •C.O.S.T. - Cardinality - Operations - Scale - Toil 11
13. Outcomes we want • To not have downtimes • To mitigate problems quickly • To debug a failure • To know how the system is behaving in real time • To co-relate an outage to a hardware failure • To fi nd anomalies and patterns • To trace a payment failure • To fi nd out unknown failures before they happen • To prevent hampering customer experience and business impact 13
14. Questions we ask • What is wrong? • Did we change anything? • What do we do so this doesn’t repeat? 14
15. Answers we want • Know • Communicate • Recover • Analyse • Debug • Root cause 15
16. Answers we want • System Health • Quick Decisions • Time • Root Cause • Testing • Correctness 16
17. M.E.L.T. 201 17
18. Logs 18
19. Logs • Can be literally anything -> Unstructured logs • Standard programs -> Structured logs 19
20. • Getting Started ✅ • Adoption ✅ • Debugging ✅ • Relationships 🥲 Logs • Volume 🥲 • Standardisation 🥲 • Health 🥲 • System insights 🥲 20
21. • Bene fi ts of aggregation • Less volume • Patterns, Trends, Insights Logs2Metrics 21
22. Metrics 22
23. Metrics • Aggregated! • Fastest and Cheapest to understand system health • Dimensions 23
24. • Getting Started 😐 • Adoption ✅ • Debugging 🥲 • Relationships 🥲 Metrics • Volume ✅ • Standardisation ✅ • Health ✅ • System insights ✅ 24
25. Traces 25
26. Traces • Relationships and Directions! • Scoped to a request/work fl ow 26
27. • Getting Started 😐 • Adoption 🥲 • Debugging ✅ • Relationships ✅ Traces • Volume 😐 • Standardisation ✅ • Health 🥲 • System insights 🥲 27
28. Events 28
29. Events • Structured logs? • Schema based? • Domain Events • Easier to adopt? • Can unlock co-relation • Dimensionality 29
30. • Getting Started 😐 • Adoption 🥲 • Debugging ✅ • Relationships 🥲 Events • Volume ✅ • Standardisation 🥲 • Health ✅ • System insights ✅ 30
31. Answers we want • Know • Communicate • Recover • Analyse • Debug • Root cause Real Time Post Factor 31
32. 32 Open Standards
33. Common Goals, Common Ground • Continue to leverage open source innovations • Multiple options • No Truck factor 33
34. Common Goals, Common Ground • Continue to leverage open source innovations • Multiple options • No Truck factor 34
35. OpenAPM • Metrics for APM • Based on open-source Prometheus, OpenMetrics and Grafana • Support for NodeJS and Golang • https://github.com/last9/nodejs-openapm/ 35
36. OpenAPM 36
37. OpenTelemetry 37 80% of Telemetry data is unused
38. 80% of Telemetry data is unused • Yet, we store it and pay for the data that is unused! • Slow dashboards, concurrent access woes • No real time alerting 38
39. For high precision monitoring, You need high precision control levers
40. Control Levers • Treat workloads di ff erently • Tiers • Policies • Declarative Observability 40
41. Control Levers • Treat workloads di ff erently • Tiers • Policies • Declarative Observability 41
42. What is your monitoring strategy? 42
43. 43 https://last9.io/cricket-scale/ last9.io/friends
44. last9.io/blog last9.io/friends Join the Discord Community Thank You! 44