Monitoring vs. Debugging

Transcript

1. QCon SF 2023 Monitoring vs. Debugging Prathamesh Sonpatki, last9.io 1
3. M.E.L.T. 3
5. Pillars of Obs Monitoring vs. Debugging 5
6. Why this matters today? • Workloads have changed • Infra is cattle - ephemeral • Services are dynamic • Push to Cloud A 3 node cluster running 10 namespaces with 5 deployments with a replica set of ~3-5 with 10 config maps emits whooping 16566 time series per minute using the popular kube-state-metrics library 6
7. Why this matters today? • Volume • Velocity • Variety • Complexity 7
8. Why this matters today? • Volume • Velocity • Variety • Complexity •C.O.S.T. - Cardinality - Operations - Scale - Toil 8
10. Outcomes we want • To not have downtimes • To mitigate problems quickly • To debug a failure • To know how the system is behaving in real time • To co-relate an outage to a hardware failure • To fi nd anomalies and patterns • To trace a payment failure • To fi nd out unknown failures before they happen • To prevent hampering customer experience and business impact 10
11. Questions we ask • What is wrong? • Did we change anything? • What do we do so this doesn’t repeat? 11
12. Answers we want • Know • Communicate • Recover • Analyse • Debug • Root cause 12
13. Answers we want • System Health • Quick Decisions • Time • Root Cause • Testing • Correctness 13
14. • Getting Started ✅ • Adoption ✅ • Debugging ✅ • Relationships 🥲 Logs • Volume 🥲 • Standardisation 🥲 • Health 🥲 • System insights 🥲 14
15. • Getting Started 😐 • Adoption ✅ • Debugging 🥲 • Relationships 🥲 Metrics • Volume ✅ • Standardisation ✅ • Health ✅ • System insights ✅ 15
16. • Getting Started 😐 • Adoption 🥲 • Debugging ✅ • Relationships ✅ Traces • Volume 😐 • Standardisation ✅ • Health 🥲 • System insights 🥲 16
17. • Getting Started 😐 • Adoption 🥲 • Debugging ✅ • Relationships 🥲 Events • Volume ✅ • Standardisation 🥲 • Health ✅ • System insights ✅ 17
18. Answers we want • Know • Communicate • Recover • Analyse • Debug • Root cause Real Time Post Factor 18
19. For high precision monitoring, You need high precision control levers
20. Control Levers • Treat workloads di ff erently • Tiers • Policies • Declarative Observability 20
21. last9.io/blog Join the Discord Community Thank You! 21