Observability

Talk from DevConf 2024

Transcript

1. Making Observability Tangible with the help of test automation
3. (slides and references are available online) Why I f ind the topic interesting O11y 101 (a brief intro and some ideas) A playground to experiment Agenda
4. https://unsplash.com/photos/time-lapse-photo-of-lighted-ferris-wheel-at-park-during-nighttime-WnADr2BG174
5. (Reenactment) A long time ago...
6. https://unsplash.com/photos/yellow-and-black-bird-on-brown-wooden-post-q77K0zIDTmI
7. https://unsplash.com/photos/photography-of-theater-chairs-e_RpjNyMgEM
8. https://web.archive.org/web/20110128133821/https://www.net f lix.com/
9. Please hold for a brief interruption https://unsplash.com/photos/white-and-red-plastic-packs-9FDI-_E29 f k
10. The o11y building blocks O`bservabilit | wc - l `y • Metrics • Traces • Logs
11. Metrics • High rate of events, with low cardinality at a reasonable cost • Gauges and counters (and distributions/histograms from those) • Think CPU load, temperature, durations, number of requests
12. Traces • Medium rate of events, supplement metrics • Durations across systems • Think waterfall graphs and pro f ilers (which some APM tools now also include in trace data)
13. Logs • Lower rate of events for comparable cost to metrics but with higher cardinality of related data • Ideally structured, ideally one line per service per request
14. Lower level questions What questions to ask? • which of my hosts have high cpu load? • which service is giving a lot of 500 error responses? • where is the time spent on this queue processing an event?
15. Slightly broader questions What questions to ask? • when will I run out of my error budget for my SLO? • how many items are we selling compared to the same time last month? • how many instances can we turn off?
16. Higher level questions What questions to ask? • how much of our daily sales are from repeat customers? • how many streams do our customers play on average per month over time? • how much are we paying per request handled?
17. https://unsplash.com/photos/a-sign-that-is-on-the-side-of-a-hill-jCfDzOQ2-C8
18. Considerations A sample project • Needs to be actively maintained • Needs to run locally • Needs to mostly maintain itself