o11yfest - Sampling in Distributed Tracing

Sampling is still one of the biggest challenges in distributed tracing. While the basic concept is easy to grasp, the number of choices and their trade-offs requires learning about the techniques and your own workload. In this session, we are giving you all the knowledge required to master the sampling techniques: we’ll talk about head and tail-based sampling, as well as adaptive sampling, and we’ll wrap it up with a bonus discussion on trace aggregation. You’ll leave this session ready to implement scenarios, from the simple “probabilistic head-sampling” up to the complex “scalable tail-based sampling” using open source tools like OpenTelemetry Collector.

Register and watch this talk now! https://o11yfest.org/attend

Transcript

1. Sampling in Distributed Tracing Juraci Paixão Kröhling Software Engineer @jpkrohling
2. o11yfest - Sampling in Distributed Tracing @jpkrohling 2 Hi 󰗜, I’m Juraci! Software Engineer @ Red Hat, distributed tracing team Maintainer on the Jaeger project Member of the OpenTelemetry project
3. o11yfest - Sampling in Distributed Tracing @jpkrohling 3 Agenda Why and what Heads and tails Other related ideas
4. o11yfest - Sampling in Distributed Tracing @jpkrohling 4 Distributed tracing Produces a high-fidelity signal.
5. o11yfest - Sampling in Distributed Tracing @jpkrohling 5 Distributed tracing
6. o11yfest - Sampling in Distributed Tracing @jpkrohling 6 Distributed tracing
7. o11yfest - Sampling in Distributed Tracing @jpkrohling 7 Distributed tracing
8. o11yfest - Sampling in Distributed Tracing @jpkrohling 8 Distributed tracing
9. o11yfest - Sampling in Distributed Tracing @jpkrohling 9 Distributed tracing
10. o11yfest - Sampling in Distributed Tracing @jpkrohling 10 Distributed tracing Beyond a point, it’s not feasible to store metadata for every transaction.
11. o11yfest - Sampling in Distributed Tracing @jpkrohling 11 Distributed tracing Beyond a point, it’s not feasible to store metadata for every transaction. (except if you are an intelligence agency)
12. o11yfest - Sampling in Distributed Tracing @jpkrohling 12 Distributed tracing Am I able to manage all this data? Do I need to keep them all?
13. o11yfest - Sampling in Distributed Tracing @jpkrohling 13 Distributed tracing Keeping everything allows you to perform better data analysis, potentially at a later time.
14. o11yfest - Sampling in Distributed Tracing @jpkrohling 14 Distributed tracing Keeping everything costs money, makes maintenance harder, and might be useless.
15. o11yfest - Sampling in Distributed Tracing @jpkrohling 15 Sampling The decision to capture or discard a specific trace.
16. o11yfest - Sampling in Distributed Tracing @jpkrohling 16 Sampling Head-based Tail-based
17. o11yfest - Sampling in Distributed Tracing @jpkrohling 17 Head-based sampling
18. o11yfest - Sampling in Distributed Tracing @jpkrohling 18 Head-based sampling
19. o11yfest - Sampling in Distributed Tracing @jpkrohling 19 Head-based sampling Constant (always or never) Probabilistic (chance of 1 in N) Rate-limiting (N per second)
20. o11yfest - Sampling in Distributed Tracing @jpkrohling 20 Head-based sampling Good when network costs are a concern.
21. o11yfest - Sampling in Distributed Tracing @jpkrohling 21 Head-based sampling Downsides: Valuable traces are not recorded Historical data analytics usually not possible
22. o11yfest - Sampling in Distributed Tracing @jpkrohling 22 Tail-based sampling The decision is made when the trace is complete.
23. o11yfest - Sampling in Distributed Tracing @jpkrohling 23 Tail-based sampling The decision is made when the trace is complete. (do we know when a trace is complete?)
24. o11yfest - Sampling in Distributed Tracing @jpkrohling 24 Tail-based sampling
25. o11yfest - Sampling in Distributed Tracing @jpkrohling 25 Tail-based sampling How long should we wait? Decide based on which attribute? How many resources does it need?
26. o11yfest - Sampling in Distributed Tracing @jpkrohling 26 Tail-based sampling How to scale?
27. o11yfest - Sampling in Distributed Tracing @jpkrohling 27 Tail-based sampling
28. o11yfest - Sampling in Distributed Tracing @jpkrohling 28 Tail-based sampling
29. o11yfest - Sampling in Distributed Tracing @jpkrohling 29 Tail-based sampling
30. o11yfest - Sampling in Distributed Tracing @jpkrohling 30 Tail-based sampling With OpenTelemetry Collector: loadbalancingexporter tailsamplingprocessor
31. o11yfest - Sampling in Distributed Tracing @jpkrohling 31 Tail-based sampling Good when we need to select only a few interesting traces.
32. o11yfest - Sampling in Distributed Tracing @jpkrohling 32 Tail-based sampling Downsides: Need to define “interesting” More complex to maintain Limited data analysis
33. o11yfest - Sampling in Distributed Tracing @jpkrohling 33 Stateless Collector Sampling The collector is responsible for downsampling, typically without having a complete view of the trace.
34. o11yfest - Sampling in Distributed Tracing @jpkrohling 34 Adaptive sampling The technique of changing the sampling strategy based on the traced application’s current behavior.
35. o11yfest - Sampling in Distributed Tracing @jpkrohling 35 Adaptive sampling Typically used to ensure that all endpoints in a given service are sampled.
36. o11yfest - Sampling in Distributed Tracing @jpkrohling 36 Remote sampling strategy
37. o11yfest - Sampling in Distributed Tracing @jpkrohling 37 Going back a bit... We use sampling mostly to reduce network traffic and storage requirements.
38. o11yfest - Sampling in Distributed Tracing @jpkrohling 38 Trace aggregation
39. o11yfest - Sampling in Distributed Tracing @jpkrohling 39 Trace aggregation
40. o11yfest - Sampling in Distributed Tracing @jpkrohling 40 Trace aggregation
41. 41 twitter.com/jpkrohling Photos from Pixabay and Pexels: Agenda, Distributed Tracing, Sampling, Head-based, Tail-based, Stateless collector sampling, Adaptive sampling, Remote sampling, Aggregation Thank you