Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

o11yfest - Sampling in Distributed Tracing

o11yfest - Sampling in Distributed Tracing

Sampling is still one of the biggest challenges in distributed tracing. While the basic concept is easy to grasp, the number of choices and their trade-offs requires learning about the techniques and your own workload. In this session, we are giving you all the knowledge required to master the sampling techniques: we’ll talk about head and tail-based sampling, as well as adaptive sampling, and we’ll wrap it up with a bonus discussion on trace aggregation. You’ll leave this session ready to implement scenarios, from the simple “probabilistic head-sampling” up to the complex “scalable tail-based sampling” using open source tools like OpenTelemetry Collector.

Register and watch this talk now! https://o11yfest.org/attend

More Decks by Juraci Paixão Kröhling

Other Decks in Programming

Transcript

  1. o11yfest - Sampling in Distributed Tracing @jpkrohling 2 Hi 󰗜,

    I’m Juraci! Software Engineer @ Red Hat, distributed tracing team Maintainer on the Jaeger project Member of the OpenTelemetry project
  2. o11yfest - Sampling in Distributed Tracing @jpkrohling 3 Agenda Why

    and what Heads and tails Other related ideas
  3. o11yfest - Sampling in Distributed Tracing @jpkrohling 10 Distributed tracing

    Beyond a point, it’s not feasible to store metadata for every transaction.
  4. o11yfest - Sampling in Distributed Tracing @jpkrohling 11 Distributed tracing

    Beyond a point, it’s not feasible to store metadata for every transaction. (except if you are an intelligence agency)
  5. o11yfest - Sampling in Distributed Tracing @jpkrohling 12 Distributed tracing

    Am I able to manage all this data? Do I need to keep them all?
  6. o11yfest - Sampling in Distributed Tracing @jpkrohling 13 Distributed tracing

    Keeping everything allows you to perform better data analysis, potentially at a later time.
  7. o11yfest - Sampling in Distributed Tracing @jpkrohling 14 Distributed tracing

    Keeping everything costs money, makes maintenance harder, and might be useless.
  8. o11yfest - Sampling in Distributed Tracing @jpkrohling 15 Sampling The

    decision to capture or discard a specific trace.
  9. o11yfest - Sampling in Distributed Tracing @jpkrohling 19 Head-based sampling

    Constant (always or never) Probabilistic (chance of 1 in N) Rate-limiting (N per second)
  10. o11yfest - Sampling in Distributed Tracing @jpkrohling 21 Head-based sampling

    Downsides: Valuable traces are not recorded Historical data analytics usually not possible
  11. o11yfest - Sampling in Distributed Tracing @jpkrohling 23 Tail-based sampling

    The decision is made when the trace is complete. (do we know when a trace is complete?)
  12. o11yfest - Sampling in Distributed Tracing @jpkrohling 25 Tail-based sampling

    How long should we wait? Decide based on which attribute? How many resources does it need?
  13. o11yfest - Sampling in Distributed Tracing @jpkrohling 30 Tail-based sampling

    With OpenTelemetry Collector: loadbalancingexporter tailsamplingprocessor
  14. o11yfest - Sampling in Distributed Tracing @jpkrohling 31 Tail-based sampling

    Good when we need to select only a few interesting traces.
  15. o11yfest - Sampling in Distributed Tracing @jpkrohling 32 Tail-based sampling

    Downsides: Need to define “interesting” More complex to maintain Limited data analysis
  16. o11yfest - Sampling in Distributed Tracing @jpkrohling 33 Stateless Collector

    Sampling The collector is responsible for downsampling, typically without having a complete view of the trace.
  17. o11yfest - Sampling in Distributed Tracing @jpkrohling 34 Adaptive sampling

    The technique of changing the sampling strategy based on the traced application’s current behavior.
  18. o11yfest - Sampling in Distributed Tracing @jpkrohling 35 Adaptive sampling

    Typically used to ensure that all endpoints in a given service are sampled.
  19. o11yfest - Sampling in Distributed Tracing @jpkrohling 37 Going back

    a bit... We use sampling mostly to reduce network traffic and storage requirements.
  20. 41 twitter.com/jpkrohling Photos from Pixabay and Pexels: Agenda, Distributed Tracing,

    Sampling, Head-based, Tail-based, Stateless collector sampling, Adaptive sampling, Remote sampling, Aggregation Thank you