Save 37% off PRO during our Black Friday Sale! »

OpenTelemetry Collector Deployment Patterns

OpenTelemetry Collector Deployment Patterns

The OpenTelemetry Collector is a versatile piece of software and can be deployed in numerous ways. In this talk, we’ll explore some of these patterns, starting with the classic “agent and collector” and landing at “scalable tail-based sampling,” talking about “multi-tenancy” along the way. We’ll see how they look in theory and practice with live demos on a real Kubernetes cluster. You’ll leave this session with knowledge about some of the essential processors for the OpenTelemetry Collector, how to employ them individually, and as part of a vaster deployment topology.

KubeCon North America 2021
https://sched.co/lV0z

E14d17ef58d81ff87d2a5a71cc72babc?s=128

Juraci Paixão Kröhling

October 04, 2021
Tweet

Transcript

  1. None
  2. Juraci Paixão Kröhling - Red Hat OpenTelemetry Collector Deployment Patterns

  3. Who am I? ➔ Juraci Paixão Kröhling ➔ @jpkrohling (github|twitter)

    ➔ Jaeger maintainer ➔ OpenTelemetry Collector approver
  4. Agenda ➔ OpenTelemetry ➔ OpenTelemetry Collector ➔ Patterns! @jpkrohling

  5. Patterns ➔ Pattern #1 - Basic I and II ➔

    Pattern #2 - Normalizer ➔ Pattern #3 - On Kubernetes ➔ Pattern #4 - Load balancer ➔ Pattern #5 - Multi-cluster ➔ Pattern #6 - Multitenant @jpkrohling
  6. Patterns ➔ Take a look at the GitHub repository for

    detailed explanations and working examples. @jpkrohling
  7. OpenTelemetry ➔ Fusion of OpenTracing and OpenCensus ➔ Specifications and

    conventions ➔ Client APIs and SDKs ➔ OTLP (Line Protocol) ➔ Collector (or “otelcol”) @jpkrohling
  8. OpenTelemetry Collector “Vendor-agnostic way to receive, process and export telemetry

    data.” -- https://opentelemetry.io/docs/collector/ @jpkrohling
  9. OpenTelemetry Collector ➔ Receivers ➔ Processors ➔ Exporters ➔ Extensions

    @jpkrohling
  10. OpenTelemetry Collector @jpkrohling

  11. OpenTelemetry Collector ➔ Receivers ◆ Jaeger ◆ Prometheus ◆ OTLP

    ◆ Zipkin ◆ ... @jpkrohling
  12. OpenTelemetry Collector ➔ Processors ◆ Sampling ◆ Change attributes ◆

    Batching ◆ ... @jpkrohling
  13. OpenTelemetry Collector ➔ Exporters ◆ Jaeger, Zipkin, ... ◆ Prometheus

    ◆ OTLP ◆ Pretty much all commercial vendors... @jpkrohling
  14. OpenTelemetry Collector ➔ Contrib is where all non-core components live,

    including vendor-specific ones ➔ Builder allows you to create your own distribution @jpkrohling
  15. OpenTelemetry Collector $ cat opentelemetry-collector-config.yaml extensions: health_check: receivers: otlp: protocols:

    grpc: processors: batch: exporters: logging: service: extensions: [health_check] pipelines: traces: receivers: [otlp] processors: [batch] exporters: [logging] @jpkrohling
  16. Pattern #1 - Basic I ➔ OpenTelemetry SDK with OTLP

    ➔ OpenTelemetry Collector ➔ Jaeger @jpkrohling
  17. Pattern #1 - Basic I @jpkrohling

  18. Pattern #1 - Basic II - Fanout ➔ OpenTelemetry SDK

    with OTLP ➔ OpenTelemetry Collector ➔ Jaeger ➔ Some vendor @jpkrohling
  19. Pattern #1 - Basic II - Fanout @jpkrohling

  20. Pattern #2 - Normalizer ➔ Prometheus client ➔ Jaeger client

    ➔ Collector with attributes processor ➔ Jaeger ➔ Prometheus @jpkrohling
  21. Pattern #2 - Normalizer @jpkrohling

  22. Pattern #3 - On Kubernetes I ➔ Workload pod with

    collector sidecar ➔ Collector ➔ Backends @jpkrohling
  23. Pattern #3 - On Kubernetes I @jpkrohling

  24. Pattern #3 - On Kubernetes I @jpkrohling

  25. Pattern #3 - On Kubernetes I @jpkrohling

  26. Pattern #3 - On Kubernetes II ➔ Collector as DaemonSet

    ➔ Collector ➔ Backends @jpkrohling
  27. Pattern #3 - On Kubernetes II @jpkrohling

  28. Pattern #3 - On Kubernetes II @jpkrohling

  29. Pattern #4 - Load balancer ➔ Multiple services sending data

    ➔ Trace ID aware load balancer ➔ Scalable tail-based sampling! ➔ Backends @jpkrohling
  30. Pattern #4 - Load balancer @jpkrohling

  31. Pattern #4 - Load balancer @jpkrohling

  32. Pattern #4 - Load balancer @jpkrohling

  33. Pattern #4 - Load balancer @jpkrohling

  34. Pattern #4 - Load balancer @jpkrohling

  35. Pattern #4 - Load balancer @jpkrohling

  36. Pattern #5 - Multi-cluster ➔ Collector as agent ➔ Collector

    in the workload cluster ➔ Collector in the control-plane cluster ➔ Backends possibly elsewhere @jpkrohling
  37. Pattern #5 - Multi-cluster @jpkrohling

  38. Pattern #5 - Multi-cluster @jpkrohling

  39. Pattern #6 - Multitenant ➔ Incoming data from different tenants

    ➔ Collector with the routing processor ➔ Multiple Jaeger instances @jpkrohling
  40. Pattern #6 - Multitenant @jpkrohling

  41. Bonus Pattern - Per signal ➔ Client emits different signals

    ➔ One collector for each signal ➔ Different backends @jpkrohling
  42. Bonus Pattern - Per signal @jpkrohling

  43. Key takeaways ➔ The OpenTelemetry Collector is very versatile ➔

    Get to know the existing components ➔ Chain collectors together ➔ Mix and match @jpkrohling
  44. Resources ➔ OpenTelemetry ➔ OpenTelemetry Collector ➔ OpenTelemetry Collector Contrib

    ➔ OpenTelemetry Collector Builder ➔ Patterns from this presentation Contribute with your own patterns! @jpkrohling
  45. Thank you!