OpenTelemetry Collector Deployment Patterns

The OpenTelemetry Collector is a versatile piece of software and can be deployed in numerous ways. In this talk, we’ll explore some of these patterns, starting with the classic “agent and collector” and landing at “scalable tail-based sampling,” talking about “multi-tenancy” along the way. We’ll see how they look in theory and practice with live demos on a real Kubernetes cluster. You’ll leave this session with knowledge about some of the essential processors for the OpenTelemetry Collector, how to employ them individually, and as part of a vaster deployment topology.

KubeCon North America 2021
https://sched.co/lV0z

Transcript

2. Juraci Paixão Kröhling - Red Hat OpenTelemetry Collector Deployment Patterns
3. Who am I? ➔ Juraci Paixão Kröhling ➔ @jpkrohling (github|twitter) ➔ Jaeger maintainer ➔ OpenTelemetry Collector approver
4. Agenda ➔ OpenTelemetry ➔ OpenTelemetry Collector ➔ Patterns! @jpkrohling
5. Patterns ➔ Pattern #1 - Basic I and II ➔ Pattern #2 - Normalizer ➔ Pattern #3 - On Kubernetes ➔ Pattern #4 - Load balancer ➔ Pattern #5 - Multi-cluster ➔ Pattern #6 - Multitenant @jpkrohling
6. Patterns ➔ Take a look at the GitHub repository for detailed explanations and working examples. @jpkrohling
7. OpenTelemetry ➔ Fusion of OpenTracing and OpenCensus ➔ Specifications and conventions ➔ Client APIs and SDKs ➔ OTLP (Line Protocol) ➔ Collector (or “otelcol”) @jpkrohling
8. OpenTelemetry Collector “Vendor-agnostic way to receive, process and export telemetry data.” -- https://opentelemetry.io/docs/collector/ @jpkrohling
9. OpenTelemetry Collector ➔ Receivers ➔ Processors ➔ Exporters ➔ Extensions @jpkrohling
10. OpenTelemetry Collector @jpkrohling
11. OpenTelemetry Collector ➔ Receivers ◆ Jaeger ◆ Prometheus ◆ OTLP ◆ Zipkin ◆ ... @jpkrohling
12. OpenTelemetry Collector ➔ Processors ◆ Sampling ◆ Change attributes ◆ Batching ◆ ... @jpkrohling
13. OpenTelemetry Collector ➔ Exporters ◆ Jaeger, Zipkin, ... ◆ Prometheus ◆ OTLP ◆ Pretty much all commercial vendors... @jpkrohling
14. OpenTelemetry Collector ➔ Contrib is where all non-core components live, including vendor-specific ones ➔ Builder allows you to create your own distribution @jpkrohling
15. OpenTelemetry Collector $ cat opentelemetry-collector-config.yaml extensions: health_check: receivers: otlp: protocols: grpc: processors: batch: exporters: logging: service: extensions: [health_check] pipelines: traces: receivers: [otlp] processors: [batch] exporters: [logging] @jpkrohling
16. Pattern #1 - Basic I ➔ OpenTelemetry SDK with OTLP ➔ OpenTelemetry Collector ➔ Jaeger @jpkrohling
17. Pattern #1 - Basic I @jpkrohling
18. Pattern #1 - Basic II - Fanout ➔ OpenTelemetry SDK with OTLP ➔ OpenTelemetry Collector ➔ Jaeger ➔ Some vendor @jpkrohling
19. Pattern #1 - Basic II - Fanout @jpkrohling
20. Pattern #2 - Normalizer ➔ Prometheus client ➔ Jaeger client ➔ Collector with attributes processor ➔ Jaeger ➔ Prometheus @jpkrohling
21. Pattern #2 - Normalizer @jpkrohling
22. Pattern #3 - On Kubernetes I ➔ Workload pod with collector sidecar ➔ Collector ➔ Backends @jpkrohling
23. Pattern #3 - On Kubernetes I @jpkrohling
24. Pattern #3 - On Kubernetes I @jpkrohling
25. Pattern #3 - On Kubernetes I @jpkrohling
26. Pattern #3 - On Kubernetes II ➔ Collector as DaemonSet ➔ Collector ➔ Backends @jpkrohling
27. Pattern #3 - On Kubernetes II @jpkrohling
28. Pattern #3 - On Kubernetes II @jpkrohling
29. Pattern #4 - Load balancer ➔ Multiple services sending data ➔ Trace ID aware load balancer ➔ Scalable tail-based sampling! ➔ Backends @jpkrohling
30. Pattern #4 - Load balancer @jpkrohling
31. Pattern #4 - Load balancer @jpkrohling
32. Pattern #4 - Load balancer @jpkrohling
33. Pattern #4 - Load balancer @jpkrohling
34. Pattern #4 - Load balancer @jpkrohling
35. Pattern #4 - Load balancer @jpkrohling
36. Pattern #5 - Multi-cluster ➔ Collector as agent ➔ Collector in the workload cluster ➔ Collector in the control-plane cluster ➔ Backends possibly elsewhere @jpkrohling
37. Pattern #5 - Multi-cluster @jpkrohling
38. Pattern #5 - Multi-cluster @jpkrohling
39. Pattern #6 - Multitenant ➔ Incoming data from different tenants ➔ Collector with the routing processor ➔ Multiple Jaeger instances @jpkrohling
40. Pattern #6 - Multitenant @jpkrohling
41. Bonus Pattern - Per signal ➔ Client emits different signals ➔ One collector for each signal ➔ Different backends @jpkrohling
42. Bonus Pattern - Per signal @jpkrohling
43. Key takeaways ➔ The OpenTelemetry Collector is very versatile ➔ Get to know the existing components ➔ Chain collectors together ➔ Mix and match @jpkrohling
44. Resources ➔ OpenTelemetry ➔ OpenTelemetry Collector ➔ OpenTelemetry Collector Contrib ➔ OpenTelemetry Collector Builder ➔ Patterns from this presentation Contribute with your own patterns! @jpkrohling
45. Thank you!