O11yfest 2021: Exporting Data with OpenTelemetry by Steve Flanders

1. Exporting Data with O11yfest 2021
2. Cloud Native Telemetry foreach(language) foreach(language) collectors, sidecars, etc w3c trace-context, wire formats for trace data, metrics, logs, etc Telemetry “verticals” Telemetry “layers” Tracing Metrics Logs, etc Instrumentation APIs Canonical implementations Data infrastructure Interop formats
3. @smflanders flands https:/ /sflanders.net Steve Flanders Director of Engineering, Splunk OpenTelemetry Website Maintainer OpenTelemetry Collector Triager CNCF SIG-Observability Member Previously: • Head of Product, Omnition • Senior Manager for Logs, VMware
4. Components & Architecture
5. OpenTelemetry Components API, SDK, Data Single library per language for all signals Receive, process, and export data Specification Instrumentation Libraries Collector
6. Application Instrumentation Libraries Otel Inst. Library API SDK Processors Exporters RUM Serverless
8. OpenTelemetry Reference Architecture
9. OpenTelemetry Reference Architecture
10. But why?
11. Application Instrumentation
12. Host Application traces metrics logs SaaS Back-end On-prem Back-end • Signals may be push or pull; back-end determines whether scraping/tailing is possible • If instrumentation direct to back-end then every application, and language-specific, needs to: ◦ Authenticate ◦ Batch ◦ Compress ◦ CRUD metadata ◦ Encryption ◦ Proxy ◦ Queue ◦ Retry • In addition, host telemetry is desirable and correlation can be challenging metrics logs
13. Host Application traces metrics logs SaaS Back-end On-prem Back-end • Signals may be push or pull; back-end determines whether scraping/tailing is possible • If instrumentation to collector then every application, and language-specific, needs to (everything else optional): ◦ Batch ◦ Queue ◦ Retry • In addition, the collector can capture host telemetry and correlate with application telemetry as well as convert between formats Collector metrics + logs traces metrics + logs
14. Summary: Exporting Application Instrumentation • Application instrumentation direct to back-end ◦ Instrumentation responsible for full exporting lifecycle; requires even more resources ◦ Instrumentation export features are language-specific; mileage may vary ◦ Adding resource information may prove challenging or impossible ◦ Summary: great for proof-of-concepts, be careful in production environments
15. Summary: Exporting Application Instrumentation • Application instrumentation direct to back-end ◦ Instrumentation responsible for full exporting lifecycle; requires even more resources ◦ Instrumentation export features are language-specific; mileage may vary ◦ Adding resource information may prove challenging or impossible ◦ Summary: great for proof-of-concepts, be careful in production environments • Application instrumentation to local Collector (agent) to back-end ◦ Instrumentation responsible for minimal exporting lifecycle ◦ Central location for most configuration changes: Collector ◦ Resource enrichment straightforward; able to translate between formats ◦ Summary: generally recommended
16. Summary: Exporting Application Instrumentation • Application instrumentation to local Collector (agent) to remote Collector (gateway) to back-end ◦ Often used for advanced use-cases such as tail-based sampling or metric aggregations ◦ May help security by limiting egress and centralizing token management ◦ Can offer more queuing and retry ◦ Summary: not required, but may provide value depending on requirements
17. Summary: Exporting Application Instrumentation • Application instrumentation to local Collector (agent) to remote Collector (gateway) to back-end ◦ Often used for advanced use-cases such as tail-based sampling or metric aggregations ◦ May help security by limiting egress and centralizing token management ◦ Can offer more queuing and retry ◦ Summary: not required, but may provide value depending on requirements • Application instrumentation to remote Collector (gateway) to back-end ◦ Pretty much same as application instrumentation direct to back-end ◦ Can offer client-side capabilities ◦ Summary: may be helpful for proof-of-concepts, be careful in production environments
18. What about serverless? NOTE: Serverless platforms offer a variety of feature sets, only common ones covered here.
19. VPC Function traces metrics logs SaaS Back-end On-prem Back-end • With serverless, signals are typically pushed or handled by cloud provider telemetry. • If instrumentation direct to back-end then every application, and language-specific, needs to: ◦ Authenticate ◦ Batch ◦ Compress ◦ CRUD metadata ◦ Encryption ◦ Proxy ◦ Queue ◦ Retry • Common challenges with instrumenting serverless include cold boot time, resource consumption, and export latency
20. VPC Function SaaS Back-end On-prem Back-end • With serverless, signals are typically pushed or handled by cloud provider telemetry. • If serverless instrumentation to collector then every application, and language-specific, needs to (everything else optional): ◦ Batch • In addition, export latency can be reduced using a Collector Collector traces metrics logs traces, metrics, logs
21. Summary: Exporting Serverless Instrumentation • Severless instrumentation direct to back-end ◦ Instrumentation responsible for full exporting lifecycle; requires even more resources ◦ Instrumentation export features are language-specific; mileage may vary ◦ Instrumentation and export latency may prove problematic ◦ Summary: generally recommended
22. Summary: Exporting Serverless Instrumentation • Severless instrumentation direct to back-end ◦ Instrumentation responsible for full exporting lifecycle; requires even more resources ◦ Instrumentation export features are language-specific; mileage may vary ◦ Instrumentation and export latency may prove problematic ◦ Summary: generally recommended • Serverless instrumentation to remote Collector to back-end ◦ Instrumentation responsible for minimal exporting lifecycle ◦ Central location for most configuration changes: Collector ◦ Summary: optional, but generally recommended
23. How about RUM?
24. Browser traces metrics logs SaaS Back-end • With RUM, signals must be pushed. • Browser instrumentation needs to: ◦ Authenticate (publicly!) ◦ Batch ◦ Compress ◦ Encrypt ◦ Queue ◦ Retry Hosted traces metrics logs • RUM data and APM data can be sent separately (stitching via context propagation)
25. Browser SaaS Back-end • With RUM, signals must be pushed. • Browser instrumentation needs to: ◦ Authenticate (publicly!) ◦ Batch ◦ Compress ◦ Encrypt ◦ Queue ◦ Retry • Also possible to centralize data collection Hosted traces metrics logs traces, metrics, logs On-prem Back-end
26. Summary: Exporting RUM Instrumentation • RUM instrumentation direct to back-end ◦ Instrumentation responsible for full exporting lifecycle; authentication often different! ◦ Summary: generally recommended
27. Summary: Exporting RUM Instrumentation • RUM instrumentation direct to back-end ◦ Instrumentation responsible for full exporting lifecycle; authentication often different! ◦ Summary: generally recommended • RUM instrumentation to remote Collector to back-end ◦ Instrumentation responsible for full exporting lifecycle; authentication often different! ◦ Collector should not be exposed to the Internet today! ◦ Summary: possible, but not recommended at this time
28. What else do I need to know?
29. OTel Me More! • Batching really matters so be sure to enable it ◦ https:/ /doordash.engineering/2021/04/07/optimizing-opentelemetrys-span-processor/
30. OTel Me More! • Batching really matters so be sure to enable it ◦ https:/ /doordash.engineering/2021/04/07/optimizing-opentelemetrys-span-processor/ • Compression really matters so be sure to enable it (~6x typical)
31. DEMO! https:/ /gist.github.com/flands/e2653e054a1ae7347a1fc68778ccbea4
32. OTel Me More! • Batching really matters so be sure to enable it ◦ https:/ /doordash.engineering/2021/04/07/optimizing-opentelemetrys-span-processor/ • Compression really matters so be sure to enable it (~6x typical) • Queuing is pay to play (more resources = more data) ◦ Instrumentation libraries: typically very small ◦ Collector as agent: moderate ◦ Collector as gateway: sky’s the limit
33. OTel Me More! • Batching really matters so be sure to enable it ◦ https:/ /doordash.engineering/2021/04/07/optimizing-opentelemetrys-span-processor/ • Compression really matters so be sure to enable it (~6x typical) • Queuing is pay to play (more resources = more data) ◦ Instrumentation libraries: typically very small ◦ Collector as agent: moderate ◦ Collector as gateway: sky’s the limit • Collector supports a wide range of use-cases ◦ CRUD metadata is very powerful ◦ Offers a lot of exporters
34. Collector Exporters... • alibabacloudlogserviceexporter • awsemfexporter • awskinesisexporter • awsprometheusremotewriteexporter • awsxrayexporter • azuremonitorexporter • carbonexporter • datadogexporter • dynatraceexporter • elasticexporter • elasticsearchexporter • f5cloudexporter • fileexporter • googlecloudexporter • honeycombexporter • humioexporter • jaegerexporter • jaegerthrifthttpexporter • kafkaexporter • loadbalancingexporter • loggingexporter • logzioexporter • lokiexporter • newrelicexporter • opencensusexporter • otlpexporter • otlphttpexporter • prometheusexporter • prometheusremotewriteexporter • sapmexporter • sentryexporter • signalfxexporter • splunkhecexporter • stackdriverexporter • sumologicexporter • uptraceexporter • zipkinexporter
35. Collector Exporters… Open-Source • alibabacloudlogserviceexporter • awsemfexporter • awskinesisexporter • awsprometheusremotewriteexporter • awsxrayexporter • azuremonitorexporter • carbonexporter • datadogexporter • dynatraceexporter • elasticexporter • elasticsearchexporter • f5cloudexporter • fileexporter • googlecloudexporter • honeycombexporter • humioexporter • jaegerexporter • jaegerthrifthttpexporter • kafkaexporter • loadbalancingexporter • loggingexporter • logzioexporter • lokiexporter • newrelicexporter • opencensusexporter • otlpexporter • otlphttpexporter • prometheusexporter • prometheusremotewriteexporter • sapmexporter • sentryexporter • signalfxexporter • splunkhecexporter • stackdriverexporter • sumologicexporter • uptraceexporter • zipkinexporter
36. Collector Exporters… Commercial • alibabacloudlogserviceexporter • awsemfexporter • awskinesisexporter • awsprometheusremotewriteexporter • awsxrayexporter • azuremonitorexporter • carbonexporter • datadogexporter • dynatraceexporter • elasticexporter • elasticsearchexporter • f5cloudexporter • fileexporter • googlecloudexporter • honeycombexporter • humioexporter • jaegerexporter • jaegerthrifthttpexporter • kafkaexporter • loadbalancingexporter • loggingexporter • logzioexporter • lokiexporter • newrelicexporter • opencensusexporter • otlpexporter • otlphttpexporter • prometheusexporter • prometheusremotewriteexporter • sapmexporter • sentryexporter • signalfxexporter • splunkhecexporter • stackdriverexporter • sumologicexporter • uptraceexporter • zipkinexporter
37. Collector Exporters… Traces • alibabacloudlogserviceexporter • awsemfexporter • awskinesisexporter • awsprometheusremotewriteexporter • awsxrayexporter • azuremonitorexporter • carbonexporter • datadogexporter • dynatraceexporter • elasticexporter • elasticsearchexporter • f5cloudexporter • fileexporter • googlecloudexporter • honeycombexporter • humioexporter • jaegerexporter • jaegerthrifthttpexporter • kafkaexporter • loadbalancingexporter • loggingexporter • logzioexporter • lokiexporter • newrelicexporter • opencensusexporter • otlpexporter • otlphttpexporter • prometheusexporter • prometheusremotewriteexporter • sapmexporter • sentryexporter • signalfxexporter • splunkhecexporter • stackdriverexporter • sumologicexporter • uptraceexporter • zipkinexporter
38. Collector Exporters… Metrics • alibabacloudlogserviceexporter • awsemfexporter • awskinesisexporter • awsprometheusremotewriteexporter • awsxrayexporter • azuremonitorexporter • carbonexporter • datadogexporter • dynatraceexporter • elasticexporter • elasticsearchexporter • f5cloudexporter • fileexporter • googlecloudexporter • honeycombexporter • humioexporter • jaegerexporter • jaegerthrifthttpexporter • kafkaexporter • loadbalancingexporter • loggingexporter • logzioexporter • lokiexporter • newrelicexporter • opencensusexporter • otlpexporter • otlphttpexporter • prometheusexporter • prometheusremotewriteexporter • sapmexporter • sentryexporter • signalfxexporter • splunkhecexporter • stackdriverexporter • sumologicexporter • uptraceexporter • zipkinexporter
39. Collector Exporters… Logs • alibabacloudlogserviceexporter • awsemfexporter • awskinesisexporter • awsprometheusremotewriteexporter • awsxrayexporter • azuremonitorexporter • carbonexporter • datadogexporter • dynatraceexporter • elasticexporter • elasticsearchexporter • f5cloudexporter • fileexporter • googlecloudexporter • honeycombexporter • humioexporter • jaegerexporter • jaegerthrifthttpexporter • kafkaexporter • loadbalancingexporter • loggingexporter • logzioexporter • lokiexporter • newrelicexporter • opencensusexporter • otlpexporter • otlphttpexporter • prometheusexporter • prometheusremotewriteexporter • sapmexporter • sentryexporter • signalfxexporter • splunkhecexporter • stackdriverexporter • sumologicexporter • uptraceexporter • zipkinexporter
40. OTel Me More! • Choose the pieces you need ◦ Already have instrumentation? Jaeger? Prometheus? Use it! ◦ Only want to use OpenTelemetry for tracing? That’s fine! ◦ Interested in semantic conventions? OTel has you covered! ◦ Looking for a unified and vendor-agnostic collector? Check! ◦ Need serverless or RUM support? No problem!
41. OTel Me More! • Choose the pieces you need ◦ Already have instrumentation? Jaeger? Prometheus? Use it! ◦ Only want to use OpenTelemetry for tracing? That’s fine! ◦ Interested in semantic conventions? OTel has you covered! ◦ Looking for a unified and vendor-agnostic collector? Check! ◦ Need serverless or RUM support? No problem! • Reference architecture is OTLP all the way! ◦ Instrumentation library to Collector as agent? OTLP ◦ Collector as agent to Collector as gateway (optional)? OTLP ◦ Sending to back-end? Ideally OTLP (limited support today https:/ /opentelemetry.io/vendors/, but expect more soon!)
42. OpenTelemetry Reference Architecture Serverless RUM
43. OTel Me More! • Open-source projects are starting to export OpenTelemetry! ◦ Ambassador: https:/ /app.getambassador.io/initializer/ ◦ Apache Camel: https:/ /camel.apache.org/components/3.7.x/others/opentelemetry.html ◦ Clickhouse: https:/ /clickhouse.tech/docs/en/operations/opentelemetry/ ◦ Dapr: https:/ /docs.dapr.io/operations/monitoring/tracing/open-telemetry-collector/ ◦ Envoy: https:/ /blog.envoyproxy.io/envoy-support-for-opentelemetry-access-logging-e4b08160d32c ◦ Jaeger: https:/ /medium.com/jaegertracing/jaeger-embraces-opentelemetry-collector-90a545cbc24 ◦ Jenkins: https:/ /plugins.jenkins.io/opentelemetry/ ◦ Open Distro for Elasticsearch: https:/ /opendistro.github.io/for-elasticsearch/blog/releases/2020/12/announcing-trace-analytics/ ◦ Spring Sleuth: https:/ /github.com/spring-cloud-incubator/spring-cloud-sleuth-otel
44. Join us!
45. 45 Next Steps • Join a SIG: https:/ /github.com/open-telemetry/community#special-interest-groups • Join the conversation: ◦ Each SIG leverages GitHub discussions ◦ CNCF Slack: https:/ /cloud-native.slack.com • Submit a PR (consider good-first-issue and help-wanted labels)
46. Thank You!