Upgrade to Pro — share decks privately, control downloads, hide ads and more …

O11yfest 2021: Exporting Data with OpenTelemetry by Steve Flanders

O11yfest 2021: Exporting Data with OpenTelemetry by Steve Flanders

0e8ceadb1cd8daafdbb4d3c2e083d56d?s=128

Steve Flanders

May 18, 2021
Tweet

Transcript

  1. Exporting Data with O11yfest 2021

  2. Cloud Native Telemetry foreach(language) foreach(language) collectors, sidecars, etc w3c trace-context,

    wire formats for trace data, metrics, logs, etc Telemetry “verticals” Telemetry “layers” Tracing Metrics Logs, etc Instrumentation APIs Canonical implementations Data infrastructure Interop formats
  3. @smflanders flands https:/ /sflanders.net Steve Flanders Director of Engineering, Splunk

    OpenTelemetry Website Maintainer OpenTelemetry Collector Triager CNCF SIG-Observability Member Previously: • Head of Product, Omnition • Senior Manager for Logs, VMware
  4. Components & Architecture

  5. OpenTelemetry Components API, SDK, Data Single library per language for

    all signals Receive, process, and export data Specification Instrumentation Libraries Collector
  6. Application Instrumentation Libraries Otel Inst. Library API SDK Processors Exporters

    RUM Serverless
  7. None
  8. OpenTelemetry Reference Architecture

  9. OpenTelemetry Reference Architecture

  10. But why?

  11. Application Instrumentation

  12. Host Application traces metrics logs SaaS Back-end On-prem Back-end •

    Signals may be push or pull; back-end determines whether scraping/tailing is possible • If instrumentation direct to back-end then every application, and language-specific, needs to: ◦ Authenticate ◦ Batch ◦ Compress ◦ CRUD metadata ◦ Encryption ◦ Proxy ◦ Queue ◦ Retry • In addition, host telemetry is desirable and correlation can be challenging metrics logs
  13. Host Application traces metrics logs SaaS Back-end On-prem Back-end •

    Signals may be push or pull; back-end determines whether scraping/tailing is possible • If instrumentation to collector then every application, and language-specific, needs to (everything else optional): ◦ Batch ◦ Queue ◦ Retry • In addition, the collector can capture host telemetry and correlate with application telemetry as well as convert between formats Collector metrics + logs traces metrics + logs
  14. Summary: Exporting Application Instrumentation • Application instrumentation direct to back-end

    ◦ Instrumentation responsible for full exporting lifecycle; requires even more resources ◦ Instrumentation export features are language-specific; mileage may vary ◦ Adding resource information may prove challenging or impossible ◦ Summary: great for proof-of-concepts, be careful in production environments
  15. Summary: Exporting Application Instrumentation • Application instrumentation direct to back-end

    ◦ Instrumentation responsible for full exporting lifecycle; requires even more resources ◦ Instrumentation export features are language-specific; mileage may vary ◦ Adding resource information may prove challenging or impossible ◦ Summary: great for proof-of-concepts, be careful in production environments • Application instrumentation to local Collector (agent) to back-end ◦ Instrumentation responsible for minimal exporting lifecycle ◦ Central location for most configuration changes: Collector ◦ Resource enrichment straightforward; able to translate between formats ◦ Summary: generally recommended
  16. Summary: Exporting Application Instrumentation • Application instrumentation to local Collector

    (agent) to remote Collector (gateway) to back-end ◦ Often used for advanced use-cases such as tail-based sampling or metric aggregations ◦ May help security by limiting egress and centralizing token management ◦ Can offer more queuing and retry ◦ Summary: not required, but may provide value depending on requirements
  17. Summary: Exporting Application Instrumentation • Application instrumentation to local Collector

    (agent) to remote Collector (gateway) to back-end ◦ Often used for advanced use-cases such as tail-based sampling or metric aggregations ◦ May help security by limiting egress and centralizing token management ◦ Can offer more queuing and retry ◦ Summary: not required, but may provide value depending on requirements • Application instrumentation to remote Collector (gateway) to back-end ◦ Pretty much same as application instrumentation direct to back-end ◦ Can offer client-side capabilities ◦ Summary: may be helpful for proof-of-concepts, be careful in production environments
  18. What about serverless? NOTE: Serverless platforms offer a variety of

    feature sets, only common ones covered here.
  19. VPC Function traces metrics logs SaaS Back-end On-prem Back-end •

    With serverless, signals are typically pushed or handled by cloud provider telemetry. • If instrumentation direct to back-end then every application, and language-specific, needs to: ◦ Authenticate ◦ Batch ◦ Compress ◦ CRUD metadata ◦ Encryption ◦ Proxy ◦ Queue ◦ Retry • Common challenges with instrumenting serverless include cold boot time, resource consumption, and export latency
  20. VPC Function SaaS Back-end On-prem Back-end • With serverless, signals

    are typically pushed or handled by cloud provider telemetry. • If serverless instrumentation to collector then every application, and language-specific, needs to (everything else optional): ◦ Batch • In addition, export latency can be reduced using a Collector Collector traces metrics logs traces, metrics, logs
  21. Summary: Exporting Serverless Instrumentation • Severless instrumentation direct to back-end

    ◦ Instrumentation responsible for full exporting lifecycle; requires even more resources ◦ Instrumentation export features are language-specific; mileage may vary ◦ Instrumentation and export latency may prove problematic ◦ Summary: generally recommended
  22. Summary: Exporting Serverless Instrumentation • Severless instrumentation direct to back-end

    ◦ Instrumentation responsible for full exporting lifecycle; requires even more resources ◦ Instrumentation export features are language-specific; mileage may vary ◦ Instrumentation and export latency may prove problematic ◦ Summary: generally recommended • Serverless instrumentation to remote Collector to back-end ◦ Instrumentation responsible for minimal exporting lifecycle ◦ Central location for most configuration changes: Collector ◦ Summary: optional, but generally recommended
  23. How about RUM?

  24. Browser traces metrics logs SaaS Back-end • With RUM, signals

    must be pushed. • Browser instrumentation needs to: ◦ Authenticate (publicly!) ◦ Batch ◦ Compress ◦ Encrypt ◦ Queue ◦ Retry Hosted traces metrics logs • RUM data and APM data can be sent separately (stitching via context propagation)
  25. Browser SaaS Back-end • With RUM, signals must be pushed.

    • Browser instrumentation needs to: ◦ Authenticate (publicly!) ◦ Batch ◦ Compress ◦ Encrypt ◦ Queue ◦ Retry • Also possible to centralize data collection Hosted traces metrics logs traces, metrics, logs On-prem Back-end
  26. Summary: Exporting RUM Instrumentation • RUM instrumentation direct to back-end

    ◦ Instrumentation responsible for full exporting lifecycle; authentication often different! ◦ Summary: generally recommended
  27. Summary: Exporting RUM Instrumentation • RUM instrumentation direct to back-end

    ◦ Instrumentation responsible for full exporting lifecycle; authentication often different! ◦ Summary: generally recommended • RUM instrumentation to remote Collector to back-end ◦ Instrumentation responsible for full exporting lifecycle; authentication often different! ◦ Collector should not be exposed to the Internet today! ◦ Summary: possible, but not recommended at this time
  28. What else do I need to know?

  29. OTel Me More! • Batching really matters so be sure

    to enable it ◦ https:/ /doordash.engineering/2021/04/07/optimizing-opentelemetrys-span-processor/
  30. OTel Me More! • Batching really matters so be sure

    to enable it ◦ https:/ /doordash.engineering/2021/04/07/optimizing-opentelemetrys-span-processor/ • Compression really matters so be sure to enable it (~6x typical)
  31. DEMO! https:/ /gist.github.com/flands/e2653e054a1ae7347a1fc68778ccbea4

  32. OTel Me More! • Batching really matters so be sure

    to enable it ◦ https:/ /doordash.engineering/2021/04/07/optimizing-opentelemetrys-span-processor/ • Compression really matters so be sure to enable it (~6x typical) • Queuing is pay to play (more resources = more data) ◦ Instrumentation libraries: typically very small ◦ Collector as agent: moderate ◦ Collector as gateway: sky’s the limit
  33. OTel Me More! • Batching really matters so be sure

    to enable it ◦ https:/ /doordash.engineering/2021/04/07/optimizing-opentelemetrys-span-processor/ • Compression really matters so be sure to enable it (~6x typical) • Queuing is pay to play (more resources = more data) ◦ Instrumentation libraries: typically very small ◦ Collector as agent: moderate ◦ Collector as gateway: sky’s the limit • Collector supports a wide range of use-cases ◦ CRUD metadata is very powerful ◦ Offers a lot of exporters
  34. Collector Exporters... • alibabacloudlogserviceexporter • awsemfexporter • awskinesisexporter • awsprometheusremotewriteexporter

    • awsxrayexporter • azuremonitorexporter • carbonexporter • datadogexporter • dynatraceexporter • elasticexporter • elasticsearchexporter • f5cloudexporter • fileexporter • googlecloudexporter • honeycombexporter • humioexporter • jaegerexporter • jaegerthrifthttpexporter • kafkaexporter • loadbalancingexporter • loggingexporter • logzioexporter • lokiexporter • newrelicexporter • opencensusexporter • otlpexporter • otlphttpexporter • prometheusexporter • prometheusremotewriteexporter • sapmexporter • sentryexporter • signalfxexporter • splunkhecexporter • stackdriverexporter • sumologicexporter • uptraceexporter • zipkinexporter
  35. Collector Exporters… Open-Source • alibabacloudlogserviceexporter • awsemfexporter • awskinesisexporter •

    awsprometheusremotewriteexporter • awsxrayexporter • azuremonitorexporter • carbonexporter • datadogexporter • dynatraceexporter • elasticexporter • elasticsearchexporter • f5cloudexporter • fileexporter • googlecloudexporter • honeycombexporter • humioexporter • jaegerexporter • jaegerthrifthttpexporter • kafkaexporter • loadbalancingexporter • loggingexporter • logzioexporter • lokiexporter • newrelicexporter • opencensusexporter • otlpexporter • otlphttpexporter • prometheusexporter • prometheusremotewriteexporter • sapmexporter • sentryexporter • signalfxexporter • splunkhecexporter • stackdriverexporter • sumologicexporter • uptraceexporter • zipkinexporter
  36. Collector Exporters… Commercial • alibabacloudlogserviceexporter • awsemfexporter • awskinesisexporter •

    awsprometheusremotewriteexporter • awsxrayexporter • azuremonitorexporter • carbonexporter • datadogexporter • dynatraceexporter • elasticexporter • elasticsearchexporter • f5cloudexporter • fileexporter • googlecloudexporter • honeycombexporter • humioexporter • jaegerexporter • jaegerthrifthttpexporter • kafkaexporter • loadbalancingexporter • loggingexporter • logzioexporter • lokiexporter • newrelicexporter • opencensusexporter • otlpexporter • otlphttpexporter • prometheusexporter • prometheusremotewriteexporter • sapmexporter • sentryexporter • signalfxexporter • splunkhecexporter • stackdriverexporter • sumologicexporter • uptraceexporter • zipkinexporter
  37. Collector Exporters… Traces • alibabacloudlogserviceexporter • awsemfexporter • awskinesisexporter •

    awsprometheusremotewriteexporter • awsxrayexporter • azuremonitorexporter • carbonexporter • datadogexporter • dynatraceexporter • elasticexporter • elasticsearchexporter • f5cloudexporter • fileexporter • googlecloudexporter • honeycombexporter • humioexporter • jaegerexporter • jaegerthrifthttpexporter • kafkaexporter • loadbalancingexporter • loggingexporter • logzioexporter • lokiexporter • newrelicexporter • opencensusexporter • otlpexporter • otlphttpexporter • prometheusexporter • prometheusremotewriteexporter • sapmexporter • sentryexporter • signalfxexporter • splunkhecexporter • stackdriverexporter • sumologicexporter • uptraceexporter • zipkinexporter
  38. Collector Exporters… Metrics • alibabacloudlogserviceexporter • awsemfexporter • awskinesisexporter •

    awsprometheusremotewriteexporter • awsxrayexporter • azuremonitorexporter • carbonexporter • datadogexporter • dynatraceexporter • elasticexporter • elasticsearchexporter • f5cloudexporter • fileexporter • googlecloudexporter • honeycombexporter • humioexporter • jaegerexporter • jaegerthrifthttpexporter • kafkaexporter • loadbalancingexporter • loggingexporter • logzioexporter • lokiexporter • newrelicexporter • opencensusexporter • otlpexporter • otlphttpexporter • prometheusexporter • prometheusremotewriteexporter • sapmexporter • sentryexporter • signalfxexporter • splunkhecexporter • stackdriverexporter • sumologicexporter • uptraceexporter • zipkinexporter
  39. Collector Exporters… Logs • alibabacloudlogserviceexporter • awsemfexporter • awskinesisexporter •

    awsprometheusremotewriteexporter • awsxrayexporter • azuremonitorexporter • carbonexporter • datadogexporter • dynatraceexporter • elasticexporter • elasticsearchexporter • f5cloudexporter • fileexporter • googlecloudexporter • honeycombexporter • humioexporter • jaegerexporter • jaegerthrifthttpexporter • kafkaexporter • loadbalancingexporter • loggingexporter • logzioexporter • lokiexporter • newrelicexporter • opencensusexporter • otlpexporter • otlphttpexporter • prometheusexporter • prometheusremotewriteexporter • sapmexporter • sentryexporter • signalfxexporter • splunkhecexporter • stackdriverexporter • sumologicexporter • uptraceexporter • zipkinexporter
  40. OTel Me More! • Choose the pieces you need ◦

    Already have instrumentation? Jaeger? Prometheus? Use it! ◦ Only want to use OpenTelemetry for tracing? That’s fine! ◦ Interested in semantic conventions? OTel has you covered! ◦ Looking for a unified and vendor-agnostic collector? Check! ◦ Need serverless or RUM support? No problem!
  41. OTel Me More! • Choose the pieces you need ◦

    Already have instrumentation? Jaeger? Prometheus? Use it! ◦ Only want to use OpenTelemetry for tracing? That’s fine! ◦ Interested in semantic conventions? OTel has you covered! ◦ Looking for a unified and vendor-agnostic collector? Check! ◦ Need serverless or RUM support? No problem! • Reference architecture is OTLP all the way! ◦ Instrumentation library to Collector as agent? OTLP ◦ Collector as agent to Collector as gateway (optional)? OTLP ◦ Sending to back-end? Ideally OTLP (limited support today https:/ /opentelemetry.io/vendors/, but expect more soon!)
  42. OpenTelemetry Reference Architecture Serverless RUM

  43. OTel Me More! • Open-source projects are starting to export

    OpenTelemetry! ◦ Ambassador: https:/ /app.getambassador.io/initializer/ ◦ Apache Camel: https:/ /camel.apache.org/components/3.7.x/others/opentelemetry.html ◦ Clickhouse: https:/ /clickhouse.tech/docs/en/operations/opentelemetry/ ◦ Dapr: https:/ /docs.dapr.io/operations/monitoring/tracing/open-telemetry-collector/ ◦ Envoy: https:/ /blog.envoyproxy.io/envoy-support-for-opentelemetry-access-logging-e4b08160d32c ◦ Jaeger: https:/ /medium.com/jaegertracing/jaeger-embraces-opentelemetry-collector-90a545cbc24 ◦ Jenkins: https:/ /plugins.jenkins.io/opentelemetry/ ◦ Open Distro for Elasticsearch: https:/ /opendistro.github.io/for-elasticsearch/blog/releases/2020/12/announcing-trace-analytics/ ◦ Spring Sleuth: https:/ /github.com/spring-cloud-incubator/spring-cloud-sleuth-otel
  44. Join us!

  45. 45 Next Steps • Join a SIG: https:/ /github.com/open-telemetry/community#special-interest-groups •

    Join the conversation: ◦ Each SIG leverages GitHub discussions ◦ CNCF Slack: https:/ /cloud-native.slack.com • Submit a PR (consider good-first-issue and help-wanted labels)
  46. Thank You!