OpenTelemetry at AWS

@rakyll OpenTelemetry at AWS Jaana Dogan Principal Engineer, AWS jbd@amazon.com

@rakyll Who? Jaana Dogan, AWS Explicit focus on instrumentation

@rakyll Five AWS stories... Too many agents Too many formats
Too little correlation Too many ways to propagate Too many products to support

@rakyll Too many agents 4-5 agents Friction in installation Operational
burden Friction in configuration delivery Performance penalty

@rakyll Too many formats EMF CloudWatch Prometheus statsd Vendor formats
... X-Ray Zipkin Jaeger Vendor formats ...

@rakyll Too little correlation Tool fatigue Disjoint views Missing metadata
Friction in troubleshooting

@rakyll Too many ways to propagate Lack of end-to-end traces
Missing label propagation No W3C TraceContext or B3 support No runtime propagation standards

@rakyll Too many products to support CloudWatch X-Ray Prometheus Elasticsearch/OpenSearch
New Relic, Datadog, Splunk, Honeycomb, Lightstep and more.

@rakyll What do we use? Specification Context Propagation Semantic Conventions
Data Model Protocol (OTLP) Collector Client Libraries

@rakyll What’s next? collector Managed on EC2, ECS, EKS, Lambda,
etc.

etc. OTLP Prometheus statsd X-Ray Jaeger Zipkin

etc. OTLP Prometheus statsd X-Ray Jaeger Zipkin CloudWatch Prometheus X-Ray Elastic/OpenSearch Jaeger Zipkin Vendors Raw storage

etc. OTLP Prometheus statsd X-Ray Jaeger Zipkin CloudWatch Prometheus X-Ray Jaeger Zipkin Vendors Raw storage enrich, transform, ...

@rakyll Container Insights now collected by OpenTelemetry.

@rakyll What do we use? Specification Context Propagation Semantic Conventions
Data Model Protocol (OTLP) Collector Client Libraries

@rakyll What works well? Flexible Composable Lightweight enough Holistic Legacy
protocol friendly Community

@rakyll What challenges us? Stability Custom builds Compatibility (Prometheus &
CloudWatch) Boilerplate in client libraries

@rakyll What are we working on next?

@rakyll Prometheus

@rakyll Prometheus Drop-in replacement for Prometheus Data model changes Remote
write compliance Discovery + scrape config compliance Kubernetes operator

@rakyll Components Container Insights receivers and processors CloudWatch histogram compatibility
CloudWatch Logs exporter S3 exporter

@rakyll Propagation Adopting 128-bit trace IDs in X-Ray Context propagation
in SQL

@rakyll Platforms EC2 ECS EKS Lambda (and control plane components...)

@rakyll Lambda support

@rakyll Others... eBPF Profiles Real time user monitoring Network diagnostics
Database performance

@rakyll One more thing...

@rakyll Exporting to vendors? Vended data streams CloudWatch Metric Streams
support OTLP CW Metrics S3 (in JSON or OTLP) Kinesis (in JSON or OTLP)

@rakyll It’s not a fork. It’s a snapshot for security,
performance, support.

@rakyll Thank you Jaana Dogan Principal Engineer, AWS jbd@amazon.com

OpenTelemetry at AWS

OpenTelemetry at AWS

JBD

More Decks by JBD

Other Decks in Programming

Featured

Transcript

@rakyll OpenTelemetry at AWS Jaana Dogan Principal Engineer, AWS jbd@amazon.com

@rakyll Who? Jaana Dogan, AWS Explicit focus on instrumentation

@rakyll Five AWS stories... Too many agents Too many formats

@rakyll Too many agents 4-5 agents Friction in installation Operational

@rakyll Too many formats EMF CloudWatch Prometheus statsd Vendor formats

@rakyll Too little correlation Tool fatigue Disjoint views Missing metadata

@rakyll Too many ways to propagate Lack of end-to-end traces

@rakyll Too many products to support CloudWatch X-Ray Prometheus Elasticsearch/OpenSearch

@rakyll What do we use? Specification Context Propagation Semantic Conventions

@rakyll What’s next? collector Managed on EC2, ECS, EKS, Lambda,

@rakyll What’s next? collector Managed on EC2, ECS, EKS, Lambda,

@rakyll What’s next? collector Managed on EC2, ECS, EKS, Lambda,

@rakyll What’s next? collector Managed on EC2, ECS, EKS, Lambda,

@rakyll What’s next? collector Managed on EC2, ECS, EKS, Lambda,

@rakyll Container Insights now collected by OpenTelemetry.

@rakyll What do we use? Specification Context Propagation Semantic Conventions

@rakyll What works well? Flexible Composable Lightweight enough Holistic Legacy

@rakyll What challenges us? Stability Custom builds Compatibility (Prometheus &

@rakyll What are we working on next?

@rakyll Prometheus

@rakyll Prometheus Drop-in replacement for Prometheus Data model changes Remote

@rakyll Components Container Insights receivers and processors CloudWatch histogram compatibility

@rakyll Propagation Adopting 128-bit trace IDs in X-Ray Context propagation

@rakyll Platforms EC2 ECS EKS Lambda (and control plane components...)

@rakyll Lambda support

@rakyll Others... eBPF Profiles Real time user monitoring Network diagnostics

@rakyll One more thing...

@rakyll Exporting to vendors? Vended data streams CloudWatch Metric Streams

@rakyll It’s not a fork. It’s a snapshot for security,

@rakyll Thank you Jaana Dogan Principal Engineer, AWS jbd@amazon.com