Slide 1

Slide 1 text

B E R L I N | M A Y 1 1 – 1 2 , 2 0 2 2

Slide 2

Slide 2 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Metriken, Logs, Traces - Observability mit Open Source Dennis Kieselhorst S E C 3 0 1 Senior Solutions Architect Amazon Web Services

Slide 3

Slide 3 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Slide 4

Slide 4 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. What is observability? DETECT INVESTIGATE REMEDIATE

Slide 5

Slide 5 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data drives decisions R E A L - T I M E D A T A D R I V E S R E A L - T I M E D E C I S I O N S Logs Metrics Traces AWS monitoring and observability services help you maintain SLAs by detecting, investigating, and remediating problems such as Availability Reliability Performance

Slide 6

Slide 6 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Customer use cases for observability M E E T Y O U R O B S E R V A B I L I T Y N E E D S W I T H E N T E R P R I S E T O O L S B U I L T F O R T H E C L O U D Optimize end user application experiences with DEM Monitor with full stack observability Observe modern applications Achieve SLOs with AIOps and DevOps Consolidate monitoring tools into a single tool Leverage managed observability services

Slide 7

Slide 7 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS observability choices MONITORING & OBSERVABILITY INSTRUMENTATION CloudWatch agent AWS Distro for OpenTelemetry X-Ray agent Amazon Managed Grafana CloudWatch ServiceLens Container Insights Synthetics Lambda Insights Contributor Insights Custom-built dashboards and data correlation, collection, configuration, etc. AWS X-Ray CloudWatch Logs CloudWatch metrics Amazon OpenSearch Service Amazon Managed Service for Prometheus

Slide 8

Slide 8 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon is customer-obsessed 90–95% of Amazon and AWS roadmaps are driven by customer needs Open source matters to our customers, so it matters to us

Slide 9

Slide 9 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Working with the open source community L E V E R A G E O P E N S O U R C E W I T H O U R M A N A G E D O B S E R V A B I L I T Y S E R V I C E S Prometheus Kubernetes Cloud Native Computing Foundation

Slide 10

Slide 10 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. OpenTelemetry & AWS Distro for OpenTelemetry I N S T R U M E N T A T I O N A N D C O L L E C T I O N • OpenTelemetry ▪ Is a Cloud Native Computing Foundation (CNCF) project ▪ Provides open-source observability agents and libraries and a data protocol to unify monitoring, managing, and debugging of applications ▪ Supports all 3 data signals in 11 languages (Java, C#, Go, JavaScript, PHP, Python, Rust, C++, Erlang/Elixir, .NET, Swift) • AWS Distro for OpenTelemetry (ADOT) ▪ Is a secure, production-ready open-source distribution supported by AWS ▪ Features code contributions upstream in OpenTelemetry ▪ Is certified by AWS for security and predictability

Slide 11

Slide 11 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. OpenTelemetry reference architecture https://opentelemetry.io/docs/

Slide 12

Slide 12 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. OpenTelemetry roadmap I N S T R U M E N T A T I O N A N D C O L L E C T I O N Tracing API: stable SDK: stable Protocol: stable • Specification is stable • Covered by long-term support • Specification is still extensible, but only in a backwards compatible manner • OpenTelemetry clients are versioned to v1.0 once their tracing implementation is complete Metrics (Preview) API: stable SDK: feature-freeze Protocol: stable • Currently under active development, the data model is stable. • Collector support for Prometheus is under development, in collaboration with the Prometheus community • Metric API and SDK specification is currently being prototyped in Java, .NET, and Python Logs API: draft SDK: draft Protocol: beta • Currently under active development • Log processing for many data formats has been added to the Collector. • Log appenders are in development for many languages. • A logging SDK is currently under development. • An OpenTelemetry logging API is not currently under development. https://opentelemetry.io/status/

Slide 13

Slide 13 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tracing concepts Frontend API Gateway Trace Amazon SQS queue Amazon DynamoDB Table Function Function AWS X-Ray Segments/ OpenTelemetry Spans Sub-segments/ Spans Request Response Amazon Rekognition

Slide 14

Slide 14 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tracing concepts • Span represents an operation within a transaction. Typically encapsulates: operation name, a start and finish timestamp, attributes (a list of key-value pairs), events, a parent span identifier, links to other spans, and a SpanContext. • Trace defined implicitly by its spans. A trace can be thought of as a directed acyclic graph (DAG) of spans where the edges between spans are defined as parent/child relationships. • SpanContext represents all the information that identifies Span in the Trace and MUST be propagated to child Spans and across process boundaries. A SpanContext contains the tracing identifiers and the options that are propagated from parent to child Spans. • Tracer is responsible for tracking the currently active span. https://github.com/open-telemetry/opentelemetry- specification/blob/main/specification/overview.md#tracing-signal

Slide 15

Slide 15 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Demo Blog: https://a.co/5sIvwvL Workshop: https://observability.workshop.aws

Slide 16

Slide 16 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Slide 17

Slide 17 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Distro for OpenTelemetry Architecture Application Code Auto-Instrumentation Agent OpenTelemetry Collector OpenTelemetry Extension/Processing OpenTelemetry In-Process Exporter OpenTelemetry API/SDK Monitoring Services Clients In application process Outside application process AWS resource metadata X-Ray Trace ID Support Correlation between metrics and traces AWS SDK Support X-Ray Trace ID Support Format conversion between OT and AWS Correlation between metrics and traces Traces Metrics X-Ray CloudWatch Metrics Prometheus Partners

Slide 18

Slide 18 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. OpenTelemetry Collector https://opentelemetry.io/docs/collector/

Slide 19

Slide 19 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. OpenTelemetry Collector Deployment • Agent: A Collector instance running with the application or on the same host as the application (e.g. binary, sidecar, or daemonset). • Gateway: One or more Collector instances running as a standalone service (e.g. container or deployment) typically per cluster, datacenter or region. Components • Receivers: How to get data into the Collector; these can be push or pull based • Processors: What to do with received data • Exporters: Where to send received data; these can be push or pull based • Pipelines: Enable the components V E N D O R - A G N O S T I C W A Y T O R E C E I V E , P R O C E S S A N D E X P O R T T E L E M E T R Y D A T A https://opentelemetry.io/docs/concepts/data-collection/

Slide 20

Slide 20 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS OTel Collector Amazon Managed Service for Prometheus Workspace AWS EMF Exporter Prometheus Exporter AWS X-Ray Exporter OTLP Receiver Processor Amazon CloudWatch AWS X-Ray OpenTelemetry Collector on AWS 𝜇Service1 Amazon EKS Amazon ECS AWS Container Insights Receiver Prometheus Receiver

Slide 21

Slide 21 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Observability is key to measure and monitor the behavior of distributed applications Flexibility of open source, powered by AWS, will help in enabling observability from infrastructure to end users 01 AWS is there to support your enterprise needs – how can we help? You can operate with greater safety, ease, and efficiency with solutions built using AWS observability services 02 03

Slide 22

Slide 22 text

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Resources • Observability Workshop: https://observability.workshop.aws • Developer Portal: https://aws-otel.github.io • Overview page: http://aws.amazon.com/otel • Roadmap: https://github.com/orgs/aws-observability/projects/4 • Feature requests, bugs: https://github.com/aws-observability/aws- otel-community/issues • OpenTelemetry spec https://github.com/open- telemetry/opentelemetry-specification/tree/main/specification • Contact us on Gitter: https://gitter.im/aws-otel-distro/community

Slide 23

Slide 23 text

Thank you! © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Dennis Kieselhorst [email protected]

Slide 24

Slide 24 text

Please complete the session survey © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.