Metrics, Logs, Traces - Observability with Open Source using OpenTelemetry (AWS Usergroup Cologne)

In cloud-native technology stacks, distributed and polyglot architectures are the norm. Distributed architectures introduce a variety of operational challenges including how to solve availability and performance issues quickly. Downtime in applications and services may cost a lot of money. Therefore, it becomes critical to have full visibility of the state of your workloads and, as important, to be able to act on it as close to real-time as possible. These challenges have led to the rise of observability. Telemetry data is needed to power observability products. Traditionally, telemetry data has been provided by either open-source projects or commercial vendors. With a lack of standardization, the net result is the lack of data portability and the burden on the user to maintain the instrumentation.

This session introduces the OpenTelemetry project which solves these problems by providing a single, vendor-agnostic solution. We'll take a look at the different parts of the specification and how OpenTelemetry integrates with Java, popular frameworks and libraries. Finally we look at the interaction with Cloud infrastructure and how the AWS Distro for OpenTelemetry (ADOT) simplifies the integration with AWS monitoring and observability services such as Amazon CloudWatch and AWS X-Ray.

Transcript

1. CLOUDLAND 2022 - OPENTELEMETRY
   © 2022, Amazon Web Services, Inc. or its affiliates.
   © 2022, Amazon Web Services, Inc. or its affiliates.
   Metrics, Logs, Traces
   Observability with Open Source
   using OpenTelemetry
   Dennis Kieselhorst
   Sr. Solutions Architect
   Amazon Web Services
   

   View Slide

2. CLOUDLAND 2022 - OPENTELEMETRY
   © 2022, Amazon Web Services, Inc. or its affiliates.
   

   View Slide

3. CLOUDLAND 2022 - OPENTELEMETRY
   © 2022, Amazon Web Services, Inc. or its affiliates.
   What is observability?
   DETECT
   INVESTIGATE
   REMEDIATE
   

   View Slide

4. CLOUDLAND 2022 - OPENTELEMETRY
   © 2022, Amazon Web Services, Inc. or its affiliates.
   Foundation for observability
   METRICS
   TRACES
   LOGS
   

   View Slide

5. CLOUDLAND 2022 - OPENTELEMETRY
   © 2022, Amazon Web Services, Inc. or its affiliates.
   Foundation for observability
   METRICS
   TRACES
   LOGS
   ✓Discrete events
   ✓Usually timestamped
   ✓Can be structured
   

   View Slide

6. CLOUDLAND 2022 - OPENTELEMETRY
   © 2022, Amazon Web Services, Inc. or its affiliates.
   Foundation for observability
   METRICS
   TRACES
   LOGS
   ✓ Regularly sampled data
   points (timestamped)
   ✓ Numeric value
   ✓ Dimension/Labels
   grafana.com/grafana/dashboards/1244
   

   View Slide

7. CLOUDLAND 2022 - OPENTELEMETRY
   © 2022, Amazon Web Services, Inc. or its affiliates.
   Foundation for observability
   METRICS
   TRACES
   LOGS
   jaegertracing.io/docs/1.18/
   Request path across
   a number of services
   

   View Slide

8. CLOUDLAND 2022 - OPENTELEMETRY
   © 2022, Amazon Web Services, Inc. or its affiliates.
   CNCF Cloud Native Landscape
   http://l.cncf.io
   

   View Slide

9. CLOUDLAND 2022 - OPENTELEMETRY
   © 2022, Amazon Web Services, Inc. or its affiliates.
   CNCF Cloud Native Landscape
   http://l.cncf.io
   

   View Slide

10. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    CNCF end-user technology radar
    „1. The most
    commonly adopted
    tools are open source.
    2. There’s no
    consolidation in the
    observability space.
    3. Prometheus and
    Grafana are
    frequently used
    together.”
    https://radar.cncf.io/2020-09-observability
    https://www.cncf.io/blog/2020/09/11/cncf-end-user-technology-radar-observability-september-2020/
    

    View Slide

11. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    What is OpenTelemetry
    OpenTelemetry is a collection of tools, APIs, and SDKs.
    You can use it to instrument, generate, collect, and export
    telemetry data (metrics, logs, and traces) for analysis in order to
    understand your software's performance and behavior.
    A N O B S E R V A B I L I T Y F R A M E W O R K F O R C L O U D - N A T I V E S O F T W A R E
    Broad Language Support
    Java | C# | Go | JavaScript | PHP | Python |
    Rust | C++ | Erlang/Elixir | .NET | Swift
    Integrates With Popular
    Frameworks and Libraries
    MySQL | Redis | Django | Kafka | Jetty | Akka |
    RabbitMQ | Spring | Flask | net/http | gorilla/mux |
    WSGI | JDBC | PostgreSQL
    https://opentelemetry.io
    

    View Slide

12. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    OpenTelemetry Reference architecture
    https://opentelemetry.io/docs/
    

    View Slide

13. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    OpenTelemetry status
    METRICS
    TRACES
    LOGS
    API: stable
    SDK: mixed
    Protocol: stable
    https://opentelemetry.io/status/
    API: draft
    SDK: draft
    Protocol: stable
    API: stable
    SDK: stable
    Protocol: stable
    

    View Slide

14. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Tracing concepts
    Frontend API Gateway
    Trace
    Amazon
    SQS queue
    Amazon
    DynamoDB
    Table
    Function
    Function
    AWS X-Ray Segments/ OpenTelemetry Spans
    Sub-segments/ Spans
    Request
    Response
    Amazon
    Rekognition
    

    View Slide

15. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Tracing concepts in a nutshell
    • Span represents an operation within a transaction. Typically encapsulates:
    operation name, a start and finish timestamp, attributes (a list of key-value
    pairs), events, a parent span identifier, links to other spans, and a
    SpanContext.
    • Trace defined implicitly by its spans. A trace can be thought of as a directed
    acyclic graph (DAG) of spans where the edges between spans are defined as
    parent/child relationships.
    • SpanContext represents all the information that identifies Span in the Trace
    and MUST be propagated to child Spans and across process boundaries. A
    SpanContext contains the tracing identifiers and the options that are
    propagated from parent to child Spans.
    • Tracer is responsible for tracking the currently active span.
    https://github.com/open-telemetry/opentelemetry-
    specification/blob/main/specification/overview.md#tracing-signal
    

    View Slide

16. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates. 19
    Demo
    Blog: https://a.co/5sIvwvL
    Workshop: https://observability.workshop.aws
    

    View Slide

17. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    

    View Slide

18. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Getting started: OpenTelemetry with Java applications
    Manual instrumentation
    opentelemetry-api and
    opentelemetry-sdk libraries
    available to e.g. create spans
    programmatically
    https://opentelemetry.io/docs/java/manual_instrumentation/
    Auto instrumentation
    Java agent attached to the JVM
    https://github.com/open-telemetry/opentelemetry-java-
    instrumentation#getting-started
    https://github.com/open-telemetry/opentelemetry-java-
    instrumentation/blob/main/docs/supported-libraries.md
    https://opentelemetry.io/docs/java/
    

    View Slide

19. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Collector
    Deployment
    • Agent: A Collector instance running with the application or on the same host as the
    application (e.g. binary, sidecar, or daemonset).
    • Gateway: One or more Collector instances running as a standalone service (e.g.
    container or deployment) typically per cluster, datacenter or region.
    Components
    • Receivers: How to get data into the Collector; these can be push or pull based
    • Processors: What to do with received data
    • Exporters: Where to send received data; these can be push or pull based
    • Pipelines: Enable the components
    V E N D O R - A G N O S T I C W A Y T O R E C E I V E , P R O C E S S A N D E X P O R T T E L E M E T R Y D A T A
    https://opentelemetry.io/docs/concepts/data-collection/
    

    View Slide

20. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Collector setup in the demo
    

    View Slide

21. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    OpenTelemetry Collector
    https://opentelemetry.io/docs/collector/
    

    View Slide

22. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Collector configuration
    https://github.com/aws-observability/aws-otel-
    lambda/blob/main/adot/collector/config.yaml
    https://github.com/open-
    telemetry/opentelemetry-
    collector/blob/main/examples/de
    mo/otel-collector-config.yaml
    https://opentelemetry.io/docs/collector/configuration/
    

    View Slide

23. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    AWS Distro for OpenTelemetry
    A S E C U R E , O P E N S O U R C E D I S T R I B U T I O N S U P P O R T E D B Y A W S
    33
    • Upstream-first distro of the popular CNCF project
    • Certified by AWS for security and predictability
    • Backed by AWS support
    • One-click deploy and configure from Amazon ECS and
    AWS Lambda consoles
    • Exporters for AWS monitoring solutions including –
    Amazon Managed Service for Prometheus, Amazon
    CloudWatch, AWS X-Ray, Amazon OpenSearch Service,
    and 3rd party Partner solutions
    

    View Slide

24. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    AWS OTel Collector
    Amazon Managed
    Service for Prometheus
    Workspace
    AWS EMF
    Exporter
    Prometheus
    Exporter
    AWS X-Ray
    Exporter
    OTLP Receiver
    Processor
    Amazon CloudWatch
    AWS X-Ray
    Example: OpenTelemetry Collector on AWS
    𝜇Service1
    Amazon Elastic
    Kubernetes Service (EKS)
    Amazon Elastic
    Container Service (ECS)
    AWS Container
    Insights Receiver
    Prometheus
    Receiver
    

    View Slide

25. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Q&A
    39
    

    View Slide

26. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Resources
    • Observability Workshop: https://observability.workshop.aws
    • Developer Portal: https://aws-otel.github.io
    • Overview page: http://aws.amazon.com/otel
    • Roadmap: https://github.com/orgs/aws-observability/projects/4
    • Feature requests, bugs: https://github.com/aws-
    observability/aws-otel-community/issues
    • OpenTelemetry spec https://github.com/open-
    telemetry/opentelemetry-specification/tree/main/specification
    • Contact us on Gitter: https://gitter.im/aws-otel-
    distro/community
    

    View Slide

27. CLOUDLAND 2022 - OPENTELEMETRY
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Thank you!
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Dennis Kieselhorst
    linkedin.com/in/kieselhorst/
    

    View Slide