Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A journey from complexity to observability

A journey from complexity to observability

AWS Community Day, Dublin, May 25th, 2023

Managing the inherent complexity of software from development to production presents a significant challenge. Drawing insights from complexity theory, we explore how dependencies can lead to unexpected effects in software applications. By employing well-bounded components and smart design patterns, we can mitigate these dependencies and promote loose coupling. Given the vast array of interacting components in modern applications, we have to make sure applications are not just working, but also easy to understand and track. Join us to learn more about how to turn complexity into clarity.

Danilo Poccia

May 25, 2023
Tweet

More Decks by Danilo Poccia

Other Decks in Programming

Transcript

  1. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    A journey from complexity
    to observability
    Danilo Poccia (he/him)
    Chief Evangelist (EMEA)
    AWS

    View Slide

  2. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Photo by Kalen Emsley on Unsplash

    View Slide

  3. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    https://github.com/SebLague/Ant-Simulation https://www.youtube.com/@SebastianLague

    View Slide

  4. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    “Complexity arises when
    the dependencies among
    the elements become
    important.”
    Scott E. Page, John H. Miller
    Complex Adaptive Systems

    View Slide

  5. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Monolithic
    Application
    Services Microservices

    View Slide

  6. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    How Amazon SQS works
    Front End
    Back End
    Metadata
    Amazon
    DynamoDB
    Load
    Manager

    View Slide

  7. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    “A complex system that
    works is invariably found
    to have evolved from a
    simple system that
    worked.”
    Gall’s Law

    View Slide

  8. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    “A complex system
    designed from scratch
    never works and cannot
    be patched up to make it
    work. You have to start
    over with a working
    simple system.”

    View Slide

  9. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    “Amazon S3 is intentionally
    built with a minimal feature set.
    The focus is on simplicity and
    robustness.”
    – Amazon S3 Press Release,
    March 14, 2006

    View Slide

  10. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Amazon S3
    8 → more than 200
    microservices
    Mai-Lan Tomsen Bukovec
    AWS Vice President (Storage, Streaming, Messaging,
    and Monitoring/Observability)

    View Slide

  11. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Teams building microservice
    architectures often find that
    integration with applications
    and external services can
    result in more monolithic and
    tightly coupled workloads

    View Slide

  12. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Coupling: Integration’s magic word
    Coupling is a measure of independent
    variability between connected systems
    Decoupling has a cost, both at design and run-time
    Coupling isn’t binary
    Coupling isn’t one-dimensional
    A B
    Source: EnterpriseIntegrationPatterns.com

    View Slide

  13. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    The many facets of coupling
    • Technology dependency: Java vs. C++
    • Location dependency: IP addresses, DNS
    • Data format dependency: Binary, XML, JSON, ProtoBuf, Avro
    • Data type dependency: int16, int32, string, UTF-8, null, empty
    • Semantic dependency: Name, middle name, ZIP
    • Temporal dependency: sync, async
    • Interaction style dependency: messaging, RPC, query-style (GraphQL)
    • Conversation dependency: pagination, caching, retries
    Source: EnterpriseIntegrationPatterns.com

    View Slide

  14. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    The appropriate level of
    coupling depends on the
    level of control you have
    over the endpoints.
    Gregor Hohpe
    Enterprise Integration Patterns

    View Slide

  15. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Synchronous request-response model
    Advantages
    • Low latency
    • Simple
    • Fail fast
    Sender Receiver
    Request
    Response

    View Slide

  16. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Synchronous request-response model
    Advantages
    • Low latency
    • Simple
    • Fail fast
    Sender Receiver
    Disadvantages
    • Receiver failure
    Request

    View Slide

  17. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Synchronous request-response model
    Advantages
    • Low latency
    • Simple
    • Fail fast
    Sender
    Sender
    Sender
    Sender
    Sender
    Request
    Request
    Request
    Request
    Request
    Disadvantages
    • Receiver failure
    • Receiver throttled
    Receiver

    View Slide

  18. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Asynchronous point-to-point model (queue)
    Sender Receiver
    M1 M2 M1 M2
    Advantages
    • Decreases temporal coupling
    • Resilient to receiver failure
    • Receiver controls consumption rate
    Queue
    Ack Ack

    View Slide

  19. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Asynchronous point-to-point model (queue)
    Sender Receiver
    M1 M2 M1
    M2
    Advantages
    • …
    • Dead-letter queue (DLQ) for errors
    Dead-letter
    queue
    Queue
    Ack Ack

    View Slide

  20. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Asynchronous point-to-point model (queue)
    Sender
    Receiver
    Receiver
    M1
    M2
    Ack
    Ack
    Advantages
    • …
    • More than one receiver consuming
    messages
    M1 M2
    Ack

    View Slide

  21. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Asynchronous point-to-point model (queue)
    Sender
    Receiver
    Receiver
    M1
    M2
    Ack
    Ack
    Advantages
    • …
    • More than one receiver consuming
    messages
    Disadvantages
    • Response correlation
    • Backlog recovery time
    • Fairness in multi-tenant systems
    M1 M2
    Ack

    View Slide

  22. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Asynchronous point-to-point model (router)
    Sender
    Receiver
    M1 M1
    Blue
    channel
    Receiver
    M2 M2
    Purple
    channel
    Blue Blue
    Purple Purple

    View Slide

  23. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Asynchronous point-to-point model (router)
    Sender
    Receiver
    M1 M1
    Blue
    channel
    Disadvantages
    • Increases location coupling
    • Sender maintains routing logic
    Receiver
    M2 M2
    Purple
    channel
    Blue Blue
    Simple
    color
    logic
    Purple Purple

    View Slide

  24. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Asynchronous point-to-point model (router)
    Sender
    Receiver
    Disadvantages
    • Increases location coupling
    • Sender maintains routing logic
    • Sender complexity increases
    with time
    Complex
    color
    logic
    Receiver
    Receiver
    Receiver
    Receiver
    M1 M2
    M1 M3
    M1 M2 M3
    M2 M3
    M1

    View Slide

  25. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Asynchronous message-router (bus)
    Sender
    Receiver
    M1
    Advantages
    • Reduces location coupling
    • Efficient for senders and receivers
    Receiver
    M2
    Blue
    Purple
    Bus
    M1 M2
    Ack
    Blue Purple
    Blue?
    Purple?

    View Slide

  26. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Event-driven architecture

    View Slide

  27. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    What is an “event” ?
    “something that
    happens”
    Events tell us a fact
    Immutable time series
    Time What
    2023-05-25 09:07:06 CustomerCreated
    2023-05-25 09:07:09 OrderCreated
    2023-05-25 09:07:13 PaymentSuccessful
    2023-05-25 09:07:17 CustomerUpdated
    . . . . . .

    View Slide

  28. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Should you focus on the current status, or what is happening?
    Current status
    Domain model
    Commands
    Control
    ”CreateAccount”
    “AddProduct”
    What happens
    Domain events
    Event-driven
    Autonomy
    ”AccountCreated”
    “ProductAdded”

    View Slide

  29. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Commands Vs Events
    Command
    Has an intent
    Directed to a target
    Personal communication
    ”CreateAccount”
    “AddProduct”
    Event
    It’s a fact
    For others to observe
    Broadcast one to many
    ”AccountCreated”
    “ProductAdded”

    View Slide

  30. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Choreograph events between domains
    using subscriptions
    Retail Fulfillment

    View Slide

  31. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Choreograph events between domains
    using subscriptions
    Retail Fulfillment
    Notify me when
    an order is created
    Subscription

    View Slide

  32. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Choreograph events between domains
    using subscriptions
    Retail Fulfillment
    Notify me when
    an order is created
    Subscription
    OrderCreated
    Notification

    View Slide

  33. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Orchestrate a business process within a domain,
    resulting in a published event
    Retail
    1
    Order up!
    Workflow
    Orchestrator
    InventoryManager InvoiceManager
    2
    Create
    order
    In
    stock?

    View Slide

  34. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    View Slide

  35. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Serverless Serverful

    View Slide

  36. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Don’t build a network of connected “black boxes”
    Building observable applications
    is a developer responsibility

    View Slide

  37. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Observability & Instrumentation
    • “Observability lets us understand a system from the outside, by
    letting us ask questions about that system without knowing its
    inner workings.”
    • “In order to be able to ask those questions of a system, the
    application must be properly instrumented. That is, the application
    code must emit signals such as traces, metrics, and logs.”
    • “An application is properly instrumented when developers don’t
    need to add more instrumentation to troubleshoot an issue,
    because they have all of the information they need.”
    https://opentelemetry.io/docs/concepts/observability-primer/#what-is-observability

    View Slide

  38. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    OpenTelemetry
    An open-source framework for instrumenting, generating,
    collecting, and exporting telemetry data

    View Slide

  39. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    What is OpenTelemetry (OTel)?
    • CNCF popular project
    • Instrument once
    • Collect and aggregate
    • Send to multiple destinations
    • Cloud-centered framework
    • Supports hundreds of
    destinations and 11
    programing languages

    View Slide

  40. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    OpenTelemetry data sources and protocol
    • Traces & Spans – What happens when a request is made
    • Metrics – A measurement about a service, captured at runtime
    • Logs – A timestamped text record with metadata
    • Baggage – Contextual information that’s passed between spans
    • OpenTelemetry Protocol (OTLP) – Telemetry data delivery protocol

    View Slide

  41. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    OpenTelemetry reference architecture
    Code
    OTel
    Instrumentation
    Destination
    1
    Destination
    2
    Destination
    N

    Receiver Exporter
    Processor
    OTel collector

    View Slide

  42. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Working with
    the open-source
    community

    View Slide

  43. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    AWS Distro for OpenTelemetry (ADOT)
    A secure, production-ready open-source distribution supported by AWS
    Certified by AWS for security and predictability
    Upstream-first distro of the popular CNCF project
    Backed by AWS support
    One-click deploy and configure from AWS container
    and AWS Lambda consoles
    Exporters for AWS monitoring solutions including –
    Amazon CloudWatch, AWS X-Ray, Amazon OpenSearch
    Service, Prometheus, SigV4Auth, and partner solutions

    View Slide

  44. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    ADOT Public Roadmap

    View Slide

  45. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Tracing with AWS auto instrumentation agent
    ADD https://github.com/aws-observability/
    aws-otel-java-instrumentation/releases/download/
    version/aws-opentelemetry-agent.jar /app/aws-opentelemetry-agent.jar
    ENV JAVA_TOOL_OPTIONS="-javaagent:/app/aws-opentelemetry-agent.jar"
    ENV OTEL_RESOURCE_ATTRIBUTES="service.name=MyApp"
    ENV OTEL_IMR_EXPORT_INTERVAL="10000”
    ENV OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
    S A M P L E D O C K E R F I L E

    View Slide

  46. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Custom tracing attributes using annotations
    public class MyObservedClass {
    public void myObservedMethod(String param1,
    long param2) {
    <...>
    }
    }
    S A M P L E J A V A C O D E

    View Slide

  47. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Custom tracing attributes using annotations
    import io.opentelemetry.extension.annotations.WithSpan;
    import io.opentelemetry.extension.annotations.SpanAttribute;
    public class MyObservedClass {
    @WithSpan
    public void myObservedMethod(@SpanAttribute("param1") String param1,
    @SpanAttribute("param2") long param2) {
    <...>
    }
    }
    S A M P L E J A V A C O D E

    View Slide

  48. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Instrument once, send to multiple destinations
    P A R T N E R S

    View Slide

  49. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Takeaways
    Complexity
    Microservices
    Integration Patterns → Loose coupling
    Observability
    OpenTelemetry
    Instrument once → Collect & analyze everywhere

    View Slide

  50. Thank you!
    © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Danilo Poccia
    @danilop

    View Slide