A journey from complexity to observability

AWS Community Day, Dublin, May 25th, 2023

Managing the inherent complexity of software from development to production presents a significant challenge. Drawing insights from complexity theory, we explore how dependencies can lead to unexpected effects in software applications. By employing well-bounded components and smart design patterns, we can mitigate these dependencies and promote loose coupling. Given the vast array of interacting components in modern applications, we have to make sure applications are not just working, but also easy to understand and track. Join us to learn more about how to turn complexity into clarity.

Transcript

1. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   A journey from complexity
   to observability
   Danilo Poccia (he/him)
   Chief Evangelist (EMEA)
   AWS
   

   View Slide

2. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   Photo by Kalen Emsley on Unsplash
   

   View Slide

3. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   https://github.com/SebLague/Ant-Simulation https://www.youtube.com/@SebastianLague
   

   View Slide

4. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   “Complexity arises when
   the dependencies among
   the elements become
   important.”
   Scott E. Page, John H. Miller
   Complex Adaptive Systems
   

   View Slide

5. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   Monolithic
   Application
   Services Microservices
   

   View Slide

6. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   How Amazon SQS works
   Front End
   Back End
   Metadata
   Amazon
   DynamoDB
   Load
   Manager
   

   View Slide

7. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   “A complex system that
   works is invariably found
   to have evolved from a
   simple system that
   worked.”
   Gall’s Law
   

   View Slide

8. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   “A complex system
   designed from scratch
   never works and cannot
   be patched up to make it
   work. You have to start
   over with a working
   simple system.”
   

   View Slide

9. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   “Amazon S3 is intentionally
   built with a minimal feature set.
   The focus is on simplicity and
   robustness.”
   – Amazon S3 Press Release,
   March 14, 2006
   

   View Slide

10. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Amazon S3
    8 → more than 200
    microservices
    Mai-Lan Tomsen Bukovec
    AWS Vice President (Storage, Streaming, Messaging,
    and Monitoring/Observability)
    

    View Slide

11. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Teams building microservice
    architectures often find that
    integration with applications
    and external services can
    result in more monolithic and
    tightly coupled workloads
    

    View Slide

12. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Coupling: Integration’s magic word
    Coupling is a measure of independent
    variability between connected systems
    Decoupling has a cost, both at design and run-time
    Coupling isn’t binary
    Coupling isn’t one-dimensional
    A B
    Source: EnterpriseIntegrationPatterns.com
    

    View Slide

13. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    The many facets of coupling
    • Technology dependency: Java vs. C++
    • Location dependency: IP addresses, DNS
    • Data format dependency: Binary, XML, JSON, ProtoBuf, Avro
    • Data type dependency: int16, int32, string, UTF-8, null, empty
    • Semantic dependency: Name, middle name, ZIP
    • Temporal dependency: sync, async
    • Interaction style dependency: messaging, RPC, query-style (GraphQL)
    • Conversation dependency: pagination, caching, retries
    Source: EnterpriseIntegrationPatterns.com
    

    View Slide

14. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    The appropriate level of
    coupling depends on the
    level of control you have
    over the endpoints.
    Gregor Hohpe
    Enterprise Integration Patterns
    

    View Slide

15. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Synchronous request-response model
    Advantages
    • Low latency
    • Simple
    • Fail fast
    Sender Receiver
    Request
    Response
    

    View Slide

16. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Synchronous request-response model
    Advantages
    • Low latency
    • Simple
    • Fail fast
    Sender Receiver
    Disadvantages
    • Receiver failure
    Request
    

    View Slide

17. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Synchronous request-response model
    Advantages
    • Low latency
    • Simple
    • Fail fast
    Sender
    Sender
    Sender
    Sender
    Sender
    Request
    Request
    Request
    Request
    Request
    Disadvantages
    • Receiver failure
    • Receiver throttled
    Receiver
    

    View Slide

18. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Asynchronous point-to-point model (queue)
    Sender Receiver
    M1 M2 M1 M2
    Advantages
    • Decreases temporal coupling
    • Resilient to receiver failure
    • Receiver controls consumption rate
    Queue
    Ack Ack
    

    View Slide

19. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Asynchronous point-to-point model (queue)
    Sender Receiver
    M1 M2 M1
    M2
    Advantages
    • …
    • Dead-letter queue (DLQ) for errors
    Dead-letter
    queue
    Queue
    Ack Ack
    

    View Slide

20. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Asynchronous point-to-point model (queue)
    Sender
    Receiver
    Receiver
    M1
    M2
    Ack
    Ack
    Advantages
    • …
    • More than one receiver consuming
    messages
    M1 M2
    Ack
    

    View Slide

21. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Asynchronous point-to-point model (queue)
    Sender
    Receiver
    Receiver
    M1
    M2
    Ack
    Ack
    Advantages
    • …
    • More than one receiver consuming
    messages
    Disadvantages
    • Response correlation
    • Backlog recovery time
    • Fairness in multi-tenant systems
    M1 M2
    Ack
    

    View Slide

22. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Asynchronous point-to-point model (router)
    Sender
    Receiver
    M1 M1
    Blue
    channel
    Receiver
    M2 M2
    Purple
    channel
    Blue Blue
    Purple Purple
    

    View Slide

23. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Asynchronous point-to-point model (router)
    Sender
    Receiver
    M1 M1
    Blue
    channel
    Disadvantages
    • Increases location coupling
    • Sender maintains routing logic
    Receiver
    M2 M2
    Purple
    channel
    Blue Blue
    Simple
    color
    logic
    Purple Purple
    

    View Slide

24. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Asynchronous point-to-point model (router)
    Sender
    Receiver
    Disadvantages
    • Increases location coupling
    • Sender maintains routing logic
    • Sender complexity increases
    with time
    Complex
    color
    logic
    Receiver
    Receiver
    Receiver
    Receiver
    M1 M2
    M1 M3
    M1 M2 M3
    M2 M3
    M1
    

    View Slide

25. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Asynchronous message-router (bus)
    Sender
    Receiver
    M1
    Advantages
    • Reduces location coupling
    • Efficient for senders and receivers
    Receiver
    M2
    Blue
    Purple
    Bus
    M1 M2
    Ack
    Blue Purple
    Blue?
    Purple?
    

    View Slide

26. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Event-driven architecture
    

    View Slide

27. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    What is an “event” ?
    “something that
    happens”
    Events tell us a fact
    Immutable time series
    Time What
    2023-05-25 09:07:06 CustomerCreated
    2023-05-25 09:07:09 OrderCreated
    2023-05-25 09:07:13 PaymentSuccessful
    2023-05-25 09:07:17 CustomerUpdated
    . . . . . .
    

    View Slide

28. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Should you focus on the current status, or what is happening?
    Current status
    Domain model
    Commands
    Control
    ”CreateAccount”
    “AddProduct”
    What happens
    Domain events
    Event-driven
    Autonomy
    ”AccountCreated”
    “ProductAdded”
    

    View Slide

29. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Commands Vs Events
    Command
    Has an intent
    Directed to a target
    Personal communication
    ”CreateAccount”
    “AddProduct”
    Event
    It’s a fact
    For others to observe
    Broadcast one to many
    ”AccountCreated”
    “ProductAdded”
    

    View Slide

30. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Choreograph events between domains
    using subscriptions
    Retail Fulfillment
    

    View Slide

31. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Choreograph events between domains
    using subscriptions
    Retail Fulfillment
    Notify me when
    an order is created
    Subscription
    

    View Slide

32. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Choreograph events between domains
    using subscriptions
    Retail Fulfillment
    Notify me when
    an order is created
    Subscription
    OrderCreated
    Notification
    

    View Slide

33. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Orchestrate a business process within a domain,
    resulting in a published event
    Retail
    1
    Order up!
    Workflow
    Orchestrator
    InventoryManager InvoiceManager
    2
    Create
    order
    In
    stock?
    

    View Slide

34. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    

    View Slide

35. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Serverless Serverful
    

    View Slide

36. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Don’t build a network of connected “black boxes”
    Building observable applications
    is a developer responsibility
    

    View Slide

37. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Observability & Instrumentation
    • “Observability lets us understand a system from the outside, by
    letting us ask questions about that system without knowing its
    inner workings.”
    • “In order to be able to ask those questions of a system, the
    application must be properly instrumented. That is, the application
    code must emit signals such as traces, metrics, and logs.”
    • “An application is properly instrumented when developers don’t
    need to add more instrumentation to troubleshoot an issue,
    because they have all of the information they need.”
    https://opentelemetry.io/docs/concepts/observability-primer/#what-is-observability
    

    View Slide

38. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    OpenTelemetry
    An open-source framework for instrumenting, generating,
    collecting, and exporting telemetry data
    

    View Slide

39. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    What is OpenTelemetry (OTel)?
    • CNCF popular project
    • Instrument once
    • Collect and aggregate
    • Send to multiple destinations
    • Cloud-centered framework
    • Supports hundreds of
    destinations and 11
    programing languages
    

    View Slide

40. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    OpenTelemetry data sources and protocol
    • Traces & Spans – What happens when a request is made
    • Metrics – A measurement about a service, captured at runtime
    • Logs – A timestamped text record with metadata
    • Baggage – Contextual information that’s passed between spans
    • OpenTelemetry Protocol (OTLP) – Telemetry data delivery protocol
    

    View Slide

41. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    OpenTelemetry reference architecture
    Code
    OTel
    Instrumentation
    Destination
    1
    Destination
    2
    Destination
    N
    …
    Receiver Exporter
    Processor
    OTel collector
    

    View Slide

42. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Working with
    the open-source
    community
    

    View Slide

43. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    AWS Distro for OpenTelemetry (ADOT)
    A secure, production-ready open-source distribution supported by AWS
    Certified by AWS for security and predictability
    Upstream-first distro of the popular CNCF project
    Backed by AWS support
    One-click deploy and configure from AWS container
    and AWS Lambda consoles
    Exporters for AWS monitoring solutions including –
    Amazon CloudWatch, AWS X-Ray, Amazon OpenSearch
    Service, Prometheus, SigV4Auth, and partner solutions
    

    View Slide

44. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    ADOT Public Roadmap
    

    View Slide

45. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Tracing with AWS auto instrumentation agent
    ADD https://github.com/aws-observability/
    aws-otel-java-instrumentation/releases/download/
    version/aws-opentelemetry-agent.jar /app/aws-opentelemetry-agent.jar
    ENV JAVA_TOOL_OPTIONS="-javaagent:/app/aws-opentelemetry-agent.jar"
    ENV OTEL_RESOURCE_ATTRIBUTES="service.name=MyApp"
    ENV OTEL_IMR_EXPORT_INTERVAL="10000”
    ENV OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
    S A M P L E D O C K E R F I L E
    

    View Slide

46. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Custom tracing attributes using annotations
    public class MyObservedClass {
    public void myObservedMethod(String param1,
    long param2) {
    <...>
    }
    }
    S A M P L E J A V A C O D E
    

    View Slide

47. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Custom tracing attributes using annotations
    import io.opentelemetry.extension.annotations.WithSpan;
    import io.opentelemetry.extension.annotations.SpanAttribute;
    public class MyObservedClass {
    @WithSpan
    public void myObservedMethod(@SpanAttribute("param1") String param1,
    @SpanAttribute("param2") long param2) {
    <...>
    }
    }
    S A M P L E J A V A C O D E
    

    View Slide

48. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Instrument once, send to multiple destinations
    P A R T N E R S
    

    View Slide

49. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Takeaways
    Complexity
    Microservices
    Integration Patterns → Loose coupling
    Observability
    OpenTelemetry
    Instrument once → Collect & analyze everywhere
    

    View Slide

50. Thank you!
    © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Danilo Poccia
    @danilop
    

    View Slide