A journey from complexity to observability

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Complexity arises when the dependencies among the elements become important.” Scott E. Page, John H. Miller Complex Adaptive Systems

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.”

Slide 9

Slide 9 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Amazon S3 is intentionally built with a minimal feature set. The focus is on simplicity and robustness.” – Amazon S3 Press Release, March 14, 2006

Slide 10

Slide 10 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon S3 8 → more than 200 microservices Mai-Lan Tomsen Bukovec AWS Vice President (Storage, Streaming, Messaging, and Monitoring/Observability)

Slide 11

Slide 11 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Teams building microservice architectures often find that integration with applications and external services can result in more monolithic and tightly coupled workloads

Slide 12

Slide 12 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Coupling: Integration’s magic word Coupling is a measure of independent variability between connected systems Decoupling has a cost, both at design and run-time Coupling isn’t binary Coupling isn’t one-dimensional A B Source: EnterpriseIntegrationPatterns.com

Slide 13

Slide 13 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. The many facets of coupling • Technology dependency: Java vs. C++ • Location dependency: IP addresses, DNS • Data format dependency: Binary, XML, JSON, ProtoBuf, Avro • Data type dependency: int16, int32, string, UTF-8, null, empty • Semantic dependency: Name, middle name, ZIP • Temporal dependency: sync, async • Interaction style dependency: messaging, RPC, query-style (GraphQL) • Conversation dependency: pagination, caching, retries Source: EnterpriseIntegrationPatterns.com

Slide 14

Slide 14 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. The appropriate level of coupling depends on the level of control you have over the endpoints. Gregor Hohpe Enterprise Integration Patterns

Slide 15

Slide 15 text

Slide 16

Slide 16 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Synchronous request-response model Advantages • Low latency • Simple • Fail fast Sender Receiver Disadvantages • Receiver failure Request

Slide 17

Slide 17 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Synchronous request-response model Advantages • Low latency • Simple • Fail fast Sender Sender Sender Sender Sender Request Request Request Request Request Disadvantages • Receiver failure • Receiver throttled Receiver

Slide 18

Slide 18 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Asynchronous point-to-point model (queue) Sender Receiver M1 M2 M1 M2 Advantages • Decreases temporal coupling • Resilient to receiver failure • Receiver controls consumption rate Queue Ack Ack

Slide 19

Slide 19 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Asynchronous point-to-point model (queue) Sender Receiver M1 M2 M1 M2 Advantages • … • Dead-letter queue (DLQ) for errors Dead-letter queue Queue Ack Ack

Slide 20

Slide 20 text

Slide 21

Slide 21 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Asynchronous point-to-point model (queue) Sender Receiver Receiver M1 M2 Ack Ack Advantages • … • More than one receiver consuming messages Disadvantages • Response correlation • Backlog recovery time • Fairness in multi-tenant systems M1 M2 Ack

Slide 22

Slide 22 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Asynchronous point-to-point model (router) Sender Receiver M1 M1 Blue channel Receiver M2 M2 Purple channel Blue Blue Purple Purple

Slide 23

Slide 23 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Asynchronous point-to-point model (router) Sender Receiver M1 M1 Blue channel Disadvantages • Increases location coupling • Sender maintains routing logic Receiver M2 M2 Purple channel Blue Blue Simple color logic Purple Purple

Slide 24

Slide 24 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Asynchronous point-to-point model (router) Sender Receiver Disadvantages • Increases location coupling • Sender maintains routing logic • Sender complexity increases with time Complex color logic Receiver Receiver Receiver Receiver M1 M2 M1 M3 M1 M2 M3 M2 M3 M1

Slide 25

Slide 25 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Asynchronous message-router (bus) Sender Receiver M1 Advantages • Reduces location coupling • Efficient for senders and receivers Receiver M2 Blue Purple Bus M1 M2 Ack Blue Purple Blue? Purple?

Slide 26

Slide 26 text

Slide 27

Slide 27 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. What is an “event” ? “something that happens” Events tell us a fact Immutable time series Time What 2023-05-25 09:07:06 CustomerCreated 2023-05-25 09:07:09 OrderCreated 2023-05-25 09:07:13 PaymentSuccessful 2023-05-25 09:07:17 CustomerUpdated . . . . . .

Slide 28

Slide 28 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Should you focus on the current status, or what is happening? Current status Domain model Commands Control ”CreateAccount” “AddProduct” What happens Domain events Event-driven Autonomy ”AccountCreated” “ProductAdded”

Slide 29

Slide 29 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Commands Vs Events Command Has an intent Directed to a target Personal communication ”CreateAccount” “AddProduct” Event It’s a fact For others to observe Broadcast one to many ”AccountCreated” “ProductAdded”

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Choreograph events between domains using subscriptions Retail Fulfillment Notify me when an order is created Subscription OrderCreated Notification

Slide 33

Slide 33 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Orchestrate a business process within a domain, resulting in a published event Retail 1 Order up! Workflow Orchestrator InventoryManager InvoiceManager 2 Create order In stock?

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Observability & Instrumentation • “Observability lets us understand a system from the outside, by letting us ask questions about that system without knowing its inner workings.” • “In order to be able to ask those questions of a system, the application must be properly instrumented. That is, the application code must emit signals such as traces, metrics, and logs.” • “An application is properly instrumented when developers don’t need to add more instrumentation to troubleshoot an issue, because they have all of the information they need.” https://opentelemetry.io/docs/concepts/observability-primer/#what-is-observability

Slide 38

Slide 38 text

Slide 39

Slide 39 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. What is OpenTelemetry (OTel)? • CNCF popular project • Instrument once • Collect and aggregate • Send to multiple destinations • Cloud-centered framework • Supports hundreds of destinations and 11 programing languages

Slide 40

Slide 40 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. OpenTelemetry data sources and protocol • Traces & Spans – What happens when a request is made • Metrics – A measurement about a service, captured at runtime • Logs – A timestamped text record with metadata • Baggage – Contextual information that’s passed between spans • OpenTelemetry Protocol (OTLP) – Telemetry data delivery protocol

Slide 41

Slide 41 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. OpenTelemetry reference architecture Code OTel Instrumentation Destination 1 Destination 2 Destination N … Receiver Exporter Processor OTel collector

Slide 42

Slide 42 text

Slide 43

Slide 43 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Distro for OpenTelemetry (ADOT) A secure, production-ready open-source distribution supported by AWS Certified by AWS for security and predictability Upstream-first distro of the popular CNCF project Backed by AWS support One-click deploy and configure from AWS container and AWS Lambda consoles Exporters for AWS monitoring solutions including – Amazon CloudWatch, AWS X-Ray, Amazon OpenSearch Service, Prometheus, SigV4Auth, and partner solutions

Slide 44

Slide 44 text

Slide 45

Slide 45 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tracing with AWS auto instrumentation agent ADD https://github.com/aws-observability/ aws-otel-java-instrumentation/releases/download/ version/aws-opentelemetry-agent.jar /app/aws-opentelemetry-agent.jar ENV JAVA_TOOL_OPTIONS="-javaagent:/app/aws-opentelemetry-agent.jar" ENV OTEL_RESOURCE_ATTRIBUTES="service.name=MyApp" ENV OTEL_IMR_EXPORT_INTERVAL="10000” ENV OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317" S A M P L E D O C K E R F I L E

Slide 46

Slide 46 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Custom tracing attributes using annotations public class MyObservedClass { public void myObservedMethod(String param1, long param2) { <...> } } S A M P L E J A V A C O D E

Slide 47

Slide 47 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Custom tracing attributes using annotations import io.opentelemetry.extension.annotations.WithSpan; import io.opentelemetry.extension.annotations.SpanAttribute; public class MyObservedClass { @WithSpan public void myObservedMethod(@SpanAttribute("param1") String param1, @SpanAttribute("param2") long param2) { <...> } } S A M P L E J A V A C O D E

Slide 48

Slide 48 text

Slide 49

Slide 49 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Takeaways Complexity Microservices Integration Patterns → Loose coupling Observability OpenTelemetry Instrument once → Collect & analyze everywhere