Building Event-Driven Applications on AWS

Slide 1

Slide 1 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Building Event Driven Applications on AWS Senior Solutions Architect Builder AWS Scott Gerring Senior Solutions Architect AWS Jerome Van Der Linden A R C 3 0 1

Slide 2

Slide 2 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Introduction Common patterns & services in event-driven architectures Combining patterns: better together Considerations with event-driven architectures

Slide 3

Slide 3 text

Slide 4

Slide 4 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Coupling – integration’s magic word Component A Component B Coupling is a measure of independent variability between connected systems Decoupling has a cost, both at design and runtime Coupling isn’t binary Coupling isn’t one-dimensional

Slide 5

Slide 5 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. The many facets of coupling • Technology dependency: Java vs. C++ • Location dependency: IP addresses, DNS • Data format dependency: Binary, XML, JSON, ProtoBuf, Avro • Data type dependency: int16, int32, string, UTF-8, null, empty • Semantic dependency: Name, middle name, ZIP • Temporal dependency: sync, async • Interaction style dependency: messaging, RPC, query-style (GraphQL) • Conversation dependency: pagination, caching, retries

Slide 6

Slide 6 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. The appropriate level of coupling depends on the level of control you have over the endpoints Gregor Hohpe Co-author of “Enterprise Integration Patterns”

Slide 7

Slide 7 text

Slide 8

Slide 8 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Synchronous request-response model Order Service Billing Service Shipping Service Inventory Service Dependencies: temporal (sync), location (host), soft (signature), …

Slide 9

Slide 9 text

Slide 10

Slide 10 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Synchronous request-response model Inventory Service ...will impact all of the other services. Shipping Service Billing Service Order Service

Slide 11

Slide 11 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Reducing coupling Reducing coupling is the act of reducing interdependency between components and the awareness each component must have of one another. Event-driven architectures help achieve loose coupling through asynchronous communication via events. Order Service Billing Service Shipping Service Inventory Service Event Broker

Slide 12

Slide 12 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Key concepts of event-driven architectures E V E N T S • Events are signals that a system’s state has changed • Events occur in the past (e.g., OrderCreated) • Events are immutable (cannot be changed) • Events are observed, not directed. "source": "com.orders", "detail-type": "detail": "metadata": "idempotency-key": "data": "order-id"

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Point-to-point messaging • Decreases temporal and location coupling • Resilient to receiver failure • Receiver control consumption rate • Only one receiver can consume each message

Slide 16

Slide 16 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Simple Queue Service (SQS) Producer SQS Queue Decouple producers from consumers. Messages polled by consumers Amazon SQS Fully managed Message Queuing service Consumers AWS Lambda Container Instance / VM Standard • At-least once delivery • Best-effort ordering Messages are stored up to 14 days. See also Amazon MQ (Apache ActiveMQ & RabbitMQ)

Slide 17

Slide 17 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Simple Queue Service (SQS) Producer Consumers AWS Lambda Amazon SQS Fully managed Message Queuing service SQS Queue Decouple producers from consumers Messages polled by consumers Container Instance / VM FIFO • Exactly once delivery • Keep ordering 1 2 3

Slide 18

Slide 18 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Event Streaming • Decreases temporal and location coupling • Resilient to receiver failure • Continuous flow of messages (real-time) • Can be processed individually or together

Slide 19

Slide 19 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Kinesis Data Stream Producer Consumers AWS Lambda Amazon Kinesis Serverless Streaming service Shards Decouple producers from consumers Messages polled by consumers Kinesis Data Analytics Kinesis Data Firehose Container Instance / VM See also Amazon Managed Streaming for Kafka (MSK)

Slide 20

Slide 20 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Pub/sub messaging (Topic) • Decreases temporal and location coupling • Push-based • Fan-out: send the same message to N receivers Producer

Slide 21

Slide 21 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Simple Notification Service (SNS) Amazon SNS Fully managed publish & subscribe messaging service SNS Topic Decouple message publisher from subscribers with topics Publisher Messages pushed to subscribers Subscribers AWS Lambda Amazon SQS HTTP/S Email / SMS / Push Amazon Kinesis Data Firehose Standard • At-least once delivery • Best-effort ordering

Slide 22

Slide 22 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Simple Notification Service (SNS) Amazon SNS Fully managed publish & subscribe messaging service SNS Topic Decouple message publisher from subscribers with topics Publisher FIFO Subscribers AWS Lambda Amazon SQS FIFO HTTP/S Email / SMS Amazon Kinesis Data Firehose 1 2

Slide 23

Slide 23 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Pub/sub messaging (Bus) • Decreases temporal and location coupling • Advanced routing / filtering • Hub & spoke: send the same message to multiple receivers

Slide 24

Slide 24 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon EventBridge Amazon EventBridge Fully managed event bus to connect applications Event Sources AWS Services Custom events SaaS events Events pushed to targets Rules Rules filter and send events to target Event Bus Default event bus Custom event bus SaaS event bus Decouple event sources and targets Targets (20+) AWS Lambda Amazon SQS Amazon SNS AWS Step Functions Amazon Kinesis AWS Batch Amazon API Gateway Amazon EC2 Instance Amazon ECS Task

Slide 25

Slide 25 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon EventBridge Event Sources Amazon EventBridge Fully managed event bus to connect applications Event Bus Targets (20+) AWS Lambda AWS Services Custom events SaaS events Default event bus Custom event bus SaaS event bus Rules AWS Step Functions AWS Batch Amazon API Gateway Amazon EC2 Instance Amazon ECS Task Schema Registry Collection of event structures (schema) … Schema Discovery Schema Publication

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Eventual consistency • Decentralized data management • If consumers don’t have data, they can request from producers or store they own version • Data may not be perfectly synchronized across the application but will ultimately become consistent • Increase availability and service autonomy • Data replicated across boundaries S T A T E S P L I T A C R O S S S Y S T E M S User API Email API Users Users Data replicated e.g. Change data capture We can operate autonously Changes will come… at some point

Slide 34

Slide 34 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Eventual consistency • Asynchrony and user interface • Does the UI need to be instant? • Can you use bidirectional events (websockets)? C U S T O M E R S U S E R E X P E R I E N C E

Slide 35

Slide 35 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Idempotency A T L E A S T O N C E D E L I V E R Y A N D D U P L I C A T I O N H A N D L I N G Sender Receiver Message Ack Message Ack Network failure Duplicate message Time-out

Slide 36

Slide 36 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Idempotency Definition: • “A message that has the same effect whether it is received once or multiple times.” – “Enterprise Integration Patterns” (Hohpe, Woolf) Best practices: • Use an idempotency token, that uniquely identify a message, generated by the client and not in the payload itself. • Use a persistence layer in the receiver to keep track of already processed message (using the idempotency token as a key). M A K E Y O U R S E R V I C E I D E M P O T E N T

Slide 37

Slide 37 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Designing events S P A R S E E V E N T S V S . F U L L S T A T E D E S C R I P T I O N S 123 10:47 AM 456 123 10:47 AM 456. Open $237.51

Slide 38

Slide 38 text

Slide 39

Slide 39 text

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Designing events • Consumers are more autonomous • More flexibility for filtering/routing with EventBridge BUT • Is it still enough? • Too much? • Backward compatibility • Bigger contract • Maintainability, coupling C O N S I D E R A T I O N S W I T H D E T A I L E D E V E N T S "source": "com.orders", "detail-type": "detail": "metadata": "idempotency-key": "data": ”order": "order-id" ”amount" ”products" ”user": ”user-id" ”first-name" ”lastName" ”address" ”email" Currency ? last-name ? PII

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Thank you! © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Jerome Van Der Linden jvdl@amazon.ch @jeromevdl jeromevdl Scott Gerring gerrings@amazon.ch @scottgerring scottgerring