$30 off During Our Annual Pro Sale. View Details »

Building Event-Driven Applications on AWS

Building Event-Driven Applications on AWS

Swiss Cloud Day 2023: Presentation about common patterns, AWS services and considerations when building Event-Driven Applications.

Jérôme Van Der Linden

September 29, 2023
Tweet

More Decks by Jérôme Van Der Linden

Other Decks in Technology

Transcript

  1. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Building Event Driven Applications on AWS Senior Solutions Architect Builder AWS Scott Gerring Senior Solutions Architect AWS Jerome Van Der Linden A R C 3 0 1
  2. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Agenda Introduction Common patterns & services in event-driven architectures Combining patterns: better together Considerations with event-driven architectures
  3. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Coupling – integration’s magic word Component A Component B Coupling is a measure of independent variability between connected systems Decoupling has a cost, both at design and runtime Coupling isn’t binary Coupling isn’t one-dimensional
  4. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. The many facets of coupling • Technology dependency: Java vs. C++ • Location dependency: IP addresses, DNS • Data format dependency: Binary, XML, JSON, ProtoBuf, Avro • Data type dependency: int16, int32, string, UTF-8, null, empty • Semantic dependency: Name, middle name, ZIP • Temporal dependency: sync, async • Interaction style dependency: messaging, RPC, query-style (GraphQL) • Conversation dependency: pagination, caching, retries
  5. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. The appropriate level of coupling depends on the level of control you have over the endpoints Gregor Hohpe Co-author of “Enterprise Integration Patterns”
  6. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Synchronous request-response model Order Service Billing Service Shipping Service Inventory Service
  7. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Synchronous request-response model Order Service Billing Service Shipping Service Inventory Service Dependencies: temporal (sync), location (host), soft (signature), …
  8. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Synchronous request-response model Order Service Billing Service Shipping Service Inventory Service A failure in one of theses services…
  9. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Synchronous request-response model Inventory Service ...will impact all of the other services. Shipping Service Billing Service Order Service
  10. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Reducing coupling Reducing coupling is the act of reducing interdependency between components and the awareness each component must have of one another. Event-driven architectures help achieve loose coupling through asynchronous communication via events. Order Service Billing Service Shipping Service Inventory Service Event Broker
  11. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Key concepts of event-driven architectures E V E N T S • Events are signals that a system’s state has changed • Events occur in the past (e.g., OrderCreated) • Events are immutable (cannot be changed) • Events are observed, not directed. "source": "com.orders", "detail-type": "detail": "metadata": "idempotency-key": "data": "order-id"
  12. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Key concepts of event-driven architectures P R O D U C E R S / C O N S U M E R S / B R O K E R S
  13. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Common patterns & AWS services in event-driven architectures
  14. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Point-to-point messaging • Decreases temporal and location coupling • Resilient to receiver failure • Receiver control consumption rate • Only one receiver can consume each message
  15. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Amazon Simple Queue Service (SQS) Producer SQS Queue Decouple producers from consumers. Messages polled by consumers Amazon SQS Fully managed Message Queuing service Consumers AWS Lambda Container Instance / VM Standard • At-least once delivery • Best-effort ordering Messages are stored up to 14 days. See also Amazon MQ (Apache ActiveMQ & RabbitMQ)
  16. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Amazon Simple Queue Service (SQS) Producer Consumers AWS Lambda Amazon SQS Fully managed Message Queuing service SQS Queue Decouple producers from consumers Messages polled by consumers Container Instance / VM FIFO • Exactly once delivery • Keep ordering 1 2 3
  17. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Event Streaming • Decreases temporal and location coupling • Resilient to receiver failure • Continuous flow of messages (real-time) • Can be processed individually or together
  18. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Amazon Kinesis Data Stream Producer Consumers AWS Lambda Amazon Kinesis Serverless Streaming service Shards Decouple producers from consumers Messages polled by consumers Kinesis Data Analytics Kinesis Data Firehose Container Instance / VM See also Amazon Managed Streaming for Kafka (MSK)
  19. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Pub/sub messaging (Topic) • Decreases temporal and location coupling • Push-based • Fan-out: send the same message to N receivers Producer
  20. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Amazon Simple Notification Service (SNS) Amazon SNS Fully managed publish & subscribe messaging service SNS Topic Decouple message publisher from subscribers with topics Publisher Messages pushed to subscribers Subscribers AWS Lambda Amazon SQS HTTP/S Email / SMS / Push Amazon Kinesis Data Firehose Standard • At-least once delivery • Best-effort ordering
  21. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Amazon Simple Notification Service (SNS) Amazon SNS Fully managed publish & subscribe messaging service SNS Topic Decouple message publisher from subscribers with topics Publisher FIFO Subscribers AWS Lambda Amazon SQS FIFO HTTP/S Email / SMS Amazon Kinesis Data Firehose 1 2
  22. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Pub/sub messaging (Bus) • Decreases temporal and location coupling • Advanced routing / filtering • Hub & spoke: send the same message to multiple receivers
  23. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Amazon EventBridge Amazon EventBridge Fully managed event bus to connect applications Event Sources AWS Services Custom events SaaS events Events pushed to targets Rules Rules filter and send events to target Event Bus Default event bus Custom event bus SaaS event bus Decouple event sources and targets Targets (20+) AWS Lambda Amazon SQS Amazon SNS AWS Step Functions Amazon Kinesis AWS Batch Amazon API Gateway Amazon EC2 Instance Amazon ECS Task
  24. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Amazon EventBridge Event Sources Amazon EventBridge Fully managed event bus to connect applications Event Bus Targets (20+) AWS Lambda AWS Services Custom events SaaS events Default event bus Custom event bus SaaS event bus Rules AWS Step Functions AWS Batch Amazon API Gateway Amazon EC2 Instance Amazon ECS Task Schema Registry Collection of event structures (schema) … Schema Discovery Schema Publication
  25. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Choreography and Orchestration
  26. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Choreography and Orchestration
  27. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Combining patterns: better together
  28. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Choreography and Orchestration
  29. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Fan-out & Topic-Queue Chaining
  30. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Filtering & Routing to SQS for persistence Amazon EventBridge Amazon SQS
  31. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Considerations with event-driven architectures
  32. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Eventual consistency • Decentralized data management • If consumers don’t have data, they can request from producers or store they own version • Data may not be perfectly synchronized across the application but will ultimately become consistent • Increase availability and service autonomy • Data replicated across boundaries S T A T E S P L I T A C R O S S S Y S T E M S User API Email API Users Users Data replicated e.g. Change data capture We can operate autonously Changes will come… at some point
  33. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Eventual consistency • Asynchrony and user interface • Does the UI need to be instant? • Can you use bidirectional events (websockets)? C U S T O M E R S U S E R E X P E R I E N C E
  34. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Idempotency A T L E A S T O N C E D E L I V E R Y A N D D U P L I C A T I O N H A N D L I N G Sender Receiver Message Ack Message Ack Network failure Duplicate message Time-out
  35. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Idempotency Definition: • “A message that has the same effect whether it is received once or multiple times.” – “Enterprise Integration Patterns” (Hohpe, Woolf) Best practices: • Use an idempotency token, that uniquely identify a message, generated by the client and not in the payload itself. • Use a persistence layer in the receiver to keep track of already processed message (using the idempotency token as a key). M A K E Y O U R S E R V I C E I D E M P O T E N T
  36. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Designing events S P A R S E E V E N T S V S . F U L L S T A T E D E S C R I P T I O N S 123 10:47 AM 456 123 10:47 AM 456. Open $237.51
  37. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Designing events C O N S I D E R A T I O N S W I T H S P A R S E E V E N T S 123 10:47 AM 456 123 GET /orders/123
  38. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Designing events • Consumers are more autonomous • More flexibility for filtering/routing with EventBridge BUT • Is it still enough? • Too much? • Backward compatibility • Bigger contract • Maintainability, coupling C O N S I D E R A T I O N S W I T H D E T A I L E D E V E N T S "source": "com.orders", "detail-type": "detail": "metadata": "idempotency-key": "data": ”order": "order-id" ”amount" ”products" ”user": ”user-id" ”first-name" ”lastName" ”address" ”email" Currency ? last-name ? PII
  39. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Resources Don’t stop here, learn more about event-driven architectures on https://serverlessland.com
  40. Thank you! © 2023, Amazon Web Services, Inc. or its

    affiliates. All rights reserved. Jerome Van Der Linden [email protected] @jeromevdl jeromevdl Scott Gerring [email protected] @scottgerring scottgerring
  41. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Please complete the session survey in the Events App App Store Google Play Store