Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Event-Driven Applications on AWS

Building Event-Driven Applications on AWS

Swiss Cloud Day 2023: Presentation about common patterns, AWS services and considerations when building Event-Driven Applications.

Jérôme Van Der Linden

September 29, 2023
Tweet

More Decks by Jérôme Van Der Linden

Other Decks in Technology

Transcript

  1. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Building Event Driven Applications
    on AWS
    Senior Solutions Architect Builder
    AWS
    Scott Gerring
    Senior Solutions Architect
    AWS
    Jerome Van Der Linden
    A R C 3 0 1

    View full-size slide

  2. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Agenda
    Introduction
    Common patterns & services in event-driven architectures
    Combining patterns: better together
    Considerations with event-driven architectures

    View full-size slide

  3. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Introduction

    View full-size slide

  4. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Coupling – integration’s magic word
    Component A Component B
    Coupling is a measure of independent variability
    between connected systems
    Decoupling has a cost, both at design and runtime
    Coupling isn’t binary
    Coupling isn’t one-dimensional

    View full-size slide

  5. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    The many facets of coupling
    • Technology dependency: Java vs. C++
    • Location dependency: IP addresses, DNS
    • Data format dependency: Binary, XML, JSON, ProtoBuf, Avro
    • Data type dependency: int16, int32, string, UTF-8, null, empty
    • Semantic dependency: Name, middle name, ZIP
    • Temporal dependency: sync, async
    • Interaction style dependency: messaging, RPC, query-style (GraphQL)
    • Conversation dependency: pagination, caching, retries

    View full-size slide

  6. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    The appropriate level of
    coupling depends on the level
    of control you have over the
    endpoints
    Gregor Hohpe
    Co-author of “Enterprise Integration Patterns”

    View full-size slide

  7. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Synchronous request-response model
    Order Service Billing Service Shipping Service Inventory Service

    View full-size slide

  8. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Synchronous request-response model
    Order Service Billing Service Shipping Service Inventory Service
    Dependencies: temporal (sync), location (host), soft (signature), …

    View full-size slide

  9. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Synchronous request-response model
    Order Service Billing Service Shipping Service Inventory Service
    A failure in one of theses services…

    View full-size slide

  10. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Synchronous request-response model
    Inventory Service
    ...will impact all of the other services.
    Shipping Service
    Billing Service
    Order Service

    View full-size slide

  11. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Reducing coupling
    Reducing coupling is the act
    of reducing
    interdependency between
    components and the
    awareness each component
    must have of one another.
    Event-driven architectures
    help achieve loose coupling
    through asynchronous
    communication via events.
    Order Service Billing Service
    Shipping Service Inventory Service
    Event Broker

    View full-size slide

  12. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Key concepts of event-driven architectures
    E V E N T S
    • Events are signals that a system’s
    state has changed
    • Events occur in the past
    (e.g., OrderCreated)
    • Events are immutable (cannot be
    changed)
    • Events are observed, not directed.
    "source": "com.orders",
    "detail-type":
    "detail":
    "metadata":
    "idempotency-key":
    "data":
    "order-id"

    View full-size slide

  13. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Key concepts of event-driven architectures
    P R O D U C E R S / C O N S U M E R S / B R O K E R S

    View full-size slide

  14. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Common patterns & AWS services
    in event-driven architectures

    View full-size slide

  15. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Point-to-point messaging
    • Decreases temporal and
    location coupling
    • Resilient to receiver failure
    • Receiver control consumption
    rate
    • Only one receiver can
    consume each message

    View full-size slide

  16. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Amazon Simple Queue Service (SQS)
    Producer SQS Queue
    Decouple
    producers from
    consumers.
    Messages polled
    by consumers
    Amazon SQS
    Fully managed
    Message Queuing
    service
    Consumers
    AWS Lambda
    Container
    Instance / VM
    Standard
    • At-least once delivery
    • Best-effort ordering
    Messages are stored
    up to 14 days.
    See also Amazon MQ (Apache ActiveMQ & RabbitMQ)

    View full-size slide

  17. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Amazon Simple Queue Service (SQS)
    Producer Consumers
    AWS Lambda
    Amazon SQS
    Fully managed
    Message Queuing
    service
    SQS Queue
    Decouple
    producers from
    consumers
    Messages polled
    by consumers
    Container
    Instance / VM
    FIFO
    • Exactly once delivery
    • Keep ordering
    1
    2
    3

    View full-size slide

  18. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Event Streaming
    • Decreases temporal and
    location coupling
    • Resilient to receiver failure
    • Continuous flow of messages
    (real-time)
    • Can be processed individually
    or together

    View full-size slide

  19. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Amazon Kinesis Data Stream
    Producer
    Consumers
    AWS Lambda
    Amazon Kinesis
    Serverless
    Streaming
    service
    Shards
    Decouple
    producers from
    consumers
    Messages polled
    by consumers
    Kinesis
    Data Analytics
    Kinesis
    Data Firehose
    Container
    Instance / VM
    See also Amazon Managed Streaming for Kafka (MSK)

    View full-size slide

  20. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Pub/sub messaging (Topic)
    • Decreases temporal and
    location coupling
    • Push-based
    • Fan-out: send the same
    message to N receivers Producer

    View full-size slide

  21. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Amazon Simple Notification Service (SNS)
    Amazon SNS
    Fully managed
    publish & subscribe
    messaging service
    SNS Topic
    Decouple message
    publisher from
    subscribers with topics
    Publisher
    Messages pushed to
    subscribers
    Subscribers
    AWS Lambda
    Amazon SQS
    HTTP/S
    Email / SMS /
    Push
    Amazon Kinesis
    Data Firehose
    Standard
    • At-least once delivery
    • Best-effort ordering

    View full-size slide

  22. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Amazon Simple Notification Service (SNS)
    Amazon SNS
    Fully managed
    publish & subscribe
    messaging service
    SNS Topic
    Decouple message
    publisher from
    subscribers with topics
    Publisher
    FIFO
    Subscribers
    AWS Lambda
    Amazon SQS
    FIFO
    HTTP/S
    Email / SMS
    Amazon Kinesis
    Data Firehose
    1
    2

    View full-size slide

  23. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Pub/sub messaging (Bus)
    • Decreases temporal and
    location coupling
    • Advanced routing / filtering
    • Hub & spoke: send the same
    message to multiple receivers

    View full-size slide

  24. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Amazon EventBridge
    Amazon
    EventBridge
    Fully managed
    event bus to
    connect applications
    Event Sources
    AWS Services
    Custom events
    SaaS events Events pushed
    to targets
    Rules
    Rules filter
    and send events
    to target
    Event Bus
    Default event bus
    Custom event bus
    SaaS event bus
    Decouple event
    sources and targets
    Targets (20+)
    AWS Lambda
    Amazon SQS
    Amazon SNS
    AWS Step
    Functions
    Amazon Kinesis
    AWS Batch
    Amazon API
    Gateway
    Amazon EC2
    Instance
    Amazon ECS
    Task

    View full-size slide

  25. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Amazon EventBridge
    Event Sources
    Amazon
    EventBridge
    Fully managed
    event bus to
    connect applications
    Event Bus
    Targets (20+)
    AWS Lambda
    AWS Services
    Custom events
    SaaS events
    Default event bus
    Custom event bus
    SaaS event bus
    Rules
    AWS Step
    Functions
    AWS Batch
    Amazon API
    Gateway
    Amazon EC2
    Instance
    Amazon ECS
    Task
    Schema Registry
    Collection of event
    structures (schema)

    Schema
    Discovery
    Schema
    Publication

    View full-size slide

  26. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Choreography and Orchestration

    View full-size slide

  27. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Choreography and Orchestration

    View full-size slide

  28. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Combining patterns:
    better together

    View full-size slide

  29. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Choreography and Orchestration

    View full-size slide

  30. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Fan-out & Topic-Queue Chaining

    View full-size slide

  31. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Filtering & Routing to SQS for persistence
    Amazon
    EventBridge
    Amazon SQS

    View full-size slide

  32. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Considerations with
    event-driven architectures

    View full-size slide

  33. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Eventual consistency
    • Decentralized data management
    • If consumers don’t have data, they can
    request from producers or store they own
    version
    • Data may not be perfectly synchronized
    across the application but will ultimately
    become consistent
    • Increase availability and service autonomy
    • Data replicated across boundaries
    S T A T E S P L I T A C R O S S S Y S T E M S
    User API Email API
    Users Users
    Data replicated
    e.g. Change data capture
    We can operate autonously
    Changes will come…
    at some point

    View full-size slide

  34. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Eventual consistency
    • Asynchrony and user interface
    • Does the UI need to be instant?
    • Can you use bidirectional events
    (websockets)?
    C U S T O M E R S U S E R E X P E R I E N C E

    View full-size slide

  35. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Idempotency
    A T L E A S T O N C E D E L I V E R Y A N D D U P L I C A T I O N H A N D L I N G
    Sender Receiver
    Message
    Ack
    Message
    Ack
    Network failure
    Duplicate message
    Time-out

    View full-size slide

  36. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Idempotency
    Definition:
    • “A message that has the same effect whether it is received once or
    multiple times.” – “Enterprise Integration Patterns” (Hohpe, Woolf)
    Best practices:
    • Use an idempotency token, that uniquely identify a message,
    generated by the client and not in the payload itself.
    • Use a persistence layer in the receiver to keep track of already
    processed message (using the idempotency token as a key).
    M A K E Y O U R S E R V I C E I D E M P O T E N T

    View full-size slide

  37. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Designing events
    S P A R S E E V E N T S V S . F U L L S T A T E D E S C R I P T I O N S
    123
    10:47 AM 456
    123
    10:47 AM 456.
    Open
    $237.51

    View full-size slide

  38. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Designing events
    C O N S I D E R A T I O N S W I T H S P A R S E E V E N T S
    123
    10:47 AM 456 123
    GET
    /orders/123

    View full-size slide

  39. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Designing events
    • Consumers are more autonomous
    • More flexibility for filtering/routing with
    EventBridge
    BUT
    • Is it still enough?
    • Too much?
    • Backward compatibility
    • Bigger contract
    • Maintainability, coupling
    C O N S I D E R A T I O N S W I T H D E T A I L E D E V E N T S
    "source": "com.orders",
    "detail-type":
    "detail":
    "metadata":
    "idempotency-key":
    "data":
    ”order":
    "order-id"
    ”amount"
    ”products"
    ”user":
    ”user-id"
    ”first-name"
    ”lastName"
    ”address"
    ”email"
    Currency ?
    last-name ?
    PII

    View full-size slide

  40. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Resources
    Don’t stop here, learn more
    about event-driven
    architectures on
    https://serverlessland.com

    View full-size slide

  41. Thank you!
    © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Jerome Van Der Linden
    [email protected]
    @jeromevdl
    jeromevdl
    Scott Gerring
    [email protected]
    @scottgerring
    scottgerring

    View full-size slide

  42. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Please complete the session
    survey in the Events App
    App Store Google Play Store

    View full-size slide