Build event-driven architectures effortlessly with AWS

Build event-driven architectures effortlessly with AWS
Session Abstract:
Many customers are choosing to build event-driven application architectures – those in which subscriber or target services automatically perform work in response to events triggered by publisher or source services. Event driven architectures enable development teams to operate more independently so they can release new features faster, while also making their applications more scalable. In this session, we will discuss how we can build event driven architectures with AWS Lambda and AWS Eventbridge along with widely used patterns.

About Speaker:
Lalit Kale is a Principal Engineer at Verizon Connect. Lalit has 16 years of experience in the software industry. His interests include large scale distributed systems and scalability of platforms.

He is a published author of a book on Microservices ("Building Microservices with .NET Core") published by Packt. He has been recognized as an AWS Community Builder by AWS for his community contributions. You can follow him on LinkedIn (https://www.linkedin.com/in/lalitkale/)

Recorded Session: https://www.youtube.com/watch?v=tDwZLiENDSM

Transcript

1. Build Event-Driven Architectures
   with AWS
   Lalit Kale
   Principal Software Engineer - Verizon Connect
   Effortlessly
   

   View Slide

2. Lalit Kale
   ● Principal Software Engineer, Verizon Connect
   ● Dublin, Ireland
   ● First AWS Use: 2006
   https://www.linkedin.com/in/lalitkale/
   @atechiethought
   

   View Slide

3. Agenda
   ● What is Event Driven Architectures
   ● AWS Serverless Application Integration Services
   ○ AWS Lambda
   ○ Selecting Right Messaging service: SQS, SNS and Eventridge
   ○ Application Integration Patterns support
   ● Amazon EventBridge
   ● Use Cases
   

   View Slide

4. Current Landscape
   Large Systems (couple of years ago)
   ● Few Kb of internet speed
   ● Couple of giant single core
   servers
   ● Seconds of response time
   ● Hours of offline maintenance
   ● Data size in Gigabytes
   Large Systems (Today & Future)
   ● GBs scale internet speed
   ● Thousands of multicore servers
   ● Milliseconds of response time
   ● Users expect 100% uptime
   ● Data size in Petabytes and
   Zetabytes
   

   View Slide

5. Reactive Systems
   

   View Slide

6. The big idea is "messaging"
   - Alan Kay
   src: https://wiki.c2.com/?AlanKayOnMessaging
   

   View Slide

7. "If your application is cloud-native or, large
   scale or, distributed, and doesn't include a
   messaging component, that is probably a
   bug."
   - Tim Bray
   Former Sr. Principal Engineer, AWS
   src: https://wiki.c2.com/?AlanKayOnMessaging
   

   View Slide

8. Communication
   Synchronous Communication Asynchronous Communication
   Face To Face Video
   Conferencing
   Email
   

   View Slide

9. Types of Communication
   Synchronous Communication Asynchronous Communication
   

   View Slide

10. Types of Messages
    Commands
    ● "I Intent to …"
    ● Can be rejected by receiver
    ● Represented as verbs in
    present tense, e.g.
    "create_customer"
    ● Source system is aware of
    side effects that command
    processing will have
    Events
    ● Something has happened
    ● Immutable—cannot change
    the past
    ● Represented as verbs in the
    past tense, e.g.
    “customer_created”
    ● Source system has no
    expectations on how an event
    is processed
    

    View Slide

11. Event Driven Architecture
    Event driven architecture is a distributed asynchronous integration pattern to create
    highly scalable and loosely coupled reactive applications
    Events drives the data processing
    

    View Slide

12. Advantages of Event driven architecture
    ● Loose coupling between components/services
    ● Ability to scale individual components
    ● Processing components can be developed independent of each other
    ● Fault Tolerance and better resiliency
    ● Improved Availability
    ● Evolutionary architecture
    ● Business Agility
    

    View Slide

13. Challenges with Event Driven Architectures
    ● Consistency Model mindshift - Eventual Consistency
    ● Observability of Event Pipelines
    ● Immutable Consumers
    ● Error Handling
    ● Event Replay Scenarios
    

    View Slide

14. Challenges with Event Driven Architectures
    ●
    

    View Slide

15. Event Driven Architecture
    Event driven architecture is a distributed asynchronous integration pattern to
    create highly scalable and loosely coupled reactive applications
    

    View Slide

16. Enterprise Integration Patterns
    ● Point to Point channel
    ● Pub-Sub pattern
    ● Message Router
    ● Event Bus
    

    View Slide

17. Event Driven Architectures with AWS Serverless
    

    View Slide

18. Serverless Story is incomplete without...
    

    View Slide

19. AWS Lambda Execution Models
    

    View Slide

20. AWS Lambda Destinations
    

    View Slide

21. AWS Serverless Application Integration Services
    

    View Slide

22. Service Selection Criteria
    Scale &
    Concurrency
    Controls
    Durability Persistence
    Consumption Retries Pricing
    

    View Slide

23. AWS Serverless Application Integration Services
    Amazon SQS
    Durable and scalable
    Buffer and Batching
    support
    Comprehensive security
    point to point channel
    Amazon SNS
    Performance at Scale
    Enterprise Ready,
    Deliver to endpoints like
    email, SMS, Https, Push
    Notifications
    Fan-Out, Pub-Sub
    Amazon EventBridge
    Serverless event bus for AWS Services,
    your own apps and SaaS providers
    Event Bus
    

    View Slide

24. Failures and Retries
    Amazon SQS Amazon SNS Amazon EventBridge
    Delete Message. Redrive
    policy for failed messages.
    Retry policy for different
    protocols
    Event Retention and Replay
    Retry upto max. 14 days Retry upto 23 days Retry upto 24 hours
    Dead Letter Queue Dead Letter Queue Dead Letter Queue
    Exactly once with FIFO Exactly Once with FIFO
    Topic
    

    View Slide

25. Amazon EventBridge
    ● Serverless, Pay as you go
    ● 90+ AWS Services as sources
    ● 17+ AWS services as targets
    ● Native integrations with SaaS Providers
    ● $1 per million events put into the bus
    ● No additional cost of delivery
    ● Cross Account, Cross Region
    

    View Slide

26. Amazon EventBridge
    

    View Slide

27. Amazon EventBridge - Events and Rules
    {
    "version": "0",
    "id": "3e3c153a-8339-4e30-8c35-687ebef853fe",
    "detail-type": "EC2 Instance Launch
    Successful",
    "source": "aws.autoscaling",
    "account": "123456789012",
    "time": "2015-11-11T21:31:47Z",
    "region": "us-east-1",
    "resources": [],
    "detail": {
    "eventVersion": "",
    "responseElements": null
    }
    }
    Example Event Example Rule
    {
    "source": ["aws.autoscaling"]
    }
    

    View Slide

28. ● Source of truth for sharing schema
    ● Explicitly published and auto-discovered
    ● Integrations for IDEs like Jetbrains and VS Code
    ● Language Bindings available for Java, Python and Typescript
    Amazon EventBridge - Schema Registry and Discovery
    

    View Slide

29. Amazon EventBridge - Event Archive and Replay
    

    View Slide

30. When Amazon EventBridge is not best fit?
    ● Ordering Guarantees
    ● Event Deduplication
    ● Push Delivery
    ● Ultra Low latency
    ● Non-JSON Transport
    

    View Slide

31. Event Modelling
    CloudEvents - A specification for describing event data in a common way
    ● Consistent schema across all your events
    ● Many Language SDK available - Java, GoLang, Python, Rust etc
    ● Portable Across AWS, GCP, Azure
    ● CNCF Serverless Group supported
    ● https://cloudevents.io/
    

    View Slide

32. Event Driven Architectures - Infrastructure as Code Support
    ● https://serverlessland.com/
    ● https://cdkpatterns.com
    ● Use patterns to quickly build
    integrations using AWS SAM
    and CDK templates.
    ● 70+ Templates
    ● You can contribute to it!
    

    View Slide

33. EventBridge - Real Time Threat Mitigation
    

    View Slide

34. Loan Broker Use case
    src: https://www.enterpriseintegrationpatterns.com/patterns/messaging/ComposedMessagingExample.html
    

    View Slide

35. ● Receive the consumer's loan quote request
    ● Obtain credit score and history from credit agency
    ● Determine the most appropriate banks to contact
    ● Send a request to each selected bank
    ● Collect responses from each selected bank
    ● Determine the best response
    ● Pass the result back to the consumer
    Loan Broker Use case - Processing Flow
    src: https://www.enterpriseintegrationpatterns.com/patterns/messaging/ComposedMessagingExample.html
    

    View Slide

36. Loan Broker Use case
    src: https://www.enterpriseintegrationpatterns.com/patterns/messaging/ComposedMessagingExample.html
    

    View Slide

37. Summary
    ● event driven architectures are essential complexity for large scale decoupled
    applications
    ● Awareness of enterprise integration patterns will ease out the journey through event
    driven architectures
    ● For choosing the aws service for your scenarios, compare scale, durability, persistence
    and consumption models, retries and pricing.
    ● Adopting AWS serverless services like EventBridge ease out complex implementations
    like content filtering and event replays and cross account, cross region scenarios.
    ● You can compose solution by combining multiple patterns and services
    ● Utilize IaC templates available in open source or build your own.
    

    View Slide

38. Further Reading
    ● http://www.sosp.org/2001/papers/welsh.pdf
    ● https://www.enterpriseintegrationpatterns.com/
    ● https://aws.amazon.com/blogs/compute/introducing-cross-region-event-routing-with-amazon-eventbridge/
    ● https://aws.amazon.com/blogs/compute/building-well-architected-serverless-applications-introduction/
    ● https://serverlessland.com/patterns
    

    View Slide

39. Q & A
    

    View Slide

40. Thank You!
    

    View Slide

41. Extras
    

    View Slide

42. Amazon EventBridge - Pricing
    AWS Service Events Free
    Custom Events $1/ million events published
    Third Party Events $1/ million events published
    Cross Account Events $1/ million cross-account events sent
    Schema Discovery $0.10 / million events ingested
    Archive Processing $0.12 per GB
    Storage Month $0.0025 per GB
    

    View Slide

43. SQS, SNS, EventBridge
    Amazon SQS Amazon SNS Amazon EventBridge
    Messaging 1 to 1 1 to Many 1 to Many
    Event Type Single Event Type Single Event Type Multiple Event Type
    Route Messages Single Single Multiple Rules
    Filter Messages No Only to Message Body Full Message
    Allow transformation? No No Yes
    Cross Account Yes Yes Yes
    Message Latency Low Low Moderate
    Throughput High High Moderate
    

    View Slide