Build event-driven architectures effortlessly with AWS

Build event-driven architectures effortlessly with AWS
Session Abstract:
Many customers are choosing to build event-driven application architectures – those in which subscriber or target services automatically perform work in response to events triggered by publisher or source services. Event driven architectures enable development teams to operate more independently so they can release new features faster, while also making their applications more scalable. In this session, we will discuss how we can build event driven architectures with AWS Lambda and AWS Eventbridge along with widely used patterns.

About Speaker:
Lalit Kale is a Principal Engineer at Verizon Connect. Lalit has 16 years of experience in the software industry. His interests include large scale distributed systems and scalability of platforms.

He is a published author of a book on Microservices ("Building Microservices with .NET Core") published by Packt. He has been recognized as an AWS Community Builder by AWS for his community contributions. You can follow him on LinkedIn (https://www.linkedin.com/in/lalitkale/)

Recorded Session: https://www.youtube.com/watch?v=tDwZLiENDSM

Transcript

1. Build Event-Driven Architectures with AWS Lalit Kale Principal Software Engineer - Verizon Connect Effortlessly
2. Lalit Kale • Principal Software Engineer, Verizon Connect • Dublin, Ireland • First AWS Use: 2006 https://www.linkedin.com/in/lalitkale/ @atechiethought
3. Agenda • What is Event Driven Architectures • AWS Serverless Application Integration Services ◦ AWS Lambda ◦ Selecting Right Messaging service: SQS, SNS and Eventridge ◦ Application Integration Patterns support • Amazon EventBridge • Use Cases
4. Current Landscape Large Systems (couple of years ago) • Few Kb of internet speed • Couple of giant single core servers • Seconds of response time • Hours of offline maintenance • Data size in Gigabytes Large Systems (Today & Future) • GBs scale internet speed • Thousands of multicore servers • Milliseconds of response time • Users expect 100% uptime • Data size in Petabytes and Zetabytes
5. Reactive Systems
6. The big idea is "messaging" - Alan Kay src: https://wiki.c2.com/?AlanKayOnMessaging
7. "If your application is cloud-native or, large scale or, distributed, and doesn't include a messaging component, that is probably a bug." - Tim Bray Former Sr. Principal Engineer, AWS src: https://wiki.c2.com/?AlanKayOnMessaging
8. Communication Synchronous Communication Asynchronous Communication Face To Face Video Conferencing Email
9. Types of Communication Synchronous Communication Asynchronous Communication
10. Types of Messages Commands • "I Intent to …" • Can be rejected by receiver • Represented as verbs in present tense, e.g. "create_customer" • Source system is aware of side effects that command processing will have Events • Something has happened • Immutable—cannot change the past • Represented as verbs in the past tense, e.g. “customer_created” • Source system has no expectations on how an event is processed
11. Event Driven Architecture Event driven architecture is a distributed asynchronous integration pattern to create highly scalable and loosely coupled reactive applications Events drives the data processing
12. Advantages of Event driven architecture • Loose coupling between components/services • Ability to scale individual components • Processing components can be developed independent of each other • Fault Tolerance and better resiliency • Improved Availability • Evolutionary architecture • Business Agility
13. Challenges with Event Driven Architectures • Consistency Model mindshift - Eventual Consistency • Observability of Event Pipelines • Immutable Consumers • Error Handling • Event Replay Scenarios
14. Challenges with Event Driven Architectures •
15. Event Driven Architecture Event driven architecture is a distributed asynchronous integration pattern to create highly scalable and loosely coupled reactive applications
16. Enterprise Integration Patterns • Point to Point channel • Pub-Sub pattern • Message Router • Event Bus
17. Event Driven Architectures with AWS Serverless
18. Serverless Story is incomplete without...
19. AWS Lambda Execution Models
20. AWS Lambda Destinations
21. AWS Serverless Application Integration Services
22. Service Selection Criteria Scale & Concurrency Controls Durability Persistence Consumption Retries Pricing
23. AWS Serverless Application Integration Services Amazon SQS Durable and scalable Buffer and Batching support Comprehensive security point to point channel Amazon SNS Performance at Scale Enterprise Ready, Deliver to endpoints like email, SMS, Https, Push Notifications Fan-Out, Pub-Sub Amazon EventBridge Serverless event bus for AWS Services, your own apps and SaaS providers Event Bus
24. Failures and Retries Amazon SQS Amazon SNS Amazon EventBridge Delete Message. Redrive policy for failed messages. Retry policy for different protocols Event Retention and Replay Retry upto max. 14 days Retry upto 23 days Retry upto 24 hours Dead Letter Queue Dead Letter Queue Dead Letter Queue Exactly once with FIFO Exactly Once with FIFO Topic
25. Amazon EventBridge • Serverless, Pay as you go • 90+ AWS Services as sources • 17+ AWS services as targets • Native integrations with SaaS Providers • $1 per million events put into the bus • No additional cost of delivery • Cross Account, Cross Region
26. Amazon EventBridge
27. Amazon EventBridge - Events and Rules { "version": "0", "id": "3e3c153a-8339-4e30-8c35-687ebef853fe", "detail-type": "EC2 Instance Launch Successful", "source": "aws.autoscaling", "account": "123456789012", "time": "2015-11-11T21:31:47Z", "region": "us-east-1", "resources": [], "detail": { "eventVersion": "", "responseElements": null } } Example Event Example Rule { "source": ["aws.autoscaling"] }
28. • Source of truth for sharing schema • Explicitly published and auto-discovered • Integrations for IDEs like Jetbrains and VS Code • Language Bindings available for Java, Python and Typescript Amazon EventBridge - Schema Registry and Discovery
29. Amazon EventBridge - Event Archive and Replay
30. When Amazon EventBridge is not best fit? • Ordering Guarantees • Event Deduplication • Push Delivery • Ultra Low latency • Non-JSON Transport
31. Event Modelling CloudEvents - A specification for describing event data in a common way • Consistent schema across all your events • Many Language SDK available - Java, GoLang, Python, Rust etc • Portable Across AWS, GCP, Azure • CNCF Serverless Group supported • https://cloudevents.io/
32. Event Driven Architectures - Infrastructure as Code Support • https://serverlessland.com/ • https://cdkpatterns.com • Use patterns to quickly build integrations using AWS SAM and CDK templates. • 70+ Templates • You can contribute to it!
33. EventBridge - Real Time Threat Mitigation
34. Loan Broker Use case src: https://www.enterpriseintegrationpatterns.com/patterns/messaging/ComposedMessagingExample.html
35. • Receive the consumer's loan quote request • Obtain credit score and history from credit agency • Determine the most appropriate banks to contact • Send a request to each selected bank • Collect responses from each selected bank • Determine the best response • Pass the result back to the consumer Loan Broker Use case - Processing Flow src: https://www.enterpriseintegrationpatterns.com/patterns/messaging/ComposedMessagingExample.html
36. Loan Broker Use case src: https://www.enterpriseintegrationpatterns.com/patterns/messaging/ComposedMessagingExample.html
37. Summary • event driven architectures are essential complexity for large scale decoupled applications • Awareness of enterprise integration patterns will ease out the journey through event driven architectures • For choosing the aws service for your scenarios, compare scale, durability, persistence and consumption models, retries and pricing. • Adopting AWS serverless services like EventBridge ease out complex implementations like content filtering and event replays and cross account, cross region scenarios. • You can compose solution by combining multiple patterns and services • Utilize IaC templates available in open source or build your own.
38. Further Reading • http://www.sosp.org/2001/papers/welsh.pdf • https://www.enterpriseintegrationpatterns.com/ • https://aws.amazon.com/blogs/compute/introducing-cross-region-event-routing-with-amazon-eventbridge/ • https://aws.amazon.com/blogs/compute/building-well-architected-serverless-applications-introduction/ • https://serverlessland.com/patterns
39. Q & A
40. Thank You!
41. Extras
42. Amazon EventBridge - Pricing AWS Service Events Free Custom Events $1/ million events published Third Party Events $1/ million events published Cross Account Events $1/ million cross-account events sent Schema Discovery $0.10 / million events ingested Archive Processing $0.12 per GB Storage Month $0.0025 per GB
43. SQS, SNS, EventBridge Amazon SQS Amazon SNS Amazon EventBridge Messaging 1 to 1 1 to Many 1 to Many Event Type Single Event Type Single Event Type Multiple Event Type Route Messages Single Single Multiple Rules Filter Messages No Only to Message Body Full Message Allow transformation? No No Yes Cross Account Yes Yes Yes Message Latency Low Low Moderate Throughput High High Moderate