Build Event-Driven Architectures with AWS Lalit Kale Principal Software Engineer - Verizon Connect Effortlessly

Lalit Kale ● Principal Software Engineer, Verizon Connect ● Dublin, Ireland ● First AWS Use: 2006 https://www.linkedin.com/in/lalitkale/ @atechiethought

Agenda ● What is Event Driven Architectures ● AWS Serverless Application Integration Services ○ AWS Lambda ○ Selecting Right Messaging service: SQS, SNS and Eventridge ○ Application Integration Patterns support ● Amazon EventBridge ● Use Cases

Current Landscape Large Systems (couple of years ago) ● Few Kb of internet speed ● Couple of giant single core servers ● Seconds of response time ● Hours of offline maintenance ● Data size in Gigabytes Large Systems (Today & Future) ● GBs scale internet speed ● Thousands of multicore servers ● Milliseconds of response time ● Users expect 100% uptime ● Data size in Petabytes and Zetabytes

Reactive Systems

The big idea is "messaging" - Alan Kay src: https://wiki.c2.com/?AlanKayOnMessaging

"If your application is cloud-native or, large scale or, distributed, and doesn't include a messaging component, that is probably a bug." - Tim Bray Former Sr. Principal Engineer, AWS src: https://wiki.c2.com/?AlanKayOnMessaging

Communication Synchronous Communication Asynchronous Communication Face To Face Video Conferencing Email

Types of Communication Synchronous Communication Asynchronous Communication

Types of Messages Commands ● "I Intent to …" ● Can be rejected by receiver ● Represented as verbs in present tense, e.g. "create_customer" ● Source system is aware of side effects that command processing will have Events ● Something has happened ● Immutable—cannot change the past ● Represented as verbs in the past tense, e.g. “customer_created” ● Source system has no expectations on how an event is processed

Event Driven Architecture Event driven architecture is a distributed asynchronous integration pattern to create highly scalable and loosely coupled reactive applications Events drives the data processing

Advantages of Event driven architecture ● Loose coupling between components/services ● Ability to scale individual components ● Processing components can be developed independent of each other ● Fault Tolerance and better resiliency ● Improved Availability ● Evolutionary architecture ● Business Agility

Challenges with Event Driven Architectures ● Consistency Model mindshift - Eventual Consistency ● Observability of Event Pipelines ● Immutable Consumers ● Error Handling ● Event Replay Scenarios

Challenges with Event Driven Architectures ●

Event Driven Architecture Event driven architecture is a distributed asynchronous integration pattern to create highly scalable and loosely coupled reactive applications

Enterprise Integration Patterns ● Point to Point channel ● Pub-Sub pattern ● Message Router ● Event Bus

Event Driven Architectures with AWS Serverless

Serverless Story is incomplete without...

AWS Lambda Execution Models

AWS Lambda Destinations

AWS Serverless Application Integration Services

Service Selection Criteria Scale & Concurrency Controls Durability Persistence Consumption Retries Pricing

AWS Serverless Application Integration Services Amazon SQS Durable and scalable Buffer and Batching support Comprehensive security point to point channel Amazon SNS Performance at Scale Enterprise Ready, Deliver to endpoints like email, SMS, Https, Push Notifications Fan-Out, Pub-Sub Amazon EventBridge Serverless event bus for AWS Services, your own apps and SaaS providers Event Bus

Failures and Retries Amazon SQS Amazon SNS Amazon EventBridge Delete Message. Redrive policy for failed messages. Retry policy for different protocols Event Retention and Replay Retry upto max. 14 days Retry upto 23 days Retry upto 24 hours Dead Letter Queue Dead Letter Queue Dead Letter Queue Exactly once with FIFO Exactly Once with FIFO Topic

Amazon EventBridge ● Serverless, Pay as you go ● 90+ AWS Services as sources ● 17+ AWS services as targets ● Native integrations with SaaS Providers ● $1 per million events put into the bus ● No additional cost of delivery ● Cross Account, Cross Region

Amazon EventBridge

Amazon EventBridge - Events and Rules { "version": "0", "id": "3e3c153a-8339-4e30-8c35-687ebef853fe", "detail-type": "EC2 Instance Launch Successful", "source": "aws.autoscaling", "account": "123456789012", "time": "2015-11-11T21:31:47Z", "region": "us-east-1", "resources": [], "detail": { "eventVersion": "", "responseElements": null } } Example Event Example Rule { "source": ["aws.autoscaling"] }

● Source of truth for sharing schema ● Explicitly published and auto-discovered ● Integrations for IDEs like Jetbrains and VS Code ● Language Bindings available for Java, Python and Typescript Amazon EventBridge - Schema Registry and Discovery

Amazon EventBridge - Event Archive and Replay

When Amazon EventBridge is not best fit? ● Ordering Guarantees ● Event Deduplication ● Push Delivery ● Ultra Low latency ● Non-JSON Transport

Event Modelling CloudEvents - A specification for describing event data in a common way ● Consistent schema across all your events ● Many Language SDK available - Java, GoLang, Python, Rust etc ● Portable Across AWS, GCP, Azure ● CNCF Serverless Group supported ● https://cloudevents.io/

Event Driven Architectures - Infrastructure as Code Support ● https://serverlessland.com/ ● https://cdkpatterns.com ● Use patterns to quickly build integrations using AWS SAM and CDK templates. ● 70+ Templates ● You can contribute to it!

EventBridge - Real Time Threat Mitigation

Loan Broker Use case src: https://www.enterpriseintegrationpatterns.com/patterns/messaging/ComposedMessagingExample.html

● Receive the consumer's loan quote request ● Obtain credit score and history from credit agency ● Determine the most appropriate banks to contact ● Send a request to each selected bank ● Collect responses from each selected bank ● Determine the best response ● Pass the result back to the consumer Loan Broker Use case - Processing Flow src: https://www.enterpriseintegrationpatterns.com/patterns/messaging/ComposedMessagingExample.html

Loan Broker Use case src: https://www.enterpriseintegrationpatterns.com/patterns/messaging/ComposedMessagingExample.html

Summary ● event driven architectures are essential complexity for large scale decoupled applications ● Awareness of enterprise integration patterns will ease out the journey through event driven architectures ● For choosing the aws service for your scenarios, compare scale, durability, persistence and consumption models, retries and pricing. ● Adopting AWS serverless services like EventBridge ease out complex implementations like content filtering and event replays and cross account, cross region scenarios. ● You can compose solution by combining multiple patterns and services ● Utilize IaC templates available in open source or build your own.

Further Reading ● http://www.sosp.org/2001/papers/welsh.pdf ● https://www.enterpriseintegrationpatterns.com/ ● https://aws.amazon.com/blogs/compute/introducing-cross-region-event-routing-with-amazon-eventbridge/ ● https://aws.amazon.com/blogs/compute/building-well-architected-serverless-applications-introduction/ ● https://serverlessland.com/patterns

Q & A

Thank You!

Extras

Amazon EventBridge - Pricing AWS Service Events Free Custom Events $1/ million events published Third Party Events $1/ million events published Cross Account Events $1/ million cross-account events sent Schema Discovery $0.10 / million events ingested Archive Processing $0.12 per GB Storage Month $0.0025 per GB

SQS, SNS, EventBridge Amazon SQS Amazon SNS Amazon EventBridge Messaging 1 to 1 1 to Many 1 to Many Event Type Single Event Type Single Event Type Multiple Event Type Route Messages Single Single Multiple Rules Filter Messages No Only to Message Body Full Message Allow transformation? No No Yes Cross Account Yes Yes Yes Message Latency Low Low Moderate Throughput High High Moderate