Message-Driven Architecture is the foundation for building reactive microservices (responsive, resilient, elastic). See why in this presentation and how to do it on AWS.
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkJerome Van Der Linden (@jeromevdl)Message-Driven Architecture with AWSWorldwide Software Architecture Summit26th of January 2021
View Slide
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkWho am I ?Jérôme Van Der LindenSolutions Architect @ AWS (Geneva, Switzerland)Former consultant, developer, tech lead, agile &devops coach, architect @ OCTO TechnologyHusband and Dad of 3@jeromevdl
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkAgendaMonolith to MicroservicesMicroservices to Reactive MicroservicesMessage-Driven Architecture on AWSWrap-up
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkMonoliths to Microservices
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkMonolithsDo not throw the baby out with the bath water…- Simple to develop and refactor- Simple to test (integration, end-to-end)- Single deployment process- Single application to monitor- Consistencyhttps://martinfowler.com/bliki/MonolithFirst.html
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkMonoliths… but definitely some limits!- Strong coupling: less flexibility / agility- High impact of change- Redeploy everything- Risks of blast radius- Reliability- One failure can bring down the whole application- Rigid scaling- Often stateful- Limited to the database scalability- Development lifecycle is typically slow- Continuous deployment is difficult- Barrier to adopting new technologies
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkMicroservicesMicroservices to the rescue…- Reduced coupling- Smaller impact of changes- Independent deployments- Better reliability- Independent scaling- Smaller unit of development:- Humanely understandable- More suitable for adopting new technologies
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkMicroservices… but it is not that magic!- Reduced coupling- Smaller impact of changes- Independent deployments- Better reliability- Independent scaling- Smaller unit of development:- Humanely understandable- More suitable for adopting new technologiesAre you sure?
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkMicroservices to Reac8veMicroservices
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkMicroservices communica=onOrderServiceInvoiceService
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkSynchronous communica=onOrderServiceInvoiceService201created201created
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkSynchronous communica=onOrderServiceInvoiceService500internalservererror503serviceunavailable☹
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkSynchronous communica=onWhat’s the problem ?- No isolation between services è still coupled- Risk of timeout or cascading failures è still unreliable- What if the signature of invoice API change ? è still dependent
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkReac=ve PrinciplesResponsiveResilientElasticMessage-Drivenhttps://www.reactivemanifesto.org/
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkMessage-Driven (Asynchronous) Communica=onOrderServiceInvoiceService202acceptedordercreated
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkMessage-Driven (Asynchronous) Communica=onIsolation between services:- Less risk of timeout è more responsive- No more cascading failure è more resilient è more responsive- What if the signature of invoice API change ? è No problem !
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkMessage-Driven Architecture on AWS
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark(Serverless) Messaging services on AWSAmazon SNS AmazonEventBridgeAmazon SQS
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkAmazon Simple Queue Service (SQS)Producer SQS QueueDecoupleproducers fromconsumers.Messages polledby consumersAmazon SQSFully managedMessage QueuingserviceConsumersAWS LambdaContainerInstance / VMStandard• At-least once delivery• Best-effort orderingMessages are storedup to 14 days.
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkAmazon Simple Queue Service (SQS)Producer ConsumersAWS LambdaAmazon SQSFully managedMessage QueuingserviceSQS QueueDecoupleproducers fromconsumersMessages polledby consumersContainerInstance / VMFIFO• Exactly once delivery• Keep ordering123
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkAmazon Simple Queue Service (SQS)Producer ConsumersAWS LambdaAmazon SQSFully managedMessage QueuingserviceSQS QueueDecoupleproducers fromconsumersLong PollingContainerInstance / VM
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkAmazon Simple Queue Service (SQS)Producer ConsumersAWS LambdaAmazon SQSFully managedMessage QueuingserviceSQS QueueDecoupleproducers fromconsumersMessages polledby consumersContainerInstance / VMDead-letter QueueIf a message can’t be processed successfullyit can be held in a queue
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkCommon paIern with SQS: Storage-firstAmazon API Gateway Amazon SQS QueueDead letter QueueAWS LambdaClient…
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkAmazon Simple No=fica=on Service (SNS)Amazon SNSFully managedpublish & subscribemessaging serviceSNS TopicDecouple messagepublisher fromsubscribers with topicsPublisherMessages pushed tosubscribersSubscribersAWS LambdaAmazon SQSHTTP/SEmail / SMSAmazon KinesisData Firehose
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkAmazon Simple No=fica=on Service (SNS)Amazon SNSFully managedpublish & subscribemessaging serviceSNS TopicDecouple messagepublisher fromsubscribers with topicsPublisherFIFOSubscribersAWS LambdaAmazon SQSFIFOHTTP/SEmail / SMSAmazon KinesisData Firehose12
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkAmazon Simple No=fica=on Service (SNS)Fan OutPublisher Amazon SNSFully managedpublish & subscribemessaging serviceSNS TopicDecouple messagepublisher fromsubscribers with topicsSubscribersAWS LambdaAmazon SQSHTTP/SEmail / SMSAmazon KinesisData Firehose
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkAmazon Simple No=fica=on Service (SNS)Publisher Amazon SNSFully managedpublish & subscribemessaging serviceSNS TopicDecouple messagepublisher fromsubscribers with topicsMessageFilteringFilter messagesaccording tosubscriptionfilter policiesSubscribersAWS LambdaAmazon SQSHTTP/SEmail / SMSAmazon KinesisData Firehose
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkAmazon Simple No=fica=on Service (SNS)Publisher Amazon SNSFully managedpublish & subscribemessaging serviceSNS TopicDecouple messagepublisher fromsubscribers with topicsMessageFilteringFilter messagesaccording tosubscriptionfilter policiesDead-letter QueueIf a subscriber is unavailable,messages can be held in a queueSubscribersAWS LambdaAmazon SQSHTTP/SEmail / SMSAmazon KinesisData Firehose
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkCommon paIern with SNS & SQS: Topic-Queue ChainingAmazon SQSQueue ABackend APublisher Amazon SNSTopicAmazon SQSQueue BBackend BAmazon SQSQueue CBackend C
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkAmazon EventBridgeAmazonEventBridgeFully managedevent bus toconnect applicationsEvent SourcesAWS ServicesCustom eventsSaaS events Events pushedto targetsRulesRules filterand send eventsto targetEvent BusDefault event busCustom event busSaaS event busDecouple eventsources and targetsTargets (20+)AWS LambdaAmazon SQSAmazon SNSAWS StepFunctionsAmazon KinesisAWS BatchAmazon APIGatewayAmazon EC2InstanceAmazon ECSTask
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkAmazon EventBridgeEvent SourcesAmazonEventBridgeFully managedevent bus toconnect applicationsEvent BusTargets (20+)AWS LambdaAmazon SQSAWS ServicesCustom eventsSaaS eventsDefault event busCustom event busSaaS event busRulesAmazon SNSAWS StepFunctionsAmazon KinesisAWS BatchAmazon APIGatewayAmazon EC2InstanceAmazon ECSTaskDead-letterQueueIf a target is unavailable,events can be held in a queue
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkAmazon EventBridgeEvent SourcesAmazonEventBridgeFully managedevent bus toconnect applicationsEvent BusTargets (20+)AWS LambdaAmazon SQSAWS ServicesCustom eventsSaaS eventsDefault event busCustom event busSaaS event busRulesAmazon SNSAWS StepFunctionsAmazon KinesisAWS BatchAmazon APIGatewayAmazon EC2InstanceAmazon ECSTaskEvent ArchiveStore all eventsand enables replays
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkAmazon EventBridgeEvent SourcesAmazonEventBridgeFully managedevent bus toconnect applicationsEvent BusTargets (20+)AWS LambdaAWS ServicesCustom eventsSaaS eventsDefault event busCustom event busSaaS event busRulesAWS StepFunctionsAWS BatchAmazon APIGatewayAmazon EC2InstanceAmazon ECSTaskSchema RegistryCollection of eventstructures (schema)…Code BindingSchemaDiscoverySchemaPublication
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkCommon paIern with EventBridge: Microservices Choreography
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkWrap-up
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkServices ComparisonAmazon SQS Amazon SNSAmazonEventBridgeType Queue (Pull) Publish / Subscribe (Push) Publish / Subscribe (Push)PersistenceYes(Up to 14 days, default: 4 days)No(retry up to 23 days + DLQ)No(retry up to 24h + DLQ + Archives)Ordering FIFO FIFO NoDelivery Standard: at least onceFIFO: exactly onceStandard: at least onceFIFO: exactly onceAt least onceFiltering Yes YesNoConsumers Millions 5 per rule/Target services Lambda (+ compute & SDK) 2 (+ HTTP/SMS/Email/Push) 20+Throughput *Price *Standard: Almost unlimitedFIFO: 3 000 TPS (soft)Standard: 30 000 TPS (soft)FIFO: 300 TPS (soft)10 000 TPS (soft)$ per million API requestsStandard ≤ 0.4 / FIFO ≤ 0.5$ per million API requests: 0.5 $ per million events: 1* depends on the region (us-east-1)
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkConclusionMessage-Driven architectures drive resilience and elas3city è responsivenessAsynchronous EventsImprove responsivenessand reduce dependenciesEvent StoresBuffer messages until servicesare available to processEvent RoutersAbstract producers andconsumers from each other
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkResources
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark• Building event-driven applications with Amazon EventBridge• https://virtual.awsevents.com/media/1_w2mhd582• Scalable serverless event-driven architectures with SNS, SQS & Lambda• https://virtual.awsevents.com/media/1_o5hp8egy• Decoupling serverless workloads with Amazon EventBridge• https://virtual.awsevents.com/media/1_upe6mtew• Choosing the right event-routing service for serverless: EventBridge, SNS, or SQS• https://lumigo.io/blog/choosing-the-right-event-routing-on-aws-eventbridge-sns-or-sqs
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and TrademarkThank you@jeromevdl