Serverless architecture as enabler for low code solution

© 2024

Transcript

1. Serverless architecture as enabler for low code solutions Daniele Frasca Roman Boiko TeamLead Content Discovery ProSiebenSat.1 Media SE Senior Specialist Solutions Architect Serverless AWS © 2023, A mazon Web Services, Inc. or its af filiates.
2. Daniele Frasca AWS Community Builder https://dev.to/ymwjbxxq Team Lead – Content Discovery
3. Key figures of the ProSiebenSat.1 Group 5600+ Amazon EC2 instances running each day Over 450 Amazon RDS instances running per day Over 1.5 petabytes transferred via Amazon CloudFront per day Over 340 million AWS Lambda invocations per day Over 1.4 terabytes of data transferred via API Gateway per day Over 12 petabytes of storage stored in Amazon S3
4. Roman Boiko Serverless Specialist SA @romannboiko
5. The challenge © 2023, A mazon Web Services, Inc. or its af filiates.
6. Content Discovery • • • • • • LayoutBookmark Entitlement Lookup Query Search WatchNext
7. Architecture at a Glance AWS account Message Self-host Kafka AWS account ECS
8. Why Kafka? © 2023, A mazon Web Services, Inc. or its af filiates.
9. What is Kafka Is it like a Service Bus? Is It like REST but Async? Streaming Platform Is it like a Database?
10. EventBridge Event Bus Event sources AWS services Targets Amazon EventBridge AWS Targets Default event bus Custom events API Destinations Custom event bus SaaS apps API Gateway Rules SaaS event bus External API Endpoint External Webhook
11. Event stores vs event routers Event stores(Kafka) • Act as a buffer between producer and consumer • Messages pulled by consumer • Persistence • Ability to invoke Lambda functions with a batch of events Event routers(EventBridge) • Decouples producers and consumers • Fan-out events to multiple consumers • Messages pushed to consumer • Automatic retries • Event filtering • Ability to chain with event stores
12. EventBridge with Kafka © 2022, A mazon Web Services, Inc. or its af filiates.
13. EventBridge Pipes: Connects sources and targets Sources Enrich and Transform Targets DynamoDB Lambda EventBridge Kinesis API Gateway Lambda Filter Choose which event to keep Apache Kafka or Amazon MSK SQS Step Functions EventBridge API Destination SQS Total 15 targets
14. Features/capabilities Filter events Adding filters to provide a subset of events for consumption ensures your applications process only the events they need. Batch events Retrieve from event produces and deliver to event consumers in batches, making integrations more efficient. Order events Provide guaranteed ordering of events for applications where the ordering is critical. High concurrency Support high concurrency so that you can process large volumes of events quickly. Advanced enrichment Offers support for enrichments using AWS Lambda, AWS Step Functions, API Gateway, or an API call to a third-party API (using API Destinations).
15. Benefits Write less integration code Build fully managed integrations quickly with an easy to use interface that seamlessly connects event producers with event consumers. Save costs with filtering and built-in integrations Lower compute and usage costs by using event filters and built-in integrations – only process and pay for events that you need. Source events in real time Deliver events to over 14 AWS services with EventBridge Pipes seamlessly polling for new events. Reduce operational load Reliably connect services without worrying about scalability, security, patching, and provisioning your infrastructure.
16. EventBridge with Kafka alternative © 2022, A mazon Web Services, Inc. or its af filiates.
17. Amazon EventBridge open-source connector for Apache Kafka Connect Data Sources External Databases Kafka Connect Kafka Connect Source Connector Connector for Amazon Event Bridge Amazon EventBridge Apache Kafka Connect is an open source framework to stream data in and out of Apache Kafka
18. Deliver Events from Kafka Topics to Amazon EventBridge Event Source Schema Registry AWS Step Functions Rule Amazon SQS Kafka Connector for EventBridge Amazon EventBridge Event Bus Amazon API Gateway Many more…
19. Amazon EventBridge Pipes or Connectors with Kafka Amazon EventBridge Pipes Event Source Amazon MSK Self Managed Kafka within VPC Confluent Cloud ( SaaS ) Includes SQS, Kinesis, DynamoDB, MQ. Polling Over 35+ AWS Services, SaaS integration, API Destinations Point-to-point Event Source Amazon MSK Self Managed Kafka within VPC Self managed Kafka on-prem Confluent Platform on-prem Confluent Cloud ( SaaS ) Amazon EventBridge Push Kafka Connector For Amazon EventBridge Over 35+ AWS Services, SaaS integration, API Destinations many-to-many Amazon EventBridge
20. How we did this? © 2023, A mazon Web Services, Inc. or its af filiates.
21. Architecture at a Glance – as it was • • • • • • • Layout Bookmark Entitlement Lookup Query Search WatchNex AWS account Message Self-host Kafka AWS account ECS
22. How can I decouple it? © 2023, A mazon Web Services, Inc. or its af filiates.
23. Hub & Spoke Pattern Account A Hub Account Event Source Event Confluent onPrem / Cloud EventBridge Pipes Amazon EventBridge Amazon EventBridge Account B Amazon EventBridge Account C Amazon EventBridge API Destination
24. Implemented Solution © 2023, A mazon Web Services, Inc. or its af filiates.
25. Architecture at a Glance – EventBus Mesh • • • • • • • • • • Layout Image Bookmark Entitlement Lookup Query Search GeoIP WatchNext Redirect Kafka as company bus Content Discovery Pipes EventBridge as team bus Microservice1 Microservice2 Local bus Local bus
26. Sparse vs Full state description vs
27. Claim check pattern AWS account AWS account Smaller message with key Message with data Pipes EventBridge Rule Get data with Key S3 Consumer
28. EDA: Bookmark ➢ Bus to Bus architecture ➢ EventBridge Pipes with the Claim Check Pattern to handle Kafka ➢ EventBridge Pipe Enrichment finds all the users that bookmarked a video ➢ S3 Event Notification with EventBridge ➢ API Gateway as a proxy with DynamoDB ➢ EventBridge API Destination to sync up with Kafka
29. Bookmark AWS account Region1 VTL Get/Add/Delete users CloudFront DynamoDB Global Tables Route53 VTL Region2
30. Bookmark AWS account Region1 Momento Serverless Cache Bookmark Bus AWS account Pipes DynamoDB Global Tables Bookmark Bus Pipes Region2 Momento Serverless Cache
31. AWS account DynamoDB Global Tables Bookmark AWS account Momento Serverless Cache Bookmark Bus VTL Region2 AWS account Smaller message with key Message with data Pipes CD Bus Rule Get data with Key S3 Consumer
32. AWS wishes ➢ Bigger payload size for EB, SNS, SQS -> 6MB like Lambda ➢ Better metrics for EventBridge Pipes and API Destination ➢ Higher quotas EventBridge API Destination ➢ Kafka as Target for EventBridge Pipes ➢ S3 as Target for EventBridge Pipes ➢ EventBridge global endpoints for EventBridge Pipes ➢ Redrive from EventBridge family DLQ
33. Summary • Integrated Kafka and EventBridge usage: You can use both Kafka and EventBridge together, optimizing data flow and event management across different systems. • Reduced Complexity and Costs: Serverless architecture minimizes the need for manual scaling and maintenance, significantly lowering operational costs. • Increased Scalability and Flexibility: The use of AWS serverless solutions provides dynamic scalability and adaptability to changing workloads. • Less Code Writing: The adoption of EventBridge Pipes enables developers to write less integration code, speeding up deployment and reducing errors.
34. Thank you! Daniele Frasca @dfrasca80 © 2023, A mazon Web Services, Inc. or its af filiates. Roman Boiko @romannboiko Please credits the authors when reusing material