Evolution of a Backend for a Streaming Application

Transcript

1. Evolution of a Backend for a Streaming Application Daniele Frasca Principal Engineer ProSiebenSat.1 Media SE
2. From: Single Monolith Region Content Discovery AWS account AWS account AWS account request ECS Message Self-host Kafka Apollo GraphQL ECS
3. To: Active – Active service Content Discovery AWS account AWS account request Amazon CloudFront Apollo GraphQL Amazon Route 53 Region Region Api Api AWS account Worker Message Self-host Kafka Worker SQS SQS
4. Daniele Frasca AWS Community Builder https://dev.to/ymwjbxxq Principal Engineer
5. The challenge Deliver resilient, high-performance backend systems and APIs that empower seamless video streaming and endto-end video syndication, with stability, scalability, and reliability at the core of everything we build.
6. Problems 1. Data Quality 2. Scalability and Resilience
7. Architecture at a Glance AWS account Message Self-host Kafka AWS account ECS
8. Hub & Spoke Pattern Account A Hub Account Event Source Event Confluent onPrem / Cloud EventBridge Pipes Amazon EventBridge Amazon EventBridge Account B Amazon EventBridge Account C Amazon EventBridge API Destination
9. Sparse vs Full state description vs
10. Claim check pattern AWS account AWS account Smaller message with key Message with data Pipes EventBridge Rule Get data with Key S3 Consumer
11. Alternative Approach - Centralized Data Replication https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2021/08/16/dbblog-part1-diag.png
12. Event Replication vs Data Replication - Trade-offs Aspect Event-Driven Data Replication Coupling Loose - Teams evolve independently Tight - Schema changes affect all accounts Technology Flexible - Each team chooses best fit Locked-in - Single database technology Scalability Horizontal - Parallel event processing Limited - Central DB becomes bottleneck Consistency Eventual - Temporary inconsistencies Stale data from the same source Debugging Complex - Trace across multiple systems Simple - Standard SQL debugging Queries Limited - Reconstruct from events Powerful - Complex joins & analytics Performance High throughput at scale Depends on DB performance Cost Variable Predictable - Fixed DB costs Operations Complex - Multiple moving parts Complex - Network + DB management Network Setup Simple - No VPC connectivity needed Complex - VPC peering/Transit Gateway required
13. Problem Scalability and Resilience When everyone shows up at once, your architecture either scales or fails
14. Services Public Internet Front door Amazon Route 53 Amazon API Gateway Computational service Database AWS Lambda Amazon Aurora AWS Fargate Amazon DynamoDB Amazon CloudFront Application Load Balancer AWS Global Accelerator Amazon RDS
15. Availability Comparison Table HT TPS: / /A WS. AMA ZON. C OM /L EGA L/ SER VI CE-LE VE L-AGR E EM E NTS/ Architecture Component SLAs Combined Availability Monthly Downtime Key Considerations ~99.94% Serverless auto-scaling ~26 minutes Built-in multi-AZ Fast cold starts Container management overhead Task restart delays ALB + Lambda ALB: 99.99% Lambda: 99.95% ALB + ECS Fargate ALB: 99.99% ECS Multi-AZ: 99.99% Task Uptime: 99.5% ~99.48% ~3.7 hours API Gateway + Lambda API GW: 99.95% Lambda: 99.95% ~99.90% ~43 minutes Fully serverless
16. Availability Comparison Table HT TPS: / /A WS. AMA ZON. C OM /L EGA L/ SER VI CE-LE VE L-AGR E EM E NTS/ Database DynamoDB SLA Single Region: 99.99% Global Tables: 99.999% Monthly Downtime Key Considerations Network Complexity ~4.3 min ~26 sec Serverless scaling No VPC required Built-in backup Low Medium - DB subnet groups Aurora Serverless v2 99.95% Multi-AZ: 99.99% ~22 min ~ 26 sec Auto-scaling compute Fast failover Storage auto-replication Requires VPC setup Aurora DSQL 99.99% ~4.3 min Fully serverless SQL No capacity planning Low - Serverless endpoint RDS Multi-AZ Single-AZ: 99.5% Multi-AZ: 99.95% ~3.6 hours ~22 min Manual failover (2-6 min) Requires VPC setup Storage limitations High - Manual subnet management
17. Database Architecture: The Make-or-Break Decision Aspect Aurora Global DB Aurora Multi-AZ RDS Single Node + Replicas RTO (Recovery Time) < 1 minute (automated) 1-2 minutes (automated) 5-15 minutes (manual) RPO (Recovery Point) < 1 second < 1 second 1-5 minutes (replication lag) Region-wide outages Natural disasters Network partitions AZ-level failures Region-wide issues Operational Complexity AWS handles cross-region complexity AWS handles failover complexity Performance Impact Minimal (local reads, async writes) None (synchronous local) Potential (read replica lag) Customer Experience During Outage Transparent (automatic failover) Brief interruption (30-120s) Read failures: Transparent Write failures: Service downtime Write Scaling Cross-region async (eventual consistency) Single writer bottleneck Single writer = critical failure point Disaster Scenarios Protected Hardware failures only AZ-level failures Human errors Application handles all complexity
18. Hidden Problems People Don't Know About Problem Category Aurora Global Aurora Multi-AZ RDS Single Node + Replicas Connection Pool Exhaustion Distributed across regions Shared pool across AZs RDS Proxy is a must Replica Lag Under Load < 100ms typical < 50ms Inconsistent read Different lag per replica Failover Complexity Automated DNS switching AWS handles seamlessly Read failures: RDS Proxy handles Write failures: Manual promotion Split-Brain Scenarios Impossible (managed service) Impossible (AWS managed) Manual promotion risks Multiple writers possible Monitoring Complexity Built-in cross-region metrics Standard CloudWatch Lag consistency tracking complex Network Partition Handling Graceful degradation Region-local traffic only Write node isolation: Application crashes Version Upgrades Rolling, zero-downtime Rolling, minimal downtime Write downtime: Primary must restart Cross-Account Network Read Replica Setup No networking required No networking required VPC Peering + Transit Gateway CIDR planning
19. Infrastructure SLA Availability: 99.78% SLA 99.9% Single Region 99.99% 99.95% 99.95% Downtime 99.99% Configuration Availability Annual Downtime Annual downtime: ~19.3 hours per year Single Region 99.78% ~19.3 hours Monthly downtime: ~1.6 hours per month Multi-Region 99.9995% ~26 minutes Weekly downtime: ~22 minutes per week Daily downtime: ~3.2 minutes per day https://aws.amazon.com/legal/service-level-agreements/ That's a 44x improvement in availability!
20. Service template AWS account Region1 Aurora Global Database users CloudFront DynamoDB Global Tables Route53 Region2 Momento Cache
21. No Single Point of Failure " THE SE R VE RL ESS TRA NSFOR M ATI ON" Architecture Evolution Journey: • Phase 1: ALB → ECS Fargate (Traditional containers) • Phase 2: API Gateway → Lambda (Full serverless) • Phase 3: ALB → Lambda (Cost-optimized serverless) Why Serverless Wins: • AWS Manages Infrastructure - We focus on code, not servers • Built-in Resilience - Multi-AZ, security, networking handled automatically • Elastic by Design - No capacity planning or sizing decisions • Secure by Default - Isolated execution environments Database Strategy: • DynamoDB Global Tables - Cross-region replication built-in • Aurora Global Database - Sub-second inter-region latency • OpenSearch Multi-Region - Active clusters in 2 regions
22. Cell-Based Architecture " CON TAI NIN G TH E B LAS T RA DIU S" Geographic Cell Isolation Platform Sub-Cells Within Countries Database Evolution: Shared → Cell-Specific Blast Radius Containment DE Cell: Germany registered/anonymous + platforms Web, OTT, iOS, Android lambdas per country Current: Single database = single point of failure DE outage ≠ AT/CH impact AT Cell: Austria registered/anonymous + platforms User type separation (registered vs anonymous) Target: One database per country cell CH Cell: Switzerland registered/anonymous + platforms Isolated processing pipelines Each cell = independent failure domain Cross-border failures contained iOS failure ≠ Web/OTT impact Database failure is limited to one country
23. Caching Strategies " WE WENT FROM SECON DS TO M ILL ISECON DS I N RESPO NSE TI MES" At peak time: • • CloudFront: Eliminates 78.6% of origin load Application Cache: Eliminates 73.3% of remaining load Result: 94.3% reduction in database queries CloudFront CDN: Transformed content delivery closest to users Reduces latency Reduce hit to the origin In-Memory Cache: 10-minute TTL with LRU eviction strategy 5,000 items max (~100MB total), optimized for 1GB Lambda Two-phase cleanup for memory efficiency Momento Cache: Distributed remote caching layer Shared cache across multiple Lambda instances Persistent hot data beyond individual function lifecycles
24. Reliability Engineering WH EN TH IN GS GO WR ONG (AND THE Y WI LL) ALB Health Checks + Target Group Failover CloudWatch Alarms + SNS Notifications • 30-second health check intervals • Real-time failure detection • automatic failover • Automated alerting pipeline Route53 Health-Based Routing • Proactive incident response • Regional health checks Weighted Target Groups • Automatic region failover • Primary target (Weight: 100) • DNS-level traffic distribution • Failover target (Weight: 0) • Instant traffic redirection
25. Multi-Region Active-Active CloudFront + Route53 1. Global edge locations for performance 2. Route53 health checks for automatic failover 3. Sub-second DNS propagation Regional Cell Independence 1. EU-Central-1: Primary region (DE/AT/CH cells) 2. EU-West-1: Secondary region (replicated cells) 3. Each region can handle 100% traffic load Data Plane Operations Only 1. Route53 health checks (data plane) 2. ALB target group failover (data plane) 3. Lambda invocations (data plane) 4. No CloudFormation during recovery Recovery Time Objectives 1. RTO (Recovery Time Objective): <5 minutes (DNS TTL + health check) 2. RPO (Recovery Point Objective): < 1 minute (Aurora Global + DynamoDB Global Tables) 3. Availability: 99.99% (compound of both regions)
26. Cost Optimization TH E HY B RI D P ATTE R N Data Plane Focus • Route53, ALB, Lambda = High availability, low fixed costs • Eliminate control plane dependencies & associated infrastructure Hybrid Pattern • High traffic → ECS (cost-effective concurrency) • Low traffic → Lambda only (zero idle costs) • ALB over Api Gateway Smart Scaling Strategies • Scale to zero when possible • Regional load balancing vs. global over-provisioning • Event-driven scaling based on actual demand
27. Summary Resilience through delegation, not duplication. The Transformation • Before: 60% time on infrastructure → After: 90% time on feature development • Before: System-wide outages → After: Cell-isolated incidents • Before: Manual failover procedures → After: 90-second detection, 5-second recovery Architecture Wins • • • • Serverless-First: ALB + Lambda = 40% API cost reduction vs API Gateway Cell-Based Isolation: Country + platform cells eliminate blast radius Triple-Layer Caching: CloudFront + In-Memory + Momento = minimal origin hits Data Plane Only: Zero control plane dependencies during recovery Measurable Results • • • • RTO: < 5minutes (DNS + health checks) RPO: <1 minute (Aurora Global + DynamoDB Global Tables) Availability: 99.99% compound across regions Cost Impact: Multi-region without multiplying costs
28. Thank you! Let’s connect LinkedIn ©2025 Daniele Frasca Evolution of a Backend for a Streaming Application Please credit the author when reusing material