Delivering Millions of Messages within seconds @ Duolingo

Building a notification system may seem trivial, but what about building one that could reach millions of users within a few seconds? What about doing that right after your advertisement airs?

Event-based notification systems are not uncommon anymore, but there’s rarely a cost-effective example of an on-demand, highly parallel notification system. The complexity of building such a system comes from the intersection of system design, site reliability, and cloud resource management. All of that while being pressured by the demands of an unhinged marketing campaign over TV and the Web.

Transcript

1. Delivering Duolingo Millions of messages within seconds @ Duolingo 08/04/2024 Presented by: Vitor Pellegrino, Zhen Zhou
2. Duolingo’s mission is to develop the best education in the world and make it universally available.
3. Our social presence
4. 26.9M
5. 123.4M
7. Philip Pacheco / Getty Images
10. 5 seconds
11. 4 million users
12. what we will cover General Architecture Deep Dive: Superb Owl
13. General Architecture
14. 100s of microservice s
15. 100s of microservice s 10s of millions of requests/min
16. Technology stack
17. Technology stack Amazon RDS Amazon DynamoDB Amazon Elastic Container Service
18. Why ECS? Why not Kubernetes? Reduced complexity was very important in the beginning AWS native constructs Size of the platform team
19. CI/CD Galaxy Apps gRPC/OpenAPI Shareable terraform modules
21. Zombie mode
22. Zombie mode
23. Zombie mode
24. Jeeves
25. Jeeves
26. Jeeves
27. Jeeves
28. Jeeves Jeeves Zendesk (cs reports, twitter) AppFigures (reviews) Jira (str tickets)
29. Jeeves Jeeves Zendesk (cs reports, twitter) AppFigures (reviews) Jira (str tickets) Shake to report
30. Jeeves Jeeves Zendesk (cs reports, twitter) AppFigures (reviews) Jira (str tickets) external format internal format s3 ElasticSearch Shake to report
31. Freeze Gun
32. TODO: Architecture diagram https://blog.duolingo.com/dogfooding-app/
33. Everything was fine… for a while
34. Introducing VBitor: The Marketing Manager
38. Oh I forgot… there is one last thing
41. Deep Dive: Superb Owl
42. Where did the “impossible” task come from?
43. To send 4M notifications in 5 seconds… Speed (How fast?) Scale (How big?) Timing (What time?) Operating principle: Embrace challenges!
44. Challenge 1: Speed
45. Challenge 2: Scale
46. Challenge 3: Timing
47. Technical Requirements How do we send notifications at the target rate? How do we get all the cloud resources we need in time? How do we ensure resiliency/idempotency?
48. Our solution: Asynchronous!
49. System Diagram
50. Use case A: upload a campaign 1. Engineer creates a campaign with a list of userID. 2. Server acknowledges the request, then asynchronously starts fetching data from DynamoDB. 3. Server puts the parsed data in S3. Logs the result to CloudWatch.
51. Use case B: prepare the send 1. Cloud operation admin scales up the ASG. 2. Engineer scales up workers in the ECS console. 3. Workers fetches data from S3, stores data in memory(a mapping of user -> deviceID). 4. Workers log their “complete” status to CloudWatch.
52. Use case C: hit the GO button 1. Marketing admin hits the “GO” button to send out the campaign. 2. API server dispatches 50+ messages to FIFO SQS queue. 3. Interim workers dispatch 10k+ SQS messages to the next SQS queue. 4. Notification workers send notifications by calling batch APNS/FCM’s API. 5. Workers log their process time to CloudWatch.
53. Does it send notifications at the target rate? Desired send rate: 4M/5s = 800,000 messages/second SQS in-flight message limit: 120,000 messages/second What’s the solution? Batching(size differs by iOS/Android)
54. Can we provision all the cloud resources we need in time? 1. A technical contact from AWS 2. An IEM (Infrastructure Event Management) document 3. Spot instances 4. Dedicated ECS cluster
55. Can we ensure resiliency? We leveraged a FIFO queue from the AWS SQS service: • Dedupe by message identifiers • Has a deduplication window of 5 minutes • Limited capacity (300 requests/second) Alternative: a cache/a table
56. System Diagram 1. Send notifications at the target rate? 2. Can provision all the cloud resources we need in time? 3. Resiliency?
57. Operating principle: Test it first! 1. Throughput tests 2. Cloud resource tests 3. Testing with real users How we tested it
58. This is “how we tested the MVP” 1. Test with silent notifications: a. Bottleneck -> Thread count 2. Test with # of Threads: a. 10 -> 5 -> 1 b. Bottleneck: ??? 3. Test with a larger audience: a. 500k -> 3M b. Bottleneck -> task count 4. Test with # of Processes: a. 1 -> 4 -> 2 How we tested it: throughput
59. This is “how we tested the Cloud” 1. Can we scale up superb-owl? 2. Can we scale up backend? 3. Can we scale up both? 4. Can we scale up both in <3 hours? How we tested it: cloud resources
60. How we tested it: real users October 1M users Novembe r 2M users January 4M users Thanks to Zombie mode, we are more comfortable with testing with real users. Lesson learned: Send yourself a copy before sending to the users.
61. 1. Make system foolproof + Write a foolproof plan How we tested it…(continued)
62. Day of Super Bowl The playbook
63. Campaign Results 99% notifications were out in 5.7 seconds, 95% in 3.9 seconds Figma blog post: The anatomy of a Super Bowl ad Wall Street Journal: Duolingo’s Mascot Has a 'Buttception' in a Five-Second Regional Super Bowl Ad …
64. Summary
65. 1. Building on a solid foundation is important. 2. Be open-minded about design, rigorous with testing. 3. Always build systems with resilience + robustness. 4. Things can always go wrong, so accept it. Lessons learned
66. 1. Python -> Async Python? -> Go? 2. Optimize memory -> Less cost and intervention 3. More automation -> Less error What would we do differently?
67. What is next for us? Platform as a Product Edge computing MLOps
68. Q&A Join a life-changing mission that brings us together! Visit careers.duolingo.com for more information.
69. stay in touch! linkedin.com/in/ zhen-zhou-cmuzz LINKEDIN linkedin.com/in/ vitorpellegrino
70. gracias!