Zero downtime migration of our distributed task queue system

Set sail with us as we navigate the complex waters of migrating a critical task queue system, all without causing a single ripple of downtime.

This session will explore the reasons why we switched systems, and the strategic use of feature flags and robust observability in ensuring a seamless transition.

We share our lessons learned so that you can gain insights that can guide your own system migrations.

Transcript

1. Zero Downtime Migration of Our Distributed Task Queue System Edmund Lam Staff Software Developer @ Poka
2. At first… Now…
3. Why this talk matters: Change needed System struggles Requirements change Build it, and it works Systems evolve - Change is inevitable
4. 6 key lessons
5. What Does a Distributed Task Queue Do?
6. What Does a Distributed Task Queue Do?
7. What Does a Distributed Task Queue Do?
8. What Does a Distributed Task Queue Do?
10. 475K+ Users 70+ Countries 36 Supported Languages 1,600+ Factories Poka by the Numbers
11. Lesson 1: The First System Wasn’t a Mistake – It Just Reached Its Limits
12. The ASDI metric Alexandre Sleep Deprivation Index • Alexandre • Adrien • Arthur • Aube • sAm Our on-call team:
13. Why it was breaking?
14. Why it was breaking? 💥 💥 💥 Large Blast Radius!
15. Paging Alex 🥱→ 󰞵 ☎→ 🛌
16. Lesson 2: Migrations Are Risk Management, Not Just Tech
17. A backlog of tasks should not bring down the system. Setting Non-Functional Requirements Availability It should be easy to see the number of tasks in the queue. Observability The system must scale automatically based on task queue usage. Scalability It should be easy to manage and extend. Maintainability
18. Potential Technologies - Which fits our requirements?
19. Before After
20. Lesson 2 (continued): Migrations Are Risk Management, Not Just Tech
21. The Migration itself is a risk How can we ensure zero downtime?
22. Risk Management - Big Bang Migration Flip a switch, config, update… then
23. Risk Management - Gradual Rollout
24. Lesson 3: Feature Flags Give You Control Over Change.
25. Feature flags Trunk flags Ops flags • Permanent • Turn parts of the app on and off • Transient • Deploy changes/fixes • Long lived • Configurations Our Flag Structure
26. Dynamic Task Routing using an Ops Flag Flag Check
27. Default Rule Run in Celery for these tasks Flag check based on task name
28. Rollouts based on region Only in this region
29. Lesson 4: Observability is a Superpower
30. Track Region Based Celery Task Rollout
31. Track Queue Backlog and Scaling Metrics
32. Problematic Tasks The new system brought in new constraints: 1. No time limit --> 3 minute time limit 2. Undefined max payload size --> 256 KiB max payload size Use observability to detect issues beforehand
33. Confirming fixes to problematic tasks
34. Lesson 5: You’ll Never Predict Every Edge Case – So Build for Adaptability
35. Resilient Systems
36. What if we need to switch back? Actually, this isn’t working, we need to run this in huey
37. Check the flag and requeue if needed Flag Check - Tasks in flight redirected to huey
38. 5. Build for Adaptability: Changes to our transition strategy
39. Horizontal Rollout Before Phase 1 After Both systems deployed, tasks routed mostly to huey Both systems deployed, tasks migrated to Celery Phase 2
40. Phased Rollout Before Phase 1 After Vertical Rollout to a subset of large instances Horizontal Rollout to Remaining instances Phase 2
41. Lesson 6: Documentation Helps You Make (and Defend) Decisions
42. RFC / Tech Designs
43. ADRs: Architectural Decision Records 1. Capture the context of a decision 2. Provide a decision snapshot 3. Help teams move forward, even without full consensus 4. Save time in the long run 5. Help prepare talks
44. 6 key lessons
45. Wrap up our journey The First System wasn’t a Mistake - it reached its limits Migrations are Risk Management, not just a Tech upgrade Feature Flags Give you Control over Change Observability is a Superpower You Can’t Predict every Edge Case - So build for Adaptability Documentation helps you make (and defend) decisions The old ship served us well Navigating choppy waters takes careful planning Adjust the sails to steer the winds A captain’s best friend is a good map and compass Prepare for rogue waves A captain’s log keeps the course accountable
46. Zero Downtime Migration of Our Distributed Task Queue System Edmund Lam Confoo 2025 Thank You!