Solid Queue internals, externals and all the things in between

We’ve used Resque and Redis to run background jobs in multiple apps for many years at 37signals. However, performance, reliability, and our own apps’ idiosyncrasies led us to use a lot of different gems, some developed by us, some forked or patched to address our struggles. After multiple war stories with background jobs, looking at our increasingly complex setup, we wanted something we could use out-of-the-box without having to port our collection of hacks to every new app and with fewer moving pieces. After exploring existing alternatives, we decided to build our own and aim to make it the default for Rails 8. In this talk, I’ll present Solid Queue, explain some of the problems we had over the years, how we designed Solid Queue to address them, and all the Fun™ we had doing that.

Transcript

1. github.com/rosa rosa.codes Principal programmer at 37signals Hi, I’m Rosa 👋
2. Solid Queue internals, externals and all the things in between Rails World 2024 Toronto - September 26th, 2024
4. “I don’t want to look” 🙈
5. “I don’t want to look” 🙈
6. solid_queue repo contributions in 2023 Promises Driven Development™
7. Our Resque setup # Jobs
 gem "resque", "~> 2.0.0"
 gem "resque_supervised_fork", bc: "resque_supervised_fork"
 gem "resque-pool", bc: "resque-pool"
 gem "resque-scheduler", github: "resque/resque-scheduler"
 gem "resque-pause", bc: "resque-pause"
 gem "sequential_jobs", bc: "sequential_jobs"
 gem "scheduled_job", bc: "scheduled_job"
8. Our Resque setup # Jobs
 gem "resque", "~> 2.0.0"
 gem "resque_supervised_fork", bc: "resque_supervised_fork"
 gem "resque-pool", bc: "resque-pool"
 gem "resque-scheduler", github: "resque/resque-scheduler"
 gem "resque-pause", bc: "resque-pause"
 gem "sequential_jobs", bc: "sequential_jobs"
 gem "scheduled_job", bc: "scheduled_job"
9. Our Resque setup # Jobs
 gem "resque", "~> 2.0.0"
 gem "resque_supervised_fork", bc: "resque_supervised_fork"
 gem "resque-pool", bc: "resque-pool"
 gem "resque-scheduler", github: "resque/resque-scheduler"
 gem "resque-pause", bc: "resque-pause"
 gem "sequential_jobs", bc: "sequential_jobs"
 gem "scheduled_job", bc: "scheduled_job" Enqueue and run jobs
10. Our Resque setup # Jobs
 gem "resque", "~> 2.0.0"
 gem "resque_supervised_fork", bc: "resque_supervised_fork"
 gem "resque-pool", bc: "resque-pool"
 gem "resque-scheduler", github: "resque/resque-scheduler"
 gem "resque-pause", bc: "resque-pause"
 gem "sequential_jobs", bc: "sequential_jobs"
 gem "scheduled_job", bc: "scheduled_job" Don’t lose jobs when a worker is killed
11. Our Resque setup # Jobs
 gem "resque", "~> 2.0.0"
 gem "resque_supervised_fork", bc: "resque_supervised_fork"
 gem "resque-pool", bc: "resque-pool"
 gem "resque-scheduler", github: "resque/resque-scheduler"
 gem "resque-pause", bc: "resque-pause"
 gem "sequential_jobs", bc: "sequential_jobs"
 gem "scheduled_job", bc: "scheduled_job" Run and manage multiple workers
12. Our Resque setup # Jobs
 gem "resque", "~> 2.0.0"
 gem "resque_supervised_fork", bc: "resque_supervised_fork"
 gem "resque-pool", bc: "resque-pool"
 gem "resque-scheduler", github: "resque/resque-scheduler"
 gem "resque-pause", bc: "resque-pause"
 gem "sequential_jobs", bc: "sequential_jobs"
 gem "scheduled_job", bc: "scheduled_job" Support delayed and recurring jobs
13. Our Resque setup # Jobs
 gem "resque", "~> 2.0.0"
 gem "resque_supervised_fork", bc: "resque_supervised_fork"
 gem "resque-pool", bc: "resque-pool"
 gem "resque-scheduler", github: "resque/resque-scheduler"
 gem "resque-pause", bc: "resque-pause"
 gem "sequential_jobs", bc: "sequential_jobs"
 gem "scheduled_job", bc: "scheduled_job" Pause queues
14. Our Resque setup # Jobs
 gem "resque", "~> 2.0.0"
 gem "resque_supervised_fork", bc: "resque_supervised_fork"
 gem "resque-pool", bc: "resque-pool"
 gem "resque-scheduler", github: "resque/resque-scheduler"
 gem "resque-pause", bc: "resque-pause"
 gem "sequential_jobs", bc: "sequential_jobs"
 gem "scheduled_job", bc: "scheduled_job" Run some jobs sequentially
15. Our Resque setup # Jobs
 gem "resque", "~> 2.0.0"
 gem "resque_supervised_fork", bc: "resque_supervised_fork"
 gem "resque-pool", bc: "resque-pool"
 gem "resque-scheduler", github: "resque/resque-scheduler"
 gem "resque-pause", bc: "resque-pause"
 gem "sequential_jobs", bc: "sequential_jobs"
 gem "scheduled_job", bc: "scheduled_job" Schedule a large number of jobs in the far-ish future and enqueue them fast in batches
16. Requirements ⬜ Enqueue and run jobs 
 ⬜ Don’t lose jobs when a worker is killed 
 ⬜ Run and manage multiple workers 
 ⬜ Support delayed jobs, a large number, enqueue them fast in batches 
 ⬜ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Support running some jobs sequentially 
 
 

17. Requirements ⬜ Enqueue and run jobs 
 ⬜ Don’t lose jobs when a worker is killed 
 ⬜ Run and manage multiple workers 
 ⬜ Support delayed jobs, a large number, enqueue them fast in batches 
 ⬜ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Support running some jobs sequentially 
 ⬜ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 

18. What if we used a relational database? 🤔
19. What if we used a relational database? 🤔 Solid Cache inspiration
20. What if we used a relational database? 🤔 Solid Cache inspiration Some successful examples: GoodJob, Delayed
21. What if we used a relational database? 🤔 Solid Cache inspiration Leverage Active Record Some successful examples: GoodJob, Delayed
22. What if we used a relational database? 🤔 Solid Cache inspiration Simplicity, working out-of-the-box, one fewer moving piece Leverage Active Record Some successful examples: GoodJob, Delayed
23. What if we used a relational database? 🤔 Solid Cache inspiration Simplicity, working out-of-the-box, one fewer moving piece Leverage Active Record Some successful examples: GoodJob, Delayed Maybe a new Rails default?
24. Requirements ⬜ Enqueue and run jobs 
 ⬜ Don’t lose jobs when a worker is killed 
 ⬜ Run and manage multiple workers 
 ⬜ Support delayed jobs, a large number, enqueue them fast in batches 
 ⬜ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Support running some jobs sequentially 
 ⬜ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 
 

25. Requirements ⬜ Enqueue and run jobs 
 ⬜ Don’t lose jobs when a worker is killed 
 ⬜ Run and manage multiple workers 
 ⬜ Support delayed jobs, a large number, enqueue them fast in batches 
 ⬜ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Support running some jobs sequentially 
 ⬜ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ⬜ Support MySQL, PostgreSQL, SQLite 
 ⬜ Support all Active Job features (priorities) 

26. Requirements ⬜ Enqueue and run jobs 
 ⬜ Don’t lose jobs when a worker is killed 
 ⬜ Run and manage multiple workers 
 ⬜ Support delayed jobs, a large number, enqueue them fast in batches 
 ⬜ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Support running some jobs sequentially 
 ⬜ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ⬜ Support MySQL, PostgreSQL, SQLite 
 ⬜ Support all Active Job features (priorities) 

27. The performance problem with polling A worker locks jobs to claim them The other workers wait
28. The performance problem with polling id queue priority claimed 1 default 0 0 2 default 1 0 3 default 2 0 4 default 1 0 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE; Claim the jobs… 
 Commit transaction… jobs table
29. The performance problem with polling id queue priority claimed 1 default 0 0 2 default 1 0 3 default 2 0 4 default 1 0 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE; Claim the jobs… 
 Commit transaction… jobs table → 1, 2
30. The performance problem with polling id queue priority claimed 1 default 0 0 2 default 1 0 3 default 2 0 4 default 1 0 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE; Claim the jobs… 
 Commit transaction… jobs table → 1, 2
31. The performance problem with polling id queue priority claimed 1 default 0 0 2 default 1 0 3 default 2 0 4 default 1 0 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE; Claim the jobs… 
 Commit transaction… jobs table All other workers waiting → 1, 2
32. The performance problem with polling id queue priority claimed 1 default 0 0 2 default 1 0 3 default 2 0 4 default 1 0 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE; Claim the jobs… 
 Commit transaction… jobs table All other workers waiting → 1, 2
33. The performance problem with polling id queue priority claimed 1 default 0 1 2 default 1 1 3 default 2 0 4 default 1 0 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE; Claim the jobs… 
 Commit transaction… jobs table → 1, 2 All other workers waiting
34. The performance problem with polling id queue priority claimed 1 default 0 1 2 default 1 1 3 default 2 0 4 default 1 0 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE; Claim the jobs… 
 Commit transaction… jobs table → 1, 2 All other workers waiting
35. The performance problem with polling id queue priority claimed 1 default 0 1 2 default 1 1 3 default 2 0 4 default 1 0 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE; Claim the jobs… 
 Commit transaction… jobs table All other workers waiting
36. The performance problem with polling id queue priority claimed 1 default 0 1 2 default 1 1 3 default 2 0 4 default 1 0 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE; Claim the jobs… 
 Commit transaction… jobs table → 4, 3
37. The performance problem with polling id queue priority claimed 1 default 0 1 2 default 1 1 3 default 2 0 4 default 1 0 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE; Claim the jobs… 
 Commit transaction… jobs table → 4, 3
38. Enter SELECT … FOR UPDATE SKIP LOCKED Available in PostgreSQL from version 9.5 (2016) and MySQL 8.0 (2018)
39. Enter SELECT … FOR UPDATE SKIP LOCKED id queue priority claimed 1 default 0 0 2 default 1 0 3 default 2 0 4 default 1 0 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE SKIP LOCKED; Claim the jobs… 
 Commit transaction… jobs table
40. Enter SELECT … FOR UPDATE SKIP LOCKED id queue priority claimed 1 default 0 0 2 default 1 0 3 default 2 0 4 default 1 0 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE SKIP LOCKED; Claim the jobs… 
 Commit transaction… jobs table → 1, 2
41. Enter SELECT … FOR UPDATE SKIP LOCKED id queue priority claimed 1 default 0 0 2 default 1 0 3 default 2 0 4 default 1 0 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE SKIP LOCKED; Claim the jobs… 
 Commit transaction… jobs table → 1, 2
42. Enter SELECT … FOR UPDATE SKIP LOCKED id queue priority claimed 1 default 0 0 2 default 1 0 3 default 2 0 4 default 1 0 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE SKIP LOCKED; Claim the jobs… 
 Commit transaction… jobs table
43. Enter SELECT … FOR UPDATE SKIP LOCKED id queue priority claimed 1 default 0 0 2 default 1 0 3 default 2 0 4 default 1 0 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE SKIP LOCKED; Claim the jobs… 
 Commit transaction… jobs table → 4, 3
44. Enter SELECT … FOR UPDATE SKIP LOCKED id queue priority claimed 1 default 0 0 2 default 1 0 3 default 2 0 4 default 1 0 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE SKIP LOCKED; Claim the jobs… 
 Commit transaction… jobs table → 4, 3
45. Enter SELECT … FOR UPDATE SKIP LOCKED id queue priority claimed 1 default 0 1 2 default 1 1 3 default 2 1 4 default 1 1 Start transaction… SELECT id 
 FROM jobs 
 WHERE queue = "default" 
 AND claimed = 0 
 ORDER BY priority, id 
 LIMIT 2 
 FOR UPDATE SKIP LOCKED; Claim the jobs… 
 Commit transaction… jobs table
46. The first prototype Main idea: keep the polled table as small as possible
47. The first prototype Main idea: keep the polled table as small as possible
48. The first prototype Main idea: keep the polled table as small as possible
49. The first prototype Main idea: keep the polled table as small as possible
50. The first prototype Main idea: keep the polled table as small as possible
51. The first prototype Main idea: keep the polled table as small as possible
52. The first prototype solid_queue_jobs 
 solid_queue_ready_executions 
 solid_queue_claimed_executions 
 solid_queue_scheduled_executions 
 solid_queue_failed_executions job data (class, arguments…) temporary execution data
53. The first prototype poll solid_queue_jobs 
 solid_queue_ready_executions 
 solid_queue_claimed_executions 
 solid_queue_scheduled_executions 
 solid_queue_failed_executions worker dispatcher poll supervisor manage
54. The first prototype move polled 
 jobs here solid_queue_jobs 
 solid_queue_ready_executions 
 solid_queue_claimed_executions 
 solid_queue_scheduled_executions 
 solid_queue_failed_executions worker dispatcher move polled 
 jobs here (in batches)
55. Requirements ⬜ Enqueue and run jobs 
 ⬜ Don’t lose jobs when a worker is killed 
 ⬜ Run and manage multiple workers 
 ⬜ Support delayed jobs, a large number, enqueue them fast in batches 
 ⬜ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Support running some jobs sequentially 
 ⬜ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ⬜ Support MySQL, PostgreSQL, SQLite* 
 ⬜ Support priorities
56. Requirements ✅ Enqueue and run jobs 
 ✅ Don’t lose jobs when a worker is killed 
 ✅ Run and manage multiple workers 
 ✅ Support delayed jobs, a large number, enqueue them fast in batches 
 ⬜ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Support running some jobs sequentially 
 ⬜ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ✅ Support MySQL, PostgreSQL, SQLite* 
 ✅ Support priorities
57. Requirements ✅ Enqueue and run jobs ⬜ Don’t keep a job claimed forever if worker dies 
 ✅ Don’t lose jobs when a worker is killed 
 ✅ Run and manage multiple workers 
 ✅ Support delayed jobs, a large number, enqueue them fast in batches 
 ⬜ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Support running some jobs sequentially 
 ⬜ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ✅ Support MySQL, PostgreSQL, SQLite* 
 ✅ Support priorities
58. Requirements ✅ Enqueue and run jobs ⬜ Don’t keep a job claimed forever if worker dies 
 ✅ Don’t lose jobs when a worker is killed 
 ✅ Run and manage multiple workers 
 ✅ Support delayed jobs, a large number, enqueue them fast in batches 
 ⬜ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Support running some jobs sequentially 
 ⬜ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ✅ Support MySQL, PostgreSQL, SQLite* 
 ✅ Support priorities
59. Process registry and heartbeats 🫀
60. Process registry and heartbeats 🫀 Let’s add another table
61. Process registry and heartbeats 🫀 solid_queue_processes 
 last_heartbeat_at solid_queue_claimed_executions 
 process_id Let’s add another table
62. Process registry and heartbeats 🫀 solid_queue_processes 
 last_heartbeat_at solid_queue_claimed_executions 
 process_id supervisor periodic checks Let’s add another table
63. Requirements ✅ Enqueue and run jobs ⬜ Don’t keep a job claimed forever if worker dies 
 ✅ Don’t lose jobs when a worker is killed 
 ✅ Run and manage multiple workers 
 ✅ Support delayed jobs, a large number, enqueue them fast in batches 
 ⬜ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ✅ Support MySQL, PostgreSQL, SQLite* 
 ✅ Support priorities
64. Requirements ✅ Enqueue and run jobs ✅ Don’t keep a job claimed forever if worker dies 
 ✅ Don’t lose jobs when a worker is killed 
 ✅ Run and manage multiple workers 
 ✅ Support delayed jobs, a large number, enqueue them fast in batches 
 ⬜ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ✅ Support MySQL, PostgreSQL, SQLite* 
 ✅ Support priorities
65. Requirements ✅ Enqueue and run jobs ✅ Don’t keep a job claimed forever if worker dies 
 ⬜ Pause queues
66. Requirements ✅ Enqueue and run jobs ✅ Don’t keep a job claimed forever if worker dies 
 ⬜ Pause queues Yep, another table! 👯 solid_queue_pauses
67. Requirements ✅ Enqueue and run jobs ✅ Don’t keep a job claimed forever if worker dies 
 ✅ Don’t lose jobs when a worker is killed 
 ✅ Run and manage multiple workers 
 ✅ Support delayed jobs, a large number, enqueue them fast in batches 
 ✅ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Support running some jobs sequentially 
 ⬜ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ✅ Support MySQL, PostgreSQL, SQLite* 
 ✅ Support priorities
68. Requirements ✅ Enqueue and run jobs ✅ Don’t keep a job claimed forever if worker dies 
 ✅ Don’t lose jobs when a worker is killed 
 ✅ Run and manage multiple workers 
 ✅ Support delayed jobs, a large number, enqueue them fast in batches 
 ✅ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Support running some jobs sequentially 
 ⬜ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ✅ Support MySQL, PostgreSQL, SQLite* 
 ✅ Support priorities
69. Further tuning the polling queries CREATE TABLE `solid_queue_ready_executions` ( 
 `id` bigint NOT NULL AUTO_INCREMENT, 
 `job_id` bigint DEFAULT NULL, 
 `queue_name` varchar(255) NOT NULL, 
 `priority` int NOT NULL DEFAULT '0', 
 `created_at` datetime(6) NOT NULL, 
 PRIMARY KEY (`id`), 
 UNIQUE KEY `index_solid_queue_ready_executions_on_job_id` (`job_id`),
70. Further tuning the polling queries CREATE TABLE `solid_queue_ready_executions` ( 
 `id` bigint NOT NULL AUTO_INCREMENT, 
 `job_id` bigint DEFAULT NULL, 
 `queue_name` varchar(255) NOT NULL, 
 `priority` int NOT NULL DEFAULT '0', 
 `created_at` datetime(6) NOT NULL, 
 PRIMARY KEY (`id`), 
 UNIQUE KEY `index_solid_queue_ready_executions_on_job_id` (`job_id`),
71. Further tuning the polling queries queues: recycling
 queues: [ incineration, action_mailbox_incineration ] 
 queues: beta*
 queues: * # all queues + paused queues
72. Further tuning the polling queries KEY `index_solid_queue_poll_by_queue` (`queue_name`,`priority`,`job_id`)) KEY `index_solid_queue_poll_all` (`priority`,`job_id`))
73. Further tuning the polling queries KEY `index_solid_queue_poll_by_queue` (`queue_name`,`priority`,`job_id`)) KEY `index_solid_queue_poll_all` (`priority`,`job_id`)) SELECT job_id 
 FROM solid_queue_ready_executions 
 WHERE queue_name = "incineration" 
 ORDER BY priority ASC, job_id ASC 
 LIMIT 4 
 FOR UPDATE SKIP LOCKED;
74. Further tuning the polling queries KEY `index_solid_queue_poll_by_queue` (`queue_name`,`priority`,`job_id`)) KEY `index_solid_queue_poll_all` (`priority`,`job_id`)) SELECT job_id 
 FROM solid_queue_ready_executions 
 WHERE queue_name = "incineration" 
 ORDER BY priority ASC, job_id ASC 
 LIMIT 4 
 FOR UPDATE SKIP LOCKED; SELECT job_id 
 FROM solid_queue_ready_executions 
 ORDER BY priority ASC, job_id ASC 
 LIMIT 4 
 FOR UPDATE SKIP LOCKED;
75. Further tuning the polling queries KEY `index_solid_queue_poll_by_queue` (`queue_name`,`priority`,`job_id`)) KEY `index_solid_queue_poll_all` (`priority`,`job_id`)) Covering indexes SELECT job_id 
 FROM solid_queue_ready_executions 
 WHERE queue_name = "incineration" 
 ORDER BY priority ASC, job_id ASC 
 LIMIT 4 
 FOR UPDATE SKIP LOCKED; SELECT job_id 
 FROM solid_queue_ready_executions 
 ORDER BY priority ASC, job_id ASC 
 LIMIT 4 
 FOR UPDATE SKIP LOCKED;
76. KEY `index_solid_queue_poll_by_queue` (`queue_name`,`priority`,`job_id`)) KEY `index_solid_queue_poll_all` (`priority`,`job_id`)) Current version
77. KEY `index_solid_queue_poll_by_queue` (`queue_name`,`priority`,`job_id`)) KEY `index_solid_queue_poll_all` (`priority`,`job_id`)) Current version Zero or one queue per query
78. KEY `index_solid_queue_poll_by_queue` (`queue_name`,`priority`,`job_id`)) KEY `index_solid_queue_poll_all` (`priority`,`job_id`)) Current version Zero or one queue per query Always the minimum number of queries
79. KEY `index_solid_queue_poll_by_queue` (`queue_name`,`priority`,`job_id`)) KEY `index_solid_queue_poll_all` (`priority`,`job_id`)) Current version 4,900 queries/second 
 110 microseconds/query 
 0.04 examined rows/query Zero or one queue per query Always the minimum number of queries
80. Requirements ✅ Enqueue and run jobs ✅ Don’t keep a job claimed forever if worker dies 
 ✅ Don’t lose jobs when a worker is killed 
 ✅ Run and manage multiple workers 
 ✅ Support delayed jobs, a large number, enqueue them fast in batches 
 ✅ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Support running some jobs sequentially 
 ⬜ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ✅ Support MySQL, PostgreSQL, SQLite* 
 ✅ Support priorities
81. Requirements ✅ Enqueue and run jobs ✅ Don’t keep a job claimed forever if worker dies 
 ✅ Don’t lose jobs when a worker is killed 
 ✅ Run and manage multiple workers 
 ✅ Support delayed jobs, a large number, enqueue them fast in batches 
 ✅ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Support running some jobs sequentially 
 ✅ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ✅ Support MySQL, PostgreSQL, SQLite* 
 ✅ Support priorities
82. Requirements ✅ Enqueue and run jobs ✅ Don’t keep a job claimed forever if worker dies 
 ✅ Don’t lose jobs when a worker is killed 
 ✅ Run and manage multiple workers 
 ✅ Support delayed jobs, a large number, enqueue them fast in batches 
 ✅ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Support running some jobs sequentially 
 ✅ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ✅ Support MySQL, PostgreSQL, SQLite* 
 ✅ Support priorities
83. Requirements ✅ Enqueue and run jobs ✅ Don’t keep a job claimed forever if worker dies 
 ✅ Don’t lose jobs when a worker is killed 
 ✅ Run and manage multiple workers 
 ✅ Support delayed jobs, a large number, enqueue them fast in batches 
 ✅ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Support running some jobs sequentially 
 ✅ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ✅ Support MySQL, PostgreSQL, SQLite* 
 ✅ Support priorities 🫣😰😱
84. Sequential jobs → concurrency controls
85. Sequential jobs → concurrency controls Make order not important
86. Sequential jobs → concurrency controls Limit on enqueuing Make order not important
87. Sequential jobs → concurrency controls Semaphores 🚦 Limit on enqueuing Make order not important
88. Sequential jobs → concurrency controls Semaphores 🚦 Limit on enqueuing Make order not important New execution type: blocked
89. Sequential jobs → concurrency controls Semaphores 🚦 Limit on enqueuing Make order not important New execution type: blocked And yes, you guessed it, more tables! 🕺
90. Now with concurrency controls
91. Now with concurrency controls
92. Many new write ops in our DB
93. Many new write ops in our DB Worst case: 11 write operations for a scheduled job with concurrency limits
94. Many new write ops in our DB Worst case: 11 write operations for a scheduled job with concurrency limits
95. Many new write ops in our DB Peak 400 jobs/second Worst case: 11 write operations for a scheduled job with concurrency limits
96. Many new write ops in our DB Peak 400 jobs/second Worst case: 11 write operations for a scheduled job with concurrency limits
97. Requirements ✅ Enqueue and run jobs ✅ Don’t keep a job claimed forever if worker dies 
 ✅ Don’t lose jobs when a worker is killed 
 ✅ Run and manage multiple workers 
 ✅ Support delayed jobs, a large number, enqueue them fast in batches 
 ✅ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ⬜ Support running some jobs sequentially 
 ✅ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ✅ Support MySQL, PostgreSQL, SQLite* 
 ✅ Support priorities
98. Requirements ✅ Enqueue and run jobs ✅ Don’t keep a job claimed forever if worker dies 
 ✅ Don’t lose jobs when a worker is killed 
 ✅ Run and manage multiple workers 
 ✅ Support delayed jobs, a large number, enqueue them fast in batches 
 ✅ Pause queues 
 ⬜ Support cron-style recurring jobs 
 ✅ Support running some jobs sequentially 
 ✅ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ✅ Support MySQL, PostgreSQL, SQLite* 
 ✅ Support priorities
99. Requirements ✅ Enqueue and run jobs ✅ Don’t keep a job claimed forever if worker dies 
 ⬜ Support cron-style recurring jobs
100. Requirements ✅ Enqueue and run jobs ✅ Don’t keep a job claimed forever if worker dies 
 ⬜ Support cron-style recurring jobs Left as an exercise for the reader* 🤭
101. Requirements ✅ Enqueue and run jobs ✅ Don’t keep a job claimed forever if worker dies 
 ⬜ Support cron-style recurring jobs Left as an exercise for the reader* 🤭 *Hint: add some new tables 💃
102. Requirements ✅ Enqueue and run jobs ✅ Don’t keep a job claimed forever if worker dies 
 ✅ Don’t lose jobs when a worker is killed 
 ✅ Run and manage multiple workers 
 ✅ Support delayed jobs, a large number, enqueue them fast in batches 
 ✅ Pause queues 
 ✅ Support cron-style recurring jobs 
 ✅ Support running some jobs sequentially 
 ✅ Allow for ~650 workers, 100ms polling interval, ~19M jobs / day 
 ✅ Support MySQL, PostgreSQL, SQLite* 
 ✅ Support priorities
103. There’s so much more 🫢 Bulk operations, logging, instrumentation, error handling, hooks, Mission Control dashboard, so many bug fixes, so many edge cases…
104. There’s so much more 🫢 Bulk operations, logging, instrumentation, error handling, hooks, Mission Control dashboard, so many bug fixes, so many edge cases…
105. 📥✉📆
106. ~20M jobs/day 📥✉📆
107. ~20M jobs/day 74 VMs 📥✉📆
108. ~20M jobs/day 74 VMs 800 workers, 4 dispatchers 📥✉📆
109. ~20M jobs/day 74 VMs 800 workers, 4 dispatchers 📥✉📆 Same latency as with Resque for user-facing jobs
110. ~20M jobs/day 74 VMs 800 workers, 4 dispatchers 📥✉📆 Same latency as with Resque for user-facing jobs Speed increase when enqueuing in bulk
111. ~20M jobs/day 74 VMs 800 workers, 4 dispatchers 📥✉📆 Same latency as with Resque for user-facing jobs Speed increase when enqueuing in bulk No impact to response time
112. Good frameworks are extracted, not invented “ DHH
114. Really battle-tested stuff v1.0.0
115. Really battle-tested stuff v1.0.0
116. Really battle-tested stuff v1.0.0
117. Really battle-tested stuff v1.0.0
118. Really battle-tested stuff v1.0.0
119. Really battle-tested stuff v1.0.0
120. Really battle-tested stuff v1.0.0
121. Really battle-tested stuff v1.0.0
122. Really battle-tested stuff v1.0.0
123. Really battle-tested stuff v1.0.0
124. Really battle-tested stuff v1.0.0
125. Really battle-tested stuff v1.0.0
126. Really battle Fun™-tested stuff v1.0.0
127. Really battle Fun™-tested stuff Help us to make it better 🙏🥹 v1.0.0
128. Thank you!
129. Thank you! Find me to talk more about queues, jobs and Fun™, and/or to get some Mochi stickers! 🐶