Lessons learned in building RQ

Creating libraries is easy. Creating libraries that are pleasurable to write and use is another thing. This talk discusses RQ (a simple asynchronous job queue library for Python) as an example of the latter. After introducing the project itself, I reflect on the reasons for its existence and the key design principles that were applied when shaping the project.

I gave this presentation at the Python devroom at FOSDEM'13 in Brussels.

Transcript

1. Lessons learned in Building RQ a talk by Vincent Driessen (@nvie)
2. @nvie
3. http://nvie.com
4. 3rd Cloud
5. I open source software
6. http://python-rq.org
7. It’s a job queue library.
8. It’s simple.
9. The Problem You have this awesome web app.
10. The Problem But now you want to send an email.
11. The Problem But now you want to send an email. Or do some data mining. Or warm up a cache. Or generate thumbnails.
12. The Problem And you don't want to do it in the request-response cycle.
13. The Problem So you look at existing solutions.
15. (nothing else, really)
16. The Problem So you go for Celery.
17. The Problem But now you've got two problems.
18. <disclaimer>
19. = awesome
20. = POWERFUL
21. = RELIABLE
22. </disclaimer>
23. The Problem != SIMPLE
24. The Problem You have to learn about Rabbit MQ, AMQP, channels, exchanges, routing keys, configure a result store, run DB migrations, etc.
26. All I want is to put some work in the background.
27. RQ
28. Example Web app Worker Queue
29. Example Web app Worker Queue
30. Example Web app Worker Queue
31. Example Web app Worker Queue
32. Example Web app Worker Queue
33. Example Web app Worker Queue
34. Example Web app Worker Queue
35. Example Web app Worker Queue
36. Example Web app Worker Queue
37. Example Web app Worker Queue
38. Scaling Web app Worker Queue Worker
39. Scaling Web app Worker Queue Worker Queue may hand out each job only once!
40. Scaling Web app Worker Queue Worker Web app
41. Scaling Web app Worker Queue Worker Web app Queue must handle concurrent writes!
42. Scaling Web app Worker Worker Web app ?
44. Super awesome key-value store
45. myqueue
46. > RPUSH myqueue bar (integer) 1 myqueue
47. > RPUSH myqueue qux baz (integer) 3 myqueue
48. myqueue
49. > LPOP myqueue myqueue
50. > LPOP myqueue "bar" myqueue
51. A PERFECT QUEUE (with FIFO semantics)
52. Web app does RPUSH Worker does LPOP
53. Simplicity + Reliability Principle
54. Principle Simplicity ≥ Reliability
55. Principle Simplicity ≥ Reliability ≥ Feature richness
56. Principle Simplicity ≥ Reliability ≥ Feature richness ≥ Performance
57. Simplicity “It is under 20 kB in size.” “It is under 500 lines of code.”
58. Simplicity Zero-config No need to define queues You only need a Redis connection
59. Simplicity Zero-config No need to define queues You only need a Redis connection You probably have that already!
60. Simplicity q = Queue(…) q.enqueue(…)
61. Simplicity q = Queue(connection=redis) q.enqueue(…)
62. Simplicity q = Queue(connection=redis) q.enqueue(send_mail, 'me@nvie.com')
63. Simplicity send_mail ( 'me@nvie.com')
65. send_mail('me@nvie.com', template='foo') (send_mail, ('me@nvie.com',), {'template': 'foo'})
66. (send_mail, ('me@nvie.com',), {'template': 'foo'}) "(cbar\nsend_mail \np0\n(S'me@nvie.com'\np1\ntp2\n(dp3\nS'template'\np 4\nS'foo'\np5\nstp6\n."
67. Reliability Warm shutdown Cold shutdowns Never lose work Jobs should run isolated
68. Reliability 1x Ctrl+C asks to stop politely 2x Ctrl+C terminates any running job
69. PTGPSL Reliability
70. Reliability Reliability ≥ Performance
71. Reliability Pssst… but something cool is cooking!
72. Maintaining open source projects ROCKS
73. Hey, look at this awesome feature I built!
74. I’ve spent countless hours building it but it’s finally finished! Hey, look at this awesome feature I built!
75. I’ve spent countless hours building it but it’s finally finished! Hey, look at this awesome feature I built! ZOMG, that’s awesome! *drools*
76. Maintaining open source projects SUCKS
77. Hey, look at this awesome feature I built!
78. Hey, look at this awesome feature I built! Don’t want it.
79. But… I’ve spent countless hours building it! Hey, look at this awesome feature I built! Don’t want it.
80. But… I’ve spent countless hours building it! Hey, look at this awesome feature I built! Don’t want it. Still don’t want it.
81. Maintaining open source projects means rejecting sometimes
82. Every contributor has best intentions
83. $ rqworker --reliable
84. $ php --reliable ^_^
85. Never dismiss a person, only features
86. Maintainers need to keep projects healthy and moving forward
87. Try to be constructive
88. Hard when you’re not judging work rationally
89. Having core project values does help
90. Having core project values does help shared!
91. Having core project values does help e.g. “simplicity ≥ reliability” shared!
92. RQ is simple
93. It’s opinionated
94. It’s opinionated (That’s why it’s simple)
95. Redis was a deliberate choice
96. Pickle was a deliberate choice
97. Unix was a deliberate choice
98. Python 2.6+ was a deliberate choice
99. RQ gets all of that awesomeness for free
100. (Still no Python 3, sorry)
101. ROADMAP
103. 0.4
104. Finally! Concurrent workers
105. Pre-forking worker Gevent worker Concurrent workers
107. 0.5
108. Integration with the excellent rq-scheduler by Selwin Ong Scheduler
109. Automatic retries Periodic jobs Scheduler
110. Thank you!
111. @nvie nvie/rq http://python-rq.org