Link
Embed
Share
Beginning
This slide
Copy link URL
Copy link URL
Copy iframe embed code
Copy iframe embed code
Copy javascript embed code
Copy javascript embed code
Share
Tweet
Share
Tweet
Slide 1
Slide 1 text
Lessons learned in Building RQ a talk by Vincent Driessen (@nvie)
Slide 2
Slide 2 text
@nvie
Slide 3
Slide 3 text
http://nvie.com
Slide 4
Slide 4 text
3rd Cloud
Slide 5
Slide 5 text
I open source software
Slide 6
Slide 6 text
http://python-rq.org
Slide 7
Slide 7 text
It’s a job queue library.
Slide 8
Slide 8 text
It’s simple.
Slide 9
Slide 9 text
The Problem You have this awesome web app.
Slide 10
Slide 10 text
The Problem But now you want to send an email.
Slide 11
Slide 11 text
The Problem But now you want to send an email. Or do some data mining. Or warm up a cache. Or generate thumbnails.
Slide 12
Slide 12 text
The Problem And you don't want to do it in the request-response cycle.
Slide 13
Slide 13 text
The Problem So you look at existing solutions.
Slide 14
Slide 14 text
No content
Slide 15
Slide 15 text
(nothing else, really)
Slide 16
Slide 16 text
The Problem So you go for Celery.
Slide 17
Slide 17 text
The Problem But now you've got two problems.
Slide 18
Slide 18 text
Slide 19
Slide 19 text
= awesome
Slide 20
Slide 20 text
= POWERFUL
Slide 21
Slide 21 text
= RELIABLE
Slide 22
Slide 22 text
Slide 23
Slide 23 text
The Problem != SIMPLE
Slide 24
Slide 24 text
The Problem You have to learn about Rabbit MQ, AMQP, channels, exchanges, routing keys, configure a result store, run DB migrations, etc.
Slide 25
Slide 25 text
No content
Slide 26
Slide 26 text
All I want is to put some work in the background.
Slide 27
Slide 27 text
RQ
Slide 28
Slide 28 text
Example Web app Worker Queue
Slide 29
Slide 29 text
Example Web app Worker Queue
Slide 30
Slide 30 text
Example Web app Worker Queue
Slide 31
Slide 31 text
Example Web app Worker Queue
Slide 32
Slide 32 text
Example Web app Worker Queue
Slide 33
Slide 33 text
Example Web app Worker Queue
Slide 34
Slide 34 text
Example Web app Worker Queue
Slide 35
Slide 35 text
Example Web app Worker Queue
Slide 36
Slide 36 text
Example Web app Worker Queue
Slide 37
Slide 37 text
Example Web app Worker Queue
Slide 38
Slide 38 text
Scaling Web app Worker Queue Worker
Slide 39
Slide 39 text
Scaling Web app Worker Queue Worker Queue may hand out each job only once!
Slide 40
Slide 40 text
Scaling Web app Worker Queue Worker Web app
Slide 41
Slide 41 text
Scaling Web app Worker Queue Worker Web app Queue must handle concurrent writes!
Slide 42
Slide 42 text
Scaling Web app Worker Worker Web app ?
Slide 43
Slide 43 text
No content
Slide 44
Slide 44 text
Super awesome key-value store
Slide 45
Slide 45 text
myqueue
Slide 46
Slide 46 text
> RPUSH myqueue bar (integer) 1 myqueue
Slide 47
Slide 47 text
> RPUSH myqueue qux baz (integer) 3 myqueue
Slide 48
Slide 48 text
myqueue
Slide 49
Slide 49 text
> LPOP myqueue myqueue
Slide 50
Slide 50 text
> LPOP myqueue "bar" myqueue
Slide 51
Slide 51 text
A PERFECT QUEUE (with FIFO semantics)
Slide 52
Slide 52 text
Web app does RPUSH Worker does LPOP
Slide 53
Slide 53 text
Simplicity + Reliability Principle
Slide 54
Slide 54 text
Principle Simplicity ≥ Reliability
Slide 55
Slide 55 text
Principle Simplicity ≥ Reliability ≥ Feature richness
Slide 56
Slide 56 text
Principle Simplicity ≥ Reliability ≥ Feature richness ≥ Performance
Slide 57
Slide 57 text
Simplicity “It is under 20 kB in size.” “It is under 500 lines of code.”
Slide 58
Slide 58 text
Simplicity Zero-config No need to define queues You only need a Redis connection
Slide 59
Slide 59 text
Simplicity Zero-config No need to define queues You only need a Redis connection You probably have that already!
Slide 60
Slide 60 text
Simplicity q = Queue(…) q.enqueue(…)
Slide 61
Slide 61 text
Simplicity q = Queue(connection=redis) q.enqueue(…)
Slide 62
Slide 62 text
Simplicity q = Queue(connection=redis) q.enqueue(send_mail, '
[email protected]
')
Slide 63
Slide 63 text
Simplicity send_mail ( '
[email protected]
')
Slide 64
Slide 64 text
No content
Slide 65
Slide 65 text
send_mail('
[email protected]
', template='foo') (send_mail, ('
[email protected]
',), {'template': 'foo'})
Slide 66
Slide 66 text
(send_mail, ('
[email protected]
',), {'template': 'foo'}) "(cbar\nsend_mail \np0\n(S'
[email protected]
'\np1\ntp2\n(dp3\nS'template'\np 4\nS'foo'\np5\nstp6\n."
Slide 67
Slide 67 text
Reliability Warm shutdown Cold shutdowns Never lose work Jobs should run isolated
Slide 68
Slide 68 text
Reliability 1x Ctrl+C asks to stop politely 2x Ctrl+C terminates any running job
Slide 69
Slide 69 text
PTGPSL Reliability
Slide 70
Slide 70 text
Reliability Reliability ≥ Performance
Slide 71
Slide 71 text
Reliability Pssst… but something cool is cooking!
Slide 72
Slide 72 text
Maintaining open source projects ROCKS
Slide 73
Slide 73 text
Hey, look at this awesome feature I built!
Slide 74
Slide 74 text
I’ve spent countless hours building it but it’s finally finished! Hey, look at this awesome feature I built!
Slide 75
Slide 75 text
I’ve spent countless hours building it but it’s finally finished! Hey, look at this awesome feature I built! ZOMG, that’s awesome! *drools*
Slide 76
Slide 76 text
Maintaining open source projects SUCKS
Slide 77
Slide 77 text
Hey, look at this awesome feature I built!
Slide 78
Slide 78 text
Hey, look at this awesome feature I built! Don’t want it.
Slide 79
Slide 79 text
But… I’ve spent countless hours building it! Hey, look at this awesome feature I built! Don’t want it.
Slide 80
Slide 80 text
But… I’ve spent countless hours building it! Hey, look at this awesome feature I built! Don’t want it. Still don’t want it.
Slide 81
Slide 81 text
Maintaining open source projects means rejecting sometimes
Slide 82
Slide 82 text
Every contributor has best intentions
Slide 83
Slide 83 text
$ rqworker --reliable
Slide 84
Slide 84 text
$ php --reliable ^_^
Slide 85
Slide 85 text
Never dismiss a person, only features
Slide 86
Slide 86 text
Maintainers need to keep projects healthy and moving forward
Slide 87
Slide 87 text
Try to be constructive
Slide 88
Slide 88 text
Hard when you’re not judging work rationally
Slide 89
Slide 89 text
Having core project values does help
Slide 90
Slide 90 text
Having core project values does help shared!
Slide 91
Slide 91 text
Having core project values does help e.g. “simplicity ≥ reliability” shared!
Slide 92
Slide 92 text
RQ is simple
Slide 93
Slide 93 text
It’s opinionated
Slide 94
Slide 94 text
It’s opinionated (That’s why it’s simple)
Slide 95
Slide 95 text
Redis was a deliberate choice
Slide 96
Slide 96 text
Pickle was a deliberate choice
Slide 97
Slide 97 text
Unix was a deliberate choice
Slide 98
Slide 98 text
Python 2.6+ was a deliberate choice
Slide 99
Slide 99 text
RQ gets all of that awesomeness for free
Slide 100
Slide 100 text
(Still no Python 3, sorry)
Slide 101
Slide 101 text
ROADMAP
Slide 102
Slide 102 text
No content
Slide 103
Slide 103 text
0.4
Slide 104
Slide 104 text
Finally! Concurrent workers
Slide 105
Slide 105 text
Pre-forking worker Gevent worker Concurrent workers
Slide 106
Slide 106 text
No content
Slide 107
Slide 107 text
0.5
Slide 108
Slide 108 text
Integration with the excellent rq-scheduler by Selwin Ong Scheduler
Slide 109
Slide 109 text
Automatic retries Periodic jobs Scheduler
Slide 110
Slide 110 text
Thank you!
Slide 111
Slide 111 text
@nvie nvie/rq http://python-rq.org