Queue it! What job queues can do for you!

Gave this talk at Code PaLOUsa 2014 in Louisville, KY on February 26th, 2014
Follow me on Twitter: @CodeStars

In times of node.js and asynchronous IO, our other beloved languages can sometimes feel like the wrong tool. But looking at it like this would be too narrow-minded - like sending a registration confirmation mail, you can do it in the “register” user request, while the user is waiting for the response. But what if the mail server isn’t reachable? You’ll have to wait for the timeout. This will make your common user pretty angry.

There are tons of different job queues around. I’d like to talk about a few different queues like Gearman and beanstalkd - what’s the differences and what are the perfect match use-cases. I also have some hints on what’s a good implementation detail ready.

Transcript

1. Queue it! What job queues can do for you! Codepalousa 2014 - February 26th 2014 - Louisville, KY, USA
4. Ole Michaelis @CodeStars // codestars.eu // slidr.io // So Coded Conference
5. Jimdo
6. Do You Queue?
7. you do, you just may not know
9. The Problem!
10. computers are slow
12. User
13. Webserver User
14. Webserver User
15. Webserver User
16. Webserver External Service User
17. Webserver External Service User
18. Webserver External Service H A R D C O R E ! W O R K I N G ! A C T I O N User
19. Webserver External Service H A R D C O R E ! W O R K I N G ! A C T I O N User
20. Webserver External Service H A R D C O R E ! W O R K I N G ! A C T I O N User
21. Webserver External Service H A R D C O R E ! W O R K I N G ! A C T I O N W A I T I N G User
26. Queues To The Rescue! yay people https://secure.flickr.com/photos/vorsprach/8997923353/
27. do it async
29. Webserver External Service User
30. Webserver External Service User
31. Webserver External Service User
32. Webserver External Service User Queue / Worker
33. Webserver External Service User Queue / Worker
34. Webserver External Service User Queue / Worker
35. Webserver External Service User Queue / Worker
36. Webserver External Service User Queue / Worker
37. Webserver External Service User H A R D C O R E ! W O R K I N G ! A C T I O N Queue / Worker
38. Webserver External Service User H A R D C O R E ! W O R K I N G ! A C T I O N Queue / Worker
42. it’s easy
43. DEMO!
44. DEMO!
45. don’t try this at home i4lcocl2 / shutterstock.com
46. REWIND!
47. REWIND! ALL THE QUEUES CHECK CHECK ALL THE QUEUES
48. Message Queues
49. anatomies of a queue
51. Producer
52. Producer Broker
53. Producer Broker Publish
54. Producer Broker Consumer Publish
55. Producer Broker Consumer Publish Consume
56. queue strategies
58. Producer
59. Producer Broker
60. Producer Broker Queue 1 Queue 2
61. Producer Broker Consumer A Queue 1 Queue 2 Consumer B
62. Producer Broker Consumer A Queue 1 Queue 2 Consumer B fanout
63. Producer Broker Consumer A Queue 1 Queue 2 Consumer B Queue 3 fanout
64. Producer Broker Consumer A Queue 1 Queue 2 Consumer B Queue 3 Consumer C fanout
65. Producer Broker Consumer A Queue 1 Queue 2 Consumer B Queue 3 Consumer C fanout pipeline
66. Producer Broker Consumer A Queue 1 Queue 2 Consumer B Queue 3 Consumer C fanout pipeline Consumer D
67. Producer Broker Consumer A Queue 1 Queue 2 Consumer B Queue 3 Consumer C fanout pipeline Consumer D pubsub
68. Producer Broker Consumer A Queue 1 Queue 2 Consumer B Queue 3 Consumer C fanout pipeline Consumer D pubsub Queue 4
69. Producer Broker Consumer A Queue 1 Queue 2 Consumer B Queue 3 Consumer C fanout pipeline Consumer D pubsub Queue 4 Consumer E
70. Producer Broker Consumer A Queue 1 Queue 2 Consumer B Queue 3 Consumer C fanout pipeline Consumer D pubsub Queue 4 Consumer E
71. Producer Broker Consumer A Queue 1 Queue 2 Consumer B Queue 3 Consumer C fanout pipeline Consumer D pubsub Queue 4 Consumer E request - reply
72. feature check
73. message distribution
74. broadcasting
75. Job Queues
76. designed for jobs
77. feature check
78. persistence
79. priorities
80. 232 20 216 priorities
81. 232 20 216 priorities sending mails
82. 232 20 216 priorities sending mails process data
83. 232 20 216 priorities sending mails process data payment
84. 232 20 216 priorities sending mails process data recover data payment
85. 232 20 216 priorities sending mails process data cleanup db recover data payment
86. 232 20 216 priorities sending mails gen. PM report process data cleanup db recover data payment
87. 232 20 216 priorities sending mails gen. PM report process data cleanup db recover data payment
88. timing
89. 12:00 timing - job delay 14:00 16:00
90. 12:00 timing - job delay sending batch #1 14:00 16:00
91. 12:00 timing - job delay sending batch #1 14:00 16:00 sending batch #2
92. 12:00 timing - job delay sending batch #1 14:00 16:00 sending batch #2 sending batch #3
93. 12:00 timing - job delay sending batch #1 14:00 16:00 sending batch #2 sending batch #3 sending batch #4
94. What Queue Should I Use?
96. USE WHATEVER MAKES FUCKING SENSE!
97. But We Can Check A Few
98. But We Can Check A Few
99. allrounders
100. SaaS
101. no queues
102. job queues
103. beanstalkd (mostly general) implementation best practices
105. ready put
106. ready reserved put reserve
107. ready reserved *poof* put reserve delete
108. ready reserved *poof* put reserve delete
109. delayed ready reserved *poof* put with delay put reserve delete
110. delayed ready reserved *poof* put with delay put reserve delete (time passes)
111. delayed ready reserved *poof* put with delay put reserve delete release (time passes)
112. delayed ready reserved *poof* put with delay put release with delay reserve delete release (time passes)
113. delayed ready reserved buried *poof* put with delay put release with delay reserve delete bury release (time passes)
114. delayed ready reserved buried *poof* put with delay put release with delay reserve delete bury release (time passes) kick
115. delayed ready reserved buried *poof* *poof* put with delay put release with delay reserve delete delete bury release (time passes) kick
116. jobs will fail you should retry
117. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️
118. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ ❌
119. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️
120. jobs will fail often use timing for retries
121. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️
122. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ ❌
123. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️
124. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️
125. don’t retry infinite count your retries
126. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️
127. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ ❌
128. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️
129. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ 3
130. jobs can’t be done separate them
131. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ 99
132. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ 100 ❌
133. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ 100
134. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ 100 DEAD LETTER QUEUE
135. fanout & prioritize scale workers effectively
136. ✉️✉️✉️ ✉️✉️✉️✉️
137. ✉️✉️✉️ ✉️ ✉️ ✉️ ✉️ ✉️✉️✉️✉️
138. ✉️✉️✉️ ✉️ ✉️ ✉️ ✉️ ✉️✉️✉️✉️ ⠎ ⠎ ᶐ ᶓ ᾬ ᾬ ᶓ
139. watch your queue monitor and measure what is going on
142. Yay or Nay?
143. Thank You
144. Let’s Talk or just ping me later @codestars or ole@jimdo.com
146. continuous reading http://blog.thecodepath.com/2013/01/06/ asynchronous-processing-in-web- applications-part-2-developers-need-to- understand-message-queues/ Mathias Meyer - Message Queues - Railsberry 2013 https://vimeo.com/ 68698333 https://github.com/kr/beanstalkd https://github.com/dp-opensource/ pheanstalk https://en.wikipedia.org/wiki/ Message_queue https://github.com/denniskuczynski/ beanstalkd_view https://github.com/twitter/kestrel https://github.com/kr/beanstalkd/blob/ master/doc/protocol.txt I want to thank my company Jimdo, my girlfriend and all my coworkers for supporting me!
147. Products / Name dropping activemq rabbitmq Ømq (zeromq) zookeeper pubnub ironmq sqs sidekiq resque kue gearman beanstalkd kestrel kafka delayed_job celery qless mqseries nsq jms qpid hornetq swiftmq mqseries stormmq dotnetmq snakemq dropr
148. A N D A LWAY S R E M E M B E R : S AY S 〿