Queue it! What job queues can do for you!

4c50caee0c6a446a0b60f827a2a1e5e5?s=47 Ole Michaelis
February 26, 2014

Queue it! What job queues can do for you!

Gave this talk at Code PaLOUsa 2014 in Louisville, KY on February 26th, 2014
Follow me on Twitter: @CodeStars

In times of node.js and asynchronous IO, our other beloved languages can sometimes feel like the wrong tool. But looking at it like this would be too narrow-minded - like sending a registration confirmation mail, you can do it in the “register” user request, while the user is waiting for the response. But what if the mail server isn’t reachable? You’ll have to wait for the timeout. This will make your common user pretty angry.

There are tons of different job queues around. I’d like to talk about a few different queues like Gearman and beanstalkd - what’s the differences and what are the perfect match use-cases. I also have some hints on what’s a good implementation detail ready.

4c50caee0c6a446a0b60f827a2a1e5e5?s=128

Ole Michaelis

February 26, 2014
Tweet

Transcript

  1. Queue it! What job queues can do for you! Codepalousa

    2014 - February 26th 2014 - Louisville, KY, USA
  2. None
  3. None
  4. Ole Michaelis @CodeStars // codestars.eu // slidr.io // So Coded

    Conference
  5. Jimdo

  6. Do You Queue?

  7. you do, you just may not know

  8. None
  9. The Problem!

  10. computers are slow

  11. None
  12. User

  13. Webserver User

  14. Webserver User

  15. Webserver User

  16. Webserver External Service User

  17. Webserver External Service User

  18. Webserver External Service H A R D C O R

    E ! W O R K I N G ! A C T I O N User
  19. Webserver External Service H A R D C O R

    E ! W O R K I N G ! A C T I O N User
  20. Webserver External Service H A R D C O R

    E ! W O R K I N G ! A C T I O N User
  21. Webserver External Service H A R D C O R

    E ! W O R K I N G ! A C T I O N W A I T I N G User
  22. None
  23. None
  24. None
  25. None
  26. Queues To The Rescue! yay people https://secure.flickr.com/photos/vorsprach/8997923353/

  27. do it async

  28. None
  29. Webserver External Service User

  30. Webserver External Service User

  31. Webserver External Service User

  32. Webserver External Service User Queue / Worker

  33. Webserver External Service User Queue / Worker

  34. Webserver External Service User Queue / Worker

  35. Webserver External Service User Queue / Worker

  36. Webserver External Service User Queue / Worker

  37. Webserver External Service User H A R D C O

    R E ! W O R K I N G ! A C T I O N Queue / Worker
  38. Webserver External Service User H A R D C O

    R E ! W O R K I N G ! A C T I O N Queue / Worker
  39. None
  40. None
  41. None
  42. it’s easy

  43. DEMO!

  44. DEMO!

  45. don’t try this at home i4lcocl2 / shutterstock.com

  46. REWIND!

  47. REWIND! ALL THE QUEUES CHECK CHECK ALL THE QUEUES

  48. Message Queues

  49. anatomies of a queue

  50. None
  51. Producer

  52. Producer Broker

  53. Producer Broker Publish

  54. Producer Broker Consumer Publish

  55. Producer Broker Consumer Publish Consume

  56. queue strategies

  57. None
  58. Producer

  59. Producer Broker

  60. Producer Broker Queue 1 Queue 2

  61. Producer Broker Consumer A Queue 1 Queue 2 Consumer B

  62. Producer Broker Consumer A Queue 1 Queue 2 Consumer B

    fanout
  63. Producer Broker Consumer A Queue 1 Queue 2 Consumer B

    Queue 3 fanout
  64. Producer Broker Consumer A Queue 1 Queue 2 Consumer B

    Queue 3 Consumer C fanout
  65. Producer Broker Consumer A Queue 1 Queue 2 Consumer B

    Queue 3 Consumer C fanout pipeline
  66. Producer Broker Consumer A Queue 1 Queue 2 Consumer B

    Queue 3 Consumer C fanout pipeline Consumer D
  67. Producer Broker Consumer A Queue 1 Queue 2 Consumer B

    Queue 3 Consumer C fanout pipeline Consumer D pubsub
  68. Producer Broker Consumer A Queue 1 Queue 2 Consumer B

    Queue 3 Consumer C fanout pipeline Consumer D pubsub Queue 4
  69. Producer Broker Consumer A Queue 1 Queue 2 Consumer B

    Queue 3 Consumer C fanout pipeline Consumer D pubsub Queue 4 Consumer E
  70. Producer Broker Consumer A Queue 1 Queue 2 Consumer B

    Queue 3 Consumer C fanout pipeline Consumer D pubsub Queue 4 Consumer E
  71. Producer Broker Consumer A Queue 1 Queue 2 Consumer B

    Queue 3 Consumer C fanout pipeline Consumer D pubsub Queue 4 Consumer E request - reply
  72. feature check

  73. message distribution

  74. broadcasting

  75. Job Queues

  76. designed for jobs

  77. feature check

  78. persistence

  79. priorities

  80. 232 20 216 priorities

  81. 232 20 216 priorities sending mails

  82. 232 20 216 priorities sending mails process data

  83. 232 20 216 priorities sending mails process data payment

  84. 232 20 216 priorities sending mails process data recover data

    payment
  85. 232 20 216 priorities sending mails process data cleanup db

    recover data payment
  86. 232 20 216 priorities sending mails gen. PM report process

    data cleanup db recover data payment
  87. 232 20 216 priorities sending mails gen. PM report process

    data cleanup db recover data payment
  88. timing

  89. 12:00 timing - job delay 14:00 16:00

  90. 12:00 timing - job delay sending batch #1 14:00 16:00

  91. 12:00 timing - job delay sending batch #1 14:00 16:00

    sending batch #2
  92. 12:00 timing - job delay sending batch #1 14:00 16:00

    sending batch #2 sending batch #3
  93. 12:00 timing - job delay sending batch #1 14:00 16:00

    sending batch #2 sending batch #3 sending batch #4
  94. What Queue Should I Use?

  95. None
  96. USE WHATEVER MAKES FUCKING SENSE!

  97. But We Can Check A Few

  98. But We Can Check A Few

  99. allrounders

  100. SaaS

  101. no queues

  102. job queues

  103. beanstalkd (mostly general) implementation best practices

  104. None
  105. ready put

  106. ready reserved put reserve

  107. ready reserved *poof* put reserve delete

  108. ready reserved *poof* put reserve delete

  109. delayed ready reserved *poof* put with delay put reserve delete

  110. delayed ready reserved *poof* put with delay put reserve delete

    (time passes)
  111. delayed ready reserved *poof* put with delay put reserve delete

    release (time passes)
  112. delayed ready reserved *poof* put with delay put release with

    delay reserve delete release (time passes)
  113. delayed ready reserved buried *poof* put with delay put release

    with delay reserve delete bury release (time passes)
  114. delayed ready reserved buried *poof* put with delay put release

    with delay reserve delete bury release (time passes) kick
  115. delayed ready reserved buried *poof* *poof* put with delay put

    release with delay reserve delete delete bury release (time passes) kick
  116. jobs will fail you should retry

  117. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️

  118. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ ❌

  119. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️

  120. jobs will fail often use timing for retries

  121. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️

  122. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ ❌

  123. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️

  124. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️

  125. don’t retry infinite count your retries

  126. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️

  127. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ ❌

  128. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️

  129. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ 3

  130. jobs can’t be done separate them

  131. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ 99

  132. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ 100 ❌

  133. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ 100

  134. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ 100 DEAD LETTER QUEUE

  135. fanout & prioritize scale workers effectively

  136. ✉️✉️✉️ ✉️✉️✉️✉️

  137. ✉️✉️✉️ ✉️ ✉️ ✉️ ✉️ ✉️✉️✉️✉️

  138. ✉️✉️✉️ ✉️ ✉️ ✉️ ✉️ ✉️✉️✉️✉️ ⠎ ⠎ ᶐ ᶓ

    ᾬ ᾬ ᶓ
  139. watch your queue monitor and measure what is going on

  140. None
  141. None
  142. Yay or Nay?

  143. Thank You

  144. Let’s Talk or just ping me later @codestars or ole@jimdo.com

  145. None
  146. continuous reading http://blog.thecodepath.com/2013/01/06/ asynchronous-processing-in-web- applications-part-2-developers-need-to- understand-message-queues/ Mathias Meyer - Message

    Queues - Railsberry 2013 https://vimeo.com/ 68698333 https://github.com/kr/beanstalkd https://github.com/dp-opensource/ pheanstalk https://en.wikipedia.org/wiki/ Message_queue https://github.com/denniskuczynski/ beanstalkd_view https://github.com/twitter/kestrel https://github.com/kr/beanstalkd/blob/ master/doc/protocol.txt I want to thank my company Jimdo, my girlfriend and all my coworkers for supporting me!
  147. Products / Name dropping activemq rabbitmq Ømq (zeromq) zookeeper pubnub

    ironmq sqs sidekiq resque kue gearman beanstalkd kestrel kafka delayed_job celery qless mqseries nsq jms qpid hornetq swiftmq mqseries stormmq dotnetmq snakemq dropr
  148. A N D A LWAY S R E M E

    M B E R : S AY S 〿