Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Queue it! What job queues can do for you!

Queue it! What job queues can do for you!

Gave this talk at Dutch PHP Conference 2014 in Amsterdam, NL on July 27th, 2014
Follow me on Twitter: @CodeStars

if you have seen this talk, please rate it: https://joind.in/10866

In times of node.js and asynchronous IO, our other beloved languages can sometimes feel like the wrong tool. But looking at it like this would be too narrow-minded - like sending a registration confirmation mail, you can do it in the “register” user request, while the user is waiting for the response. But what if the mail server isn’t reachable? You’ll have to wait for the timeout. This will make your common user pretty angry.

There are tons of different job queues around. I’d like to talk about a few different queues like Gearman and beanstalkd - what’s the differences and what are the perfect match use-cases. I also have some hints on what’s a good implementation detail ready.

Ole Michaelis

June 27, 2014

More Decks by Ole Michaelis

Other Decks in Technology


  1. Queue it! What job queues can do for you! Dutch

    PHP Conference 2014 - June 27th 2014 - Amsterdam, NL
  2. Ole Michaelis @CodeStars // codestars.eu // slidr.io // So Coded

  3. Jimdo

  4. Do You Queue?

  5. you do, you just may not know

  6. None
  7. The Problem!

  8. computers are slow

  9. Webserver External Service H A R D C O R

    E ! W O R K I N G ! A C T I O N W A I T I N G User
  10. None
  11. None
  12. None
  13. None
  14. Queues To The Rescue! yay people https://secure.flickr.com/photos/vorsprach/8997923353/

  15. do it async

  16. Webserver External Service User H A R D C O

    R E ! W O R K I N G ! A C T I O N Queue / Worker
  17. None
  18. None
  19. it’s easy

  20. DEMO!

  21. don’t try this at home i4lcocl2 / shutterstock.com


  23. Message Queues

  24. anatomies of a queue

  25. Producer Broker Consumer Publish Consume

  26. queue strategies

  27. Producer Broker Consumer A Queue 1 Queue 2 Consumer B

    Queue 3 Consumer C fanout pipeline Consumer D pubsub Queue 4 Consumer E request - reply
  28. feature check

  29. message distribution

  30. broadcasting

  31. Job Queues

  32. designed for jobs

  33. feature check

  34. persistence

  35. priorities

  36. 232 20 216 priorities sending mails gen. PM report process

    data cleanup db recover data payment
  37. timing

  38. 12:00 timing - job delay sending batch #1 14:00 16:00

    sending batch #2 sending batch #3 sending batch #4
  39. What Queue Should I Use?


  41. But We Can Check A Few

  42. allrounders

  43. SaaS

  44. no queues

  45. job queues

  46. beanstalkd (mostly general) implementation best practices

  47. ready reserved *poof* put reserve delete

  48. delayed ready reserved buried *poof* *poof* put with delay put

    release with delay reserve delete delete bury release (time passes) kick
  49. jobs will fail you should retry

  50. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ ❌

  51. jobs will fail often use timing for retries

  52. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ ❌

  53. don’t retry infinite count your retries

  54. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ ❌ 3

  55. jobs can’t be done separate them

  56. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ 100 ❌ 99 DEAD LETTER QUEUE

  57. fanout & prioritize scale workers effectively

  58. ✉️✉️✉️ ✉️✉️✉️✉️ ⠎ ⠎ ᶐ ᶓ ᾬ ᾬ ᶓ

  59. watch your queue monitor and measure what is going on

  60. None
  61. None
  62. Yay or Nay?

  63. Thank You

  64. Let’s Talk or just ping me later @codestars or ole@jimdo.com

    RATE ME https://joind.in/10866
  65. None
  66. continuous reading http://blog.thecodepath.com/2013/01/06/ asynchronous-processing-in-web- applications-part-2-developers-need-to- understand-message-queues/ Mathias Meyer - Message

    Queues - Railsberry 2013 https://vimeo.com/ 68698333 https://github.com/kr/beanstalkd https://github.com/dp-opensource/ pheanstalk https://en.wikipedia.org/wiki/ Message_queue https://github.com/denniskuczynski/ beanstalkd_view https://github.com/twitter/kestrel https://github.com/kr/beanstalkd/blob/ master/doc/protocol.txt I want to thank my company Jimdo, my girlfriend and all my coworkers for supporting me!
  67. Products / Name dropping activemq rabbitmq Ømq (zeromq) zookeeper pubnub

    ironmq sqs sidekiq resque kue gearman beanstalkd kestrel kafka delayed_job celery qless mqseries nsq jms qpid hornetq swiftmq mqseries stormmq dotnetmq snakemq dropr
  68. A N D A LWAY S R E M E

    M B E R : S AY S 〿