Queue it! What job queues can do for you!

Queue it! What job queues can do for you!

Gave this talk at Dutch PHP Conference 2014 in Amsterdam, NL on July 27th, 2014
Follow me on Twitter: @CodeStars

if you have seen this talk, please rate it: https://joind.in/10866

In times of node.js and asynchronous IO, our other beloved languages can sometimes feel like the wrong tool. But looking at it like this would be too narrow-minded - like sending a registration confirmation mail, you can do it in the “register” user request, while the user is waiting for the response. But what if the mail server isn’t reachable? You’ll have to wait for the timeout. This will make your common user pretty angry.

There are tons of different job queues around. I’d like to talk about a few different queues like Gearman and beanstalkd - what’s the differences and what are the perfect match use-cases. I also have some hints on what’s a good implementation detail ready.

4c50caee0c6a446a0b60f827a2a1e5e5?s=128

Ole Michaelis

June 27, 2014
Tweet

Transcript

  1. 1.

    Queue it! What job queues can do for you! Dutch

    PHP Conference 2014 - June 27th 2014 - Amsterdam, NL
  2. 3.
  3. 6.
  4. 9.

    Webserver External Service H A R D C O R

    E ! W O R K I N G ! A C T I O N W A I T I N G User
  5. 10.
  6. 11.
  7. 12.
  8. 13.
  9. 16.

    Webserver External Service User H A R D C O

    R E ! W O R K I N G ! A C T I O N Queue / Worker
  10. 17.
  11. 18.
  12. 20.
  13. 27.

    Producer Broker Consumer A Queue 1 Queue 2 Consumer B

    Queue 3 Consumer C fanout pipeline Consumer D pubsub Queue 4 Consumer E request - reply
  14. 36.

    232 20 216 priorities sending mails gen. PM report process

    data cleanup db recover data payment
  15. 37.
  16. 38.

    12:00 timing - job delay sending batch #1 14:00 16:00

    sending batch #2 sending batch #3 sending batch #4
  17. 43.
  18. 44.
  19. 48.

    delayed ready reserved buried *poof* *poof* put with delay put

    release with delay reserve delete delete bury release (time passes) kick
  20. 60.
  21. 61.
  22. 63.
  23. 65.
  24. 66.

    continuous reading http://blog.thecodepath.com/2013/01/06/ asynchronous-processing-in-web- applications-part-2-developers-need-to- understand-message-queues/ Mathias Meyer - Message

    Queues - Railsberry 2013 https://vimeo.com/ 68698333 https://github.com/kr/beanstalkd https://github.com/dp-opensource/ pheanstalk https://en.wikipedia.org/wiki/ Message_queue https://github.com/denniskuczynski/ beanstalkd_view https://github.com/twitter/kestrel https://github.com/kr/beanstalkd/blob/ master/doc/protocol.txt I want to thank my company Jimdo, my girlfriend and all my coworkers for supporting me!
  25. 67.

    Products / Name dropping activemq rabbitmq Ømq (zeromq) zookeeper pubnub

    ironmq sqs sidekiq resque kue gearman beanstalkd kestrel kafka delayed_job celery qless mqseries nsq jms qpid hornetq swiftmq mqseries stormmq dotnetmq snakemq dropr
  26. 68.

    A N D A LWAY S R E M E

    M B E R : S AY S 〿