Queue it! What job queues can do for you!

Gave this talk at Dutch PHP Conference 2014 in Amsterdam, NL on July 27th, 2014
Follow me on Twitter: @CodeStars

if you have seen this talk, please rate it: https://joind.in/10866

In times of node.js and asynchronous IO, our other beloved languages can sometimes feel like the wrong tool. But looking at it like this would be too narrow-minded - like sending a registration confirmation mail, you can do it in the “register” user request, while the user is waiting for the response. But what if the mail server isn’t reachable? You’ll have to wait for the timeout. This will make your common user pretty angry.

There are tons of different job queues around. I’d like to talk about a few different queues like Gearman and beanstalkd - what’s the differences and what are the perfect match use-cases. I also have some hints on what’s a good implementation detail ready.

Transcript

1. Queue it! What job queues can do for you! Dutch PHP Conference 2014 - June 27th 2014 - Amsterdam, NL
2. Ole Michaelis @CodeStars // codestars.eu // slidr.io // So Coded Conference
3. Jimdo
4. Do You Queue?
5. you do, you just may not know
7. The Problem!
8. computers are slow
9. Webserver External Service H A R D C O R E ! W O R K I N G ! A C T I O N W A I T I N G User
14. Queues To The Rescue! yay people https://secure.flickr.com/photos/vorsprach/8997923353/
15. do it async
16. Webserver External Service User H A R D C O R E ! W O R K I N G ! A C T I O N Queue / Worker
19. it’s easy
20. DEMO!
21. don’t try this at home i4lcocl2 / shutterstock.com
22. REWIND! ALL THE QUEUES CHECK CHECK ALL THE QUEUES
23. Message Queues
24. anatomies of a queue
25. Producer Broker Consumer Publish Consume
26. queue strategies
27. Producer Broker Consumer A Queue 1 Queue 2 Consumer B Queue 3 Consumer C fanout pipeline Consumer D pubsub Queue 4 Consumer E request - reply
28. feature check
29. message distribution
30. broadcasting
31. Job Queues
32. designed for jobs
33. feature check
34. persistence
35. priorities
36. 232 20 216 priorities sending mails gen. PM report process data cleanup db recover data payment
37. timing
38. 12:00 timing - job delay sending batch #1 14:00 16:00 sending batch #2 sending batch #3 sending batch #4
39. What Queue Should I Use?
40. USE WHATEVER MAKES FUCKING SENSE!
41. But We Can Check A Few
42. allrounders
43. SaaS
44. no queues
45. job queues
46. beanstalkd (mostly general) implementation best practices
47. ready reserved *poof* put reserve delete
48. delayed ready reserved buried *poof* *poof* put with delay put release with delay reserve delete delete bury release (time passes) kick
49. jobs will fail you should retry
50. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ ❌
51. jobs will fail often use timing for retries
52. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ ❌
53. don’t retry infinite count your retries
54. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ ❌ 3
55. jobs can’t be done separate them
56. ✉️✉️✉️✉️✉️✉️ ✉️ ✉️ 100 ❌ 99 DEAD LETTER QUEUE
57. fanout & prioritize scale workers effectively
58. ✉️✉️✉️ ✉️✉️✉️✉️ ⠎ ⠎ ᶐ ᶓ ᾬ ᾬ ᶓ
59. watch your queue monitor and measure what is going on
62. Yay or Nay?
63. Thank You
64. Let’s Talk or just ping me later @codestars or ole@jimdo.com RATE ME https://joind.in/10866
66. continuous reading http://blog.thecodepath.com/2013/01/06/ asynchronous-processing-in-web- applications-part-2-developers-need-to- understand-message-queues/ Mathias Meyer - Message Queues - Railsberry 2013 https://vimeo.com/ 68698333 https://github.com/kr/beanstalkd https://github.com/dp-opensource/ pheanstalk https://en.wikipedia.org/wiki/ Message_queue https://github.com/denniskuczynski/ beanstalkd_view https://github.com/twitter/kestrel https://github.com/kr/beanstalkd/blob/ master/doc/protocol.txt I want to thank my company Jimdo, my girlfriend and all my coworkers for supporting me!
67. Products / Name dropping activemq rabbitmq Ømq (zeromq) zookeeper pubnub ironmq sqs sidekiq resque kue gearman beanstalkd kestrel kafka delayed_job celery qless mqseries nsq jms qpid hornetq swiftmq mqseries stormmq dotnetmq snakemq dropr
68. A N D A LWAY S R E M E M B E R : S AY S 〿