Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Queues and the beanstalkd

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Daniel Cousineau Daniel Cousineau
February 06, 2015
Programming
1
690

Queues and the beanstalkd

Given at Sunshine PHP 2015

Avatar for Daniel Cousineau

Daniel Cousineau

February 06, 2015
Tweet

More Decks by Daniel Cousineau

See All by Daniel Cousineau
Time is a Social Construct
Avatar for Daniel Cousineau dcousineau
1
640
React @ Scale
Avatar for Daniel Cousineau dcousineau
0
200
Frontend Performance & You
Avatar for Daniel Cousineau dcousineau
0
310
Feature Flags & You
Avatar for Daniel Cousineau dcousineau
2
110
React+Redux @ Scale
Avatar for Daniel Cousineau dcousineau
1
340
Reframing The Problem - DCJS July 2016
Avatar for Daniel Cousineau dcousineau
0
140
YAFT
Avatar for Daniel Cousineau dcousineau
2
150
How Not Writing PHP Makes You Better At PHP
Avatar for Daniel Cousineau dcousineau
0
390
JavaScript for PHP Developers
Avatar for Daniel Cousineau dcousineau
4
710

Other Decks in Programming

See All in Programming
KIKI_MBSD Cybersecurity Challenges 2025
Avatar for ikema ikema
0
1.3k
The Past, Present, and Future of Enterprise Java
Avatar for ivargrimstad ivargrimstad
0
590
AWS re:Invent 2025参加 直前 Seattle-Tacoma Airport(SEA)におけるハードウェア紛失インシデントLT
Avatar for tetutetu214 tetutetu214
2
120
AI時代のキャリアプラン「技術の引力」からの脱出と「問い」へのいざない / tech-gravity
Avatar for MinoDriven minodriven
21
7.3k
Basic Architectures
Avatar for Denys Poltorak denyspoltorak
0
680
AI巻き込み型コードレビューのススメ
Avatar for Nealle nealle
2
420
今から始めるClaude Code超入門
Avatar for 448jp | OKI Yoshiya 448jp
8
8.9k
AIで開発はどれくらい加速したのか？AIエージェントによるコード生成を、現場の評価と研究開発の評価の両面からdeep diveしてみる
Avatar for どすこい daisuketakeda
1
2.5k
組織で育むオブザーバビリティ
Avatar for ryotaro kobayashi ryota_hnk
0
180
dchart: charts from deck markup
Avatar for Anthony Starks ajstarks
3
1k
ノイジーネイバー問題を解決する 公平なキューイング
Avatar for Shoichi Ochi occhi
0
110
「ブロックテーマでは再現できない」は本当か？
Avatar for Takashi Kitajima inc2734
0
1k

Featured

See All Featured
Applied NLP in the Age of Generative AI
Avatar for Ines Montani inesmontani
PRO
4
2k
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
39
3k
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
58k
Joys of Absence: A Defence of Solitary Play
Avatar for Sebastian Deterding codingconduct
1
290
The AI Revolution Will Not Be Monopolized: How open-source beats economies of scale, even for LLMs
Avatar for Ines Montani inesmontani
PRO
3
3k
How to Get Subject Matter Experts Bought In and Actively Contributing to SEO & PR Initiatives.
Avatar for Liv Day livdayseo
0
67
Making the Leap to Tech Lead
Avatar for Ryan Cromwell cromwellryan
135
9.7k
Stewardship and Sustainability of Urban and Community Forests
Avatar for Eric Wiseman pwiseman
0
110
Embracing the Ebb and Flow
Avatar for Simon Collison colly
88
5k
Google's AI Overviews - The New Search
Avatar for Barry Adams badams
0
910
Building an army of robots
Avatar for Kyle Neath kneath
306
46k
Stop Working from a Prison Cell
Avatar for Chris Michel hatefulcrawdad
273
21k

Transcript

  1. QUEUES & THE GIANT BEANSTALKD

  2. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM @dcousineau

  3. @croscon

  4. None
  5. None

  6. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM USE

    CASE

  7. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  8. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  9. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  10. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM THEORY

  11. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUE:

    The Data Structure

  12. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM FIFO

  13. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Synchronous

    Computation N seconds m Seconds n+m seconds

  14. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A

    Fetch Z Work Z 5 5 Synchronous Computation

  15. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A

    Fetch Z Work Z 5 10 10 Work 5 Synchronous Computation

  16. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Z

    15 5 10 A Fetch Z Work 5 10 Work 15 Synchronous Computation

  17. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous

    Computation N seconds m Seconds N seconds

  18. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous

    Computation Multi-Threading Multi-Process Memory Thread Thread Thread process process process memory Memory Memory

  19. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race

    Condition A B Fetch Z Work Z 5 5 Work Fetch Z 5

  20. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race

    Condition A B Fetch Z Work Z 5 10 Work 10 Work Fetch Z 5 5

  21. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race

    Condition A B Fetch Z Fetch Z Work Z 5 5 Work 10 Work 15 15 5 10

  22. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking

    A B Fetch Z Z 5 5 Fetch Z A Work

  23. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking

    A B Fetch Z Fetch Z Work Z 5 Work 15 15 15 B 5 A wait wait

  24. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking

    A B Fetch Z Fetch Z Work Z 5 Work 15 15 Work 25 25 15 B 5 A wait wait

  25. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead

    Locking A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10

  26. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead

    Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z

  27. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead

    Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z wait wait wait wait wait wait wait wait wait

  28. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUING

  29. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    1 job 2 worker 1 worker 2

  30. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    1 job 2 job 3 worker 1 worker 2 reserve job 1

  31. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    1 job 2 job 3 worker 1 worker 2 reserve job 1 job 2

  32. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    2 job 3 worker 1 worker 2 Completed Reserve job 3 job 2

  33. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    3 worker 1 worker 2 job 3

  34. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2

  35. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue

    worker worker worker web app cron daemon db

  36. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    input output

  37. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM do

    work update database or save file[s] or new job all data necessary to complete job

  38. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM resize

    image make thumbs Original Location thumbnail[s] image file new Name[s]

  39. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM compute

    stats account Report account id account data

  40. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM add

    to master report account report

  41. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM post

    tweet api response twitter id access tokens tweet text metadata

  42. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM update

    database api response

  43. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM EXAMPLE

  44. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM http://kr.github.io/beanstalkd/

  45. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  46. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  47. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  48. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://github.com/dcousineau/queues-example

  49. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM TIPS

  50. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Either

    Read, or Write, never both

  51. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM If

    you write, design for OVERWRITE

  52. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 worker 1 worker 2 A: ? B: ?

  53. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?

  54. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?

  55. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?

  56. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    25 worker 1 worker 2 B: 25 A: 10 B: 20

  57. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2 A: 10 B: 25

  58. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 worker 1 worker 2 A: ? B: ?

  59. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?

  60. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?

  61. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?

  62. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    25 worker 1 worker 2 A: 10 B: 25 B: 20

  63. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2 A: 10 B: 20 B: 25 Completed Reserve B: 25

  64. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2 A: 10 B: 25

  65. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue

    worker web app cron db

  66. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM scheduler

    Every X minutes, check: “What should be done NOW” worker db web app Assume DB is up to date

  67. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM its

    fine to run a process let it die then resurrect it

  68. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM OR:

    have long running process listen for jobs fork single-run job process

  69. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Keep

    jobs & workers small keep surface area for failures small

  70. THANKS. FOR YOUR ATTENTION DANIEL COUSINEAU // FOLLOW ME :

    @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://joind.in/13443 http://bit.ly/queues-ssp15