Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Queues and the beanstalkd
Search
Sponsored
·
Ship Features Fearlessly
Turn features on and off without deploys. Used by thousands of Ruby developers.
→
Daniel Cousineau
February 06, 2015
Programming
700
1
Share
Queues and the beanstalkd
Given at Sunshine PHP 2015
Daniel Cousineau
February 06, 2015
More Decks by Daniel Cousineau
See All by Daniel Cousineau
Time is a Social Construct
dcousineau
1
660
React @ Scale
dcousineau
0
210
Frontend Performance & You
dcousineau
0
380
Feature Flags & You
dcousineau
2
110
React+Redux @ Scale
dcousineau
1
360
Reframing The Problem - DCJS July 2016
dcousineau
0
150
YAFT
dcousineau
2
160
How Not Writing PHP Makes You Better At PHP
dcousineau
0
400
JavaScript for PHP Developers
dcousineau
4
720
Other Decks in Programming
See All in Programming
Vibe하게 만드는 Flutter GenUI App With ADK , 박제창, BWAI Incheon 2026
itsmedreamwalker
0
550
感情を設計する
ichimichi
5
1.4k
「話せることがない」を乗り越える 〜日常業務から登壇テーマをつくる思考法〜
shoheimitani
4
760
2026-03-27 #terminalnight 変数展開とコマンド展開でターミナル作業をスマートにする方法
masasuzu
0
320
PDI: Como Alavancar Sua Carreira e Seu Negócio
marcelgsantos
0
120
RSAが破られる前に知っておきたい 耐量子計算機暗号(PQC)入門 / Intro to PQC: Preparing for the Post-RSA Era
mackey0225
3
130
Reactive ❤️ Loom: A Forbidden Love Story
franz1981
2
230
Symfonyの特性(設計思想)を手軽に活かす特性(trait)
ickx
0
130
L’IA au service des devs : Anatomie d'un assistant de Code Review
toham
0
230
Nuxt Server Components
wattanx
0
270
PHP で mp3 プレイヤーを実装しよう
m3m0r7
PRO
0
260
CDK Deployのための ”反響定位”
watany
4
710
Featured
See All Featured
[RailsConf 2023] Rails as a piece of cake
palkan
59
6.5k
The Cost Of JavaScript in 2023
addyosmani
55
9.8k
SEO for Brand Visibility & Recognition
aleyda
0
4.5k
HDC tutorial
michielstock
2
620
Why Mistakes Are the Best Teachers: Turning Failure into a Pathway for Growth
auna
0
120
4 Signs Your Business is Dying
shpigford
187
22k
BBQ
matthewcrist
89
10k
Facilitating Awesome Meetings
lara
57
6.8k
16th Malabo Montpellier Forum Presentation
akademiya2063
PRO
0
97
Lessons Learnt from Crawling 1000+ Websites
charlesmeaden
PRO
1
1.2k
Dominate Local Search Results - an insider guide to GBP, reviews, and Local SEO
greggifford
PRO
0
140
The SEO identity crisis: Don't let AI make you average
varn
0
440
Transcript
QUEUES & THE GIANT BEANSTALKD
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM @dcousineau
@croscon
None
None
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM USE
CASE
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM THEORY
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUE:
The Data Structure
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM FIFO
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Synchronous
Computation N seconds m Seconds n+m seconds
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A
Fetch Z Work Z 5 5 Synchronous Computation
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A
Fetch Z Work Z 5 10 10 Work 5 Synchronous Computation
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Z
15 5 10 A Fetch Z Work 5 10 Work 15 Synchronous Computation
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous
Computation N seconds m Seconds N seconds
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous
Computation Multi-Threading Multi-Process Memory Thread Thread Thread process process process memory Memory Memory
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race
Condition A B Fetch Z Work Z 5 5 Work Fetch Z 5
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race
Condition A B Fetch Z Work Z 5 10 Work 10 Work Fetch Z 5 5
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race
Condition A B Fetch Z Fetch Z Work Z 5 5 Work 10 Work 15 15 5 10
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking
A B Fetch Z Z 5 5 Fetch Z A Work
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking
A B Fetch Z Fetch Z Work Z 5 Work 15 15 15 B 5 A wait wait
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking
A B Fetch Z Fetch Z Work Z 5 Work 15 15 Work 25 25 15 B 5 A wait wait
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead
Locking A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead
Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead
Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z wait wait wait wait wait wait wait wait wait
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUING
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
1 job 2 worker 1 worker 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
1 job 2 job 3 worker 1 worker 2 reserve job 1
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
1 job 2 job 3 worker 1 worker 2 reserve job 1 job 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
2 job 3 worker 1 worker 2 Completed Reserve job 3 job 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
3 worker 1 worker 2 job 3
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue
worker worker worker web app cron daemon db
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
input output
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM do
work update database or save file[s] or new job all data necessary to complete job
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM resize
image make thumbs Original Location thumbnail[s] image file new Name[s]
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM compute
stats account Report account id account data
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM add
to master report account report
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM post
tweet api response twitter id access tokens tweet text metadata
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM update
database api response
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM EXAMPLE
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM http://kr.github.io/beanstalkd/
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://github.com/dcousineau/queues-example
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM TIPS
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Either
Read, or Write, never both
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM If
you write, design for OVERWRITE
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 worker 1 worker 2 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
25 worker 1 worker 2 B: 25 A: 10 B: 20
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2 A: 10 B: 25
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 worker 1 worker 2 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
25 worker 1 worker 2 A: 10 B: 25 B: 20
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2 A: 10 B: 20 B: 25 Completed Reserve B: 25
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2 A: 10 B: 25
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue
worker web app cron db
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM scheduler
Every X minutes, check: “What should be done NOW” worker db web app Assume DB is up to date
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM its
fine to run a process let it die then resurrect it
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM OR:
have long running process listen for jobs fork single-run job process
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Keep
jobs & workers small keep surface area for failures small
THANKS. FOR YOUR ATTENTION DANIEL COUSINEAU // FOLLOW ME :
@DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://joind.in/13443 http://bit.ly/queues-ssp15