Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Queues and the beanstalkd
Search
Daniel Cousineau
February 06, 2015
Programming
1
630
Queues and the beanstalkd
Given at Sunshine PHP 2015
Daniel Cousineau
February 06, 2015
Tweet
Share
More Decks by Daniel Cousineau
See All by Daniel Cousineau
Time is a Social Construct
dcousineau
1
480
React @ Scale
dcousineau
0
130
Frontend Performance & You
dcousineau
0
220
Feature Flags & You
dcousineau
2
80
React+Redux @ Scale
dcousineau
1
300
Reframing The Problem - DCJS July 2016
dcousineau
0
120
YAFT
dcousineau
2
150
How Not Writing PHP Makes You Better At PHP
dcousineau
0
360
JavaScript for PHP Developers
dcousineau
4
660
Other Decks in Programming
See All in Programming
Amazon ECS とマイクロサービスから考えるシステム構成
hiyanger
2
350
asdf-ecspresso作って 友達が増えた話 / Fujiwara Tech Conference 2025
koluku
0
2.7k
AHC041解説
terryu16
0
560
動作確認やテストで漏れがちな観点3選
starfish719
6
940
Swiftコンパイラ超入門+async関数の仕組み
shiz
0
200
AIの力でお手軽Chrome拡張機能作り
taiseiue
0
150
DMMオンラインサロンアプリのSwift化
hayatan
0
290
chibiccをCILに移植した結果 (NGK2025S版)
kekyo
PRO
0
210
令和7年版 あなたが使ってよいフロントエンド機能とは
mugi_uno
12
6.1k
SwiftUIで単方向アーキテクチャを導入して得られた成果
takuyaosawa
0
230
VitePressを2週間使ってみた感想
hal_spidernight
0
110
ファインディの テックブログ爆誕までの軌跡
starfish719
2
950
Featured
See All Featured
Automating Front-end Workflow
addyosmani
1367
200k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9.3k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
39
1.9k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
A better future with KSS
kneath
238
17k
A Modern Web Designer's Workflow
chriscoyier
693
190k
Java REST API Framework Comparison - PWX 2021
mraible
28
8.4k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
45
2.3k
Speed Design
sergeychernyshev
25
760
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.3k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
6
530
Transcript
QUEUES & THE GIANT BEANSTALKD
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM @dcousineau
@croscon
None
None
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM USE
CASE
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM THEORY
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUE:
The Data Structure
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM FIFO
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Synchronous
Computation N seconds m Seconds n+m seconds
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A
Fetch Z Work Z 5 5 Synchronous Computation
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A
Fetch Z Work Z 5 10 10 Work 5 Synchronous Computation
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Z
15 5 10 A Fetch Z Work 5 10 Work 15 Synchronous Computation
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous
Computation N seconds m Seconds N seconds
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous
Computation Multi-Threading Multi-Process Memory Thread Thread Thread process process process memory Memory Memory
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race
Condition A B Fetch Z Work Z 5 5 Work Fetch Z 5
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race
Condition A B Fetch Z Work Z 5 10 Work 10 Work Fetch Z 5 5
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race
Condition A B Fetch Z Fetch Z Work Z 5 5 Work 10 Work 15 15 5 10
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking
A B Fetch Z Z 5 5 Fetch Z A Work
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking
A B Fetch Z Fetch Z Work Z 5 Work 15 15 15 B 5 A wait wait
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking
A B Fetch Z Fetch Z Work Z 5 Work 15 15 Work 25 25 15 B 5 A wait wait
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead
Locking A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead
Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead
Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z wait wait wait wait wait wait wait wait wait
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUING
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
1 job 2 worker 1 worker 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
1 job 2 job 3 worker 1 worker 2 reserve job 1
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
1 job 2 job 3 worker 1 worker 2 reserve job 1 job 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
2 job 3 worker 1 worker 2 Completed Reserve job 3 job 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
3 worker 1 worker 2 job 3
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue
worker worker worker web app cron daemon db
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
input output
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM do
work update database or save file[s] or new job all data necessary to complete job
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM resize
image make thumbs Original Location thumbnail[s] image file new Name[s]
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM compute
stats account Report account id account data
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM add
to master report account report
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM post
tweet api response twitter id access tokens tweet text metadata
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM update
database api response
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM EXAMPLE
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM http://kr.github.io/beanstalkd/
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://github.com/dcousineau/queues-example
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM TIPS
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Either
Read, or Write, never both
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM If
you write, design for OVERWRITE
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 worker 1 worker 2 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
25 worker 1 worker 2 B: 25 A: 10 B: 20
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2 A: 10 B: 25
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 worker 1 worker 2 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
25 worker 1 worker 2 A: 10 B: 25 B: 20
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2 A: 10 B: 20 B: 25 Completed Reserve B: 25
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2 A: 10 B: 25
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue
worker web app cron db
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM scheduler
Every X minutes, check: “What should be done NOW” worker db web app Assume DB is up to date
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM its
fine to run a process let it die then resurrect it
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM OR:
have long running process listen for jobs fork single-run job process
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Keep
jobs & workers small keep surface area for failures small
THANKS. FOR YOUR ATTENTION DANIEL COUSINEAU // FOLLOW ME :
@DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://joind.in/13443 http://bit.ly/queues-ssp15