Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Queues and the beanstalkd
Search
Daniel Cousineau
February 06, 2015
Programming
1
640
Queues and the beanstalkd
Given at Sunshine PHP 2015
Daniel Cousineau
February 06, 2015
Tweet
Share
More Decks by Daniel Cousineau
See All by Daniel Cousineau
Time is a Social Construct
dcousineau
1
530
React @ Scale
dcousineau
0
160
Frontend Performance & You
dcousineau
0
260
Feature Flags & You
dcousineau
2
89
React+Redux @ Scale
dcousineau
1
310
Reframing The Problem - DCJS July 2016
dcousineau
0
120
YAFT
dcousineau
2
150
How Not Writing PHP Makes You Better At PHP
dcousineau
0
370
JavaScript for PHP Developers
dcousineau
4
680
Other Decks in Programming
See All in Programming
TypeScript を活かしてデザインシステム MCP を作る / #tskaigi_after_night
izumin5210
5
520
カクヨムAndroidアプリのリブート
numeroanddev
0
380
RubyKaigi Hack Space in Tokyo & 函館最速 "予習" 会 / RubyKaigi Hack Space in Tokyo & The Fastest Briefing of RubyKaigi 2026 in Hakodate
moznion
1
130
ReadMoreTextView
fornewid
0
220
ワイがおすすめする新潟の食 / 20250530phpconf-niigata-eve
kasacchiful
0
290
機械学習って何? 5分で解説頑張ってみる
kuroneko2828
0
190
Rails産でないDBを Railsに引っ越すHACK - Omotesando.rb #110
lnit
1
160
「ElixirでIoT!!」のこれまでとこれから
takasehideki
0
320
FastMCPでMCPサーバー/クライアントを構築してみる
ttnyt8701
2
130
事業戦略を理解してソフトウェアを設計する
masuda220
PRO
21
5.7k
Spring gRPC で始める gRPC 入門 / Introduction to gRPC with Spring gRPC
mackey0225
2
460
iOSアプリ開発で 関数型プログラミングを実現する The Composable Architectureの紹介
yimajo
2
190
Featured
See All Featured
Building a Scalable Design System with Sketch
lauravandoore
462
33k
The Straight Up "How To Draw Better" Workshop
denniskardys
233
140k
The Language of Interfaces
destraynor
158
25k
Automating Front-end Workflow
addyosmani
1370
200k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
123
52k
Facilitating Awesome Meetings
lara
54
6.4k
Reflections from 52 weeks, 52 projects
jeffersonlam
349
20k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2.1k
For a Future-Friendly Web
brad_frost
179
9.8k
Code Reviewing Like a Champion
maltzj
524
40k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Transcript
QUEUES & THE GIANT BEANSTALKD
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM @dcousineau
@croscon
None
None
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM USE
CASE
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM THEORY
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUE:
The Data Structure
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM FIFO
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Synchronous
Computation N seconds m Seconds n+m seconds
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A
Fetch Z Work Z 5 5 Synchronous Computation
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A
Fetch Z Work Z 5 10 10 Work 5 Synchronous Computation
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Z
15 5 10 A Fetch Z Work 5 10 Work 15 Synchronous Computation
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous
Computation N seconds m Seconds N seconds
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous
Computation Multi-Threading Multi-Process Memory Thread Thread Thread process process process memory Memory Memory
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race
Condition A B Fetch Z Work Z 5 5 Work Fetch Z 5
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race
Condition A B Fetch Z Work Z 5 10 Work 10 Work Fetch Z 5 5
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race
Condition A B Fetch Z Fetch Z Work Z 5 5 Work 10 Work 15 15 5 10
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking
A B Fetch Z Z 5 5 Fetch Z A Work
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking
A B Fetch Z Fetch Z Work Z 5 Work 15 15 15 B 5 A wait wait
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking
A B Fetch Z Fetch Z Work Z 5 Work 15 15 Work 25 25 15 B 5 A wait wait
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead
Locking A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead
Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead
Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z wait wait wait wait wait wait wait wait wait
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUING
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
1 job 2 worker 1 worker 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
1 job 2 job 3 worker 1 worker 2 reserve job 1
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
1 job 2 job 3 worker 1 worker 2 reserve job 1 job 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
2 job 3 worker 1 worker 2 Completed Reserve job 3 job 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
3 worker 1 worker 2 job 3
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue
worker worker worker web app cron daemon db
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
input output
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM do
work update database or save file[s] or new job all data necessary to complete job
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM resize
image make thumbs Original Location thumbnail[s] image file new Name[s]
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM compute
stats account Report account id account data
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM add
to master report account report
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM post
tweet api response twitter id access tokens tweet text metadata
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM update
database api response
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM EXAMPLE
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM http://kr.github.io/beanstalkd/
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://github.com/dcousineau/queues-example
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM TIPS
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Either
Read, or Write, never both
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM If
you write, design for OVERWRITE
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 worker 1 worker 2 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
25 worker 1 worker 2 B: 25 A: 10 B: 20
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2 A: 10 B: 25
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 worker 1 worker 2 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
25 worker 1 worker 2 A: 10 B: 25 B: 20
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2 A: 10 B: 20 B: 25 Completed Reserve B: 25
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2 A: 10 B: 25
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue
worker web app cron db
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM scheduler
Every X minutes, check: “What should be done NOW” worker db web app Assume DB is up to date
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM its
fine to run a process let it die then resurrect it
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM OR:
have long running process listen for jobs fork single-run job process
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Keep
jobs & workers small keep surface area for failures small
THANKS. FOR YOUR ATTENTION DANIEL COUSINEAU // FOLLOW ME :
@DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://joind.in/13443 http://bit.ly/queues-ssp15