Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Queues and the beanstalkd

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Daniel Cousineau Daniel Cousineau
February 06, 2015
Programming
730
1

Queues and the beanstalkd

Given at Sunshine PHP 2015

Avatar for Daniel Cousineau

Daniel Cousineau

February 06, 2015

More Decks by Daniel Cousineau

See All by Daniel Cousineau
Time is a Social Construct
Avatar for Daniel Cousineau dcousineau
1
700
React @ Scale
Avatar for Daniel Cousineau dcousineau
0
250
Frontend Performance & You
Avatar for Daniel Cousineau dcousineau
0
400
Feature Flags & You
Avatar for Daniel Cousineau dcousineau
2
130
React+Redux @ Scale
Avatar for Daniel Cousineau dcousineau
1
370
Reframing The Problem - DCJS July 2016
Avatar for Daniel Cousineau dcousineau
0
160
YAFT
Avatar for Daniel Cousineau dcousineau
2
170
How Not Writing PHP Makes You Better At PHP
Avatar for Daniel Cousineau dcousineau
0
420
JavaScript for PHP Developers
Avatar for Daniel Cousineau dcousineau
4
750

Other Decks in Programming

See All in Programming
速く作れる。その次は、速く確かめられる開発へ 〜AIネイティブ開発を支える、Shift Down〜 / Can build fast. Next, moving to development where we can verify fast.
Avatar for r-kagaya rkaga
8
5.3k
mrbgem 三角測量 開発
Avatar for ogom ogom
0
160
iOS 27でニュースアプリはどう変わる！？ 〜日経電子版の新機能対応と、開発事例から〜
Avatar for Kazuki Nakashima lynnswap
7
13k
Herb in Rails 8.2: Your ERB views, now HTML-aware @ Rails World 2026, Austin, Texas
Avatar for Marco Roth marcoroth
0
140
動作中のプログラムの中身をリアルタイムに覗く / Realtime Debugger for CSharp with Roslyn
Avatar for Prota prota
1
1.7k
Heart of Swift Concurrency
Avatar for Yuta Koshizawa koher
0
1.1k
Androidだけじゃない、DroidKaigiで広がった私の技術スタック
Avatar for Akio Itaya akkeylab
0
110
AgentCore CLI で進化した AWS での AI エージェントの作り方 : 必要な機能を必要な時に
Avatar for Takahiro Kubo icoxfog417
PRO
4
400
Augmenting AI with the Power of Jakarta EE
Avatar for ivargrimstad ivargrimstad
0
380
テストを司るデーモンに会いに行く 〜隔離した仮想マシンでテストを通すまで〜
Avatar for Hidemune Takahashi h1d3mun3
1
600
UPDATE をやめる — EF Core でマスタをバージョン管理する
Avatar for HideyukiKitao panda728
PRO
0
1k
設計次第でAIコードの読む量は減らせる / designing-for-code-reading
Avatar for MinoDriven minodriven
30
13k

Featured

See All Featured
CSS Pre-Processors: Stylus, Less & Sass
Avatar for Bermon Painter bermonpainter
360
31k
Skip the Path - Find Your Career Trail
Avatar for Mark Kilby mkilby
1
240
Optimising Largest Contentful Paint
Avatar for Harry Roberts csswizardry
37
4k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
Avatar for Kevin Liebholz kevinliebholz
37
6.6k
Redefining SEO in the New Era of Traffic Generation
Avatar for Szymon Słowik szymonslowik
1
450
The innovator’s Mindset - 
Leading Through an Era of Exponential Change - McGill University 2025
Avatar for Jean-Marc De Jonghe jdejongh
PRO
1
350
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
Avatar for Joe Casabona jcasabona
35
2.9k
WENDY [Excerpt]
Avatar for Tessa Abrams tessaabrams
14
40k
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
348
41k
So, you think you're a good person
Avatar for Per Axbom axbom
PRO
2
2.2k
Art, The Web, and Tiny UX
Avatar for Lynn Fisher lynnandtonic
304
22k
Building an army of robots
Avatar for Kyle Neath kneath
307
47k

Transcript

  1. QUEUES & THE GIANT BEANSTALKD

  2. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM @dcousineau

  3. @croscon

  4. None
  5. None

  6. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM USE

    CASE

  7. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  8. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  9. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  10. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM THEORY

  11. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUE:

    The Data Structure

  12. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM FIFO

  13. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Synchronous

    Computation N seconds m Seconds n+m seconds

  14. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A

    Fetch Z Work Z 5 5 Synchronous Computation

  15. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A

    Fetch Z Work Z 5 10 10 Work 5 Synchronous Computation

  16. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Z

    15 5 10 A Fetch Z Work 5 10 Work 15 Synchronous Computation

  17. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous

    Computation N seconds m Seconds N seconds

  18. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous

    Computation Multi-Threading Multi-Process Memory Thread Thread Thread process process process memory Memory Memory

  19. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race

    Condition A B Fetch Z Work Z 5 5 Work Fetch Z 5

  20. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race

    Condition A B Fetch Z Work Z 5 10 Work 10 Work Fetch Z 5 5

  21. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race

    Condition A B Fetch Z Fetch Z Work Z 5 5 Work 10 Work 15 15 5 10

  22. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking

    A B Fetch Z Z 5 5 Fetch Z A Work

  23. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking

    A B Fetch Z Fetch Z Work Z 5 Work 15 15 15 B 5 A wait wait

  24. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking

    A B Fetch Z Fetch Z Work Z 5 Work 15 15 Work 25 25 15 B 5 A wait wait

  25. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead

    Locking A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10

  26. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead

    Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z

  27. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead

    Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z wait wait wait wait wait wait wait wait wait

  28. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUING

  29. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    1 job 2 worker 1 worker 2

  30. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    1 job 2 job 3 worker 1 worker 2 reserve job 1

  31. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    1 job 2 job 3 worker 1 worker 2 reserve job 1 job 2

  32. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    2 job 3 worker 1 worker 2 Completed Reserve job 3 job 2

  33. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    3 worker 1 worker 2 job 3

  34. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2

  35. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue

    worker worker worker web app cron daemon db

  36. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    input output

  37. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM do

    work update database or save file[s] or new job all data necessary to complete job

  38. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM resize

    image make thumbs Original Location thumbnail[s] image file new Name[s]

  39. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM compute

    stats account Report account id account data

  40. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM add

    to master report account report

  41. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM post

    tweet api response twitter id access tokens tweet text metadata

  42. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM update

    database api response

  43. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM EXAMPLE

  44. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM http://kr.github.io/beanstalkd/

  45. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  46. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  47. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  48. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://github.com/dcousineau/queues-example

  49. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM TIPS

  50. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Either

    Read, or Write, never both

  51. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM If

    you write, design for OVERWRITE

  52. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 worker 1 worker 2 A: ? B: ?

  53. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?

  54. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?

  55. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?

  56. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    25 worker 1 worker 2 B: 25 A: 10 B: 20

  57. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2 A: 10 B: 25

  58. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 worker 1 worker 2 A: ? B: ?

  59. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?

  60. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?

  61. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?

  62. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    25 worker 1 worker 2 A: 10 B: 25 B: 20

  63. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2 A: 10 B: 20 B: 25 Completed Reserve B: 25

  64. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2 A: 10 B: 25

  65. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue

    worker web app cron db

  66. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM scheduler

    Every X minutes, check: “What should be done NOW” worker db web app Assume DB is up to date

  67. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM its

    fine to run a process let it die then resurrect it

  68. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM OR:

    have long running process listen for jobs fork single-run job process

  69. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Keep

    jobs & workers small keep surface area for failures small

  70. THANKS. FOR YOUR ATTENTION DANIEL COUSINEAU // FOLLOW ME :

    @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://joind.in/13443 http://bit.ly/queues-ssp15