Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Queues and the beanstalkd

Avatar for Daniel Cousineau Daniel Cousineau
February 06, 2015
Programming
1
650

Queues and the beanstalkd

Given at Sunshine PHP 2015

Avatar for Daniel Cousineau

Daniel Cousineau

February 06, 2015
Tweet

More Decks by Daniel Cousineau

See All by Daniel Cousineau
Time is a Social Construct
Avatar for Daniel Cousineau dcousineau
1
560
React @ Scale
Avatar for Daniel Cousineau dcousineau
0
170
Frontend Performance & You
Avatar for Daniel Cousineau dcousineau
0
280
Feature Flags & You
Avatar for Daniel Cousineau dcousineau
2
96
React+Redux @ Scale
Avatar for Daniel Cousineau dcousineau
1
330
Reframing The Problem - DCJS July 2016
Avatar for Daniel Cousineau dcousineau
0
120
YAFT
Avatar for Daniel Cousineau dcousineau
2
150
How Not Writing PHP Makes You Better At PHP
Avatar for Daniel Cousineau dcousineau
0
380
JavaScript for PHP Developers
Avatar for Daniel Cousineau dcousineau
4
690

Other Decks in Programming

See All in Programming
Vue・React マルチプロダクト開発を支える Vite
Avatar for ANDPAD inc andpad
0
110
CSC305 Summer Lecture 12
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
0
130
1から理解するWeb Push
Avatar for Minoru Takeuchi dora1998
7
1.7k
コンテキストエンジニアリング Cursor編
Avatar for kinopee kinopeee
1
760
「待たせ上手」なスケルトンスクリーン、 そのUXの裏側
Avatar for teamLab teamlab
PRO
0
300
レガシープロジェクトで最大限AIの恩恵を受けられるようClaude Codeを利用する
Avatar for tk1351 tk1351
4
1.6k
print("Hello, World")
Avatar for 高見龍 eddie
1
500
実用的なGOCACHEPROG実装をするために / golang.tokyo #40
Avatar for mazrean mazrean
1
230
OSS開発者という働き方
Avatar for ANDPAD inc andpad
5
1.7k
機能追加とリーダー業務の類似性
Avatar for Rinchoku rinchoku
2
1.1k
もうちょっといいRubyプロファイラを作りたい (2025)
Avatar for osyoyu osyoyu
0
320
Claude Codeで実装以外の開発フロー、どこまで自動化できるか？失敗と成功
Avatar for ndaDayo.Sugawawa ndadayo
4
1.9k

Featured

See All Featured
Typedesign – Prime Four
Avatar for Hannes Fritz hannesfritz
42
2.8k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
Avatar for Tammy Everts tammyeverts
53
2.9k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
Avatar for Eileen M. Uchitelle eileencodes
26
3k
Keith and Marios Guide to Fast Websites
Avatar for Keith Pitt keithpitt
411
22k
Become a Pro
Avatar for Speaker Deck speakerdeck
PRO
29
5.5k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
83
9.2k
Code Reviewing Like a Champion
Avatar for maltzj maltzj
525
40k
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
37
2.8k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
285
13k
StorybookのUI Testing Handbookを読んだ
Avatar for Shingo Yamazaki zakiyama
31
6.1k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
Avatar for Marc Duiker marcduiker
31
2.2k
Optimising Largest Contentful Paint
Avatar for Harry Roberts csswizardry
37
3.4k

Transcript

  1. QUEUES & THE GIANT BEANSTALKD

  2. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM @dcousineau

  3. @croscon

  4. None
  5. None

  6. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM USE

    CASE

  7. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  8. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  9. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  10. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM THEORY

  11. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUE:

    The Data Structure

  12. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM FIFO

  13. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Synchronous

    Computation N seconds m Seconds n+m seconds

  14. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A

    Fetch Z Work Z 5 5 Synchronous Computation

  15. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A

    Fetch Z Work Z 5 10 10 Work 5 Synchronous Computation

  16. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Z

    15 5 10 A Fetch Z Work 5 10 Work 15 Synchronous Computation

  17. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous

    Computation N seconds m Seconds N seconds

  18. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous

    Computation Multi-Threading Multi-Process Memory Thread Thread Thread process process process memory Memory Memory

  19. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race

    Condition A B Fetch Z Work Z 5 5 Work Fetch Z 5

  20. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race

    Condition A B Fetch Z Work Z 5 10 Work 10 Work Fetch Z 5 5

  21. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race

    Condition A B Fetch Z Fetch Z Work Z 5 5 Work 10 Work 15 15 5 10

  22. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking

    A B Fetch Z Z 5 5 Fetch Z A Work

  23. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking

    A B Fetch Z Fetch Z Work Z 5 Work 15 15 15 B 5 A wait wait

  24. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking

    A B Fetch Z Fetch Z Work Z 5 Work 15 15 Work 25 25 15 B 5 A wait wait

  25. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead

    Locking A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10

  26. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead

    Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z

  27. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead

    Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z wait wait wait wait wait wait wait wait wait

  28. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUING

  29. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    1 job 2 worker 1 worker 2

  30. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    1 job 2 job 3 worker 1 worker 2 reserve job 1

  31. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    1 job 2 job 3 worker 1 worker 2 reserve job 1 job 2

  32. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    2 job 3 worker 1 worker 2 Completed Reserve job 3 job 2

  33. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    3 worker 1 worker 2 job 3

  34. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2

  35. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue

    worker worker worker web app cron daemon db

  36. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    input output

  37. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM do

    work update database or save file[s] or new job all data necessary to complete job

  38. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM resize

    image make thumbs Original Location thumbnail[s] image file new Name[s]

  39. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM compute

    stats account Report account id account data

  40. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM add

    to master report account report

  41. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM post

    tweet api response twitter id access tokens tweet text metadata

  42. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM update

    database api response

  43. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM EXAMPLE

  44. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM http://kr.github.io/beanstalkd/

  45. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  46. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  47. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  48. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://github.com/dcousineau/queues-example

  49. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM TIPS

  50. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Either

    Read, or Write, never both

  51. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM If

    you write, design for OVERWRITE

  52. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 worker 1 worker 2 A: ? B: ?

  53. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?

  54. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?

  55. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?

  56. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    25 worker 1 worker 2 B: 25 A: 10 B: 20

  57. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2 A: 10 B: 25

  58. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 worker 1 worker 2 A: ? B: ?

  59. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?

  60. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?

  61. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?

  62. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    25 worker 1 worker 2 A: 10 B: 25 B: 20

  63. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2 A: 10 B: 20 B: 25 Completed Reserve B: 25

  64. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2 A: 10 B: 25

  65. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue

    worker web app cron db

  66. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM scheduler

    Every X minutes, check: “What should be done NOW” worker db web app Assume DB is up to date

  67. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM its

    fine to run a process let it die then resurrect it

  68. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM OR:

    have long running process listen for jobs fork single-run job process

  69. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Keep

    jobs & workers small keep surface area for failures small

  70. THANKS. FOR YOUR ATTENTION DANIEL COUSINEAU // FOLLOW ME :

    @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://joind.in/13443 http://bit.ly/queues-ssp15