Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Queues and the beanstalkd

Daniel Cousineau
February 06, 2015
Programming
1
580

Queues and the beanstalkd

Given at Sunshine PHP 2015

Daniel Cousineau

February 06, 2015
Tweet

More Decks by Daniel Cousineau

See All by Daniel Cousineau
Time is a Social Construct
dcousineau
1
410
React @ Scale
dcousineau
0
100
Frontend Performance & You
dcousineau
0
170
Feature Flags & You
dcousineau
2
77
React+Redux @ Scale
dcousineau
1
300
Reframing The Problem - DCJS July 2016
dcousineau
0
110
YAFT
dcousineau
2
140
How Not Writing PHP Makes You Better At PHP
dcousineau
0
350
JavaScript for PHP Developers
dcousineau
4
610

Other Decks in Programming

See All in Programming
社内 LT 会を発足し、アウトプット文化を醸成させるために考えたこと・やったこと / Starting internal LT meetings and fostering an output culture
mackey0225
3
120
20240706_CDKConf
takuyay0ne
0
1.2k
Architectures with Lightweight Stores: New Rules and Options
manfredsteyer
PRO
0
100
CSC307 Lecture 09
javiergs
PRO
1
500
Javaの現状2024夏 / Java current status 2024 summer
kishida
4
1.4k
APIのない大学ログインWebサービスをWKWebViewとJavaScriptでアプリ化した話
akidon0000
1
330
Android開発者のための Kotlin Multiplatform入門
ntaro
0
190
SRE チーム立ち上げ前に考えたこと・取り組んだこと / Considerations and Preparations Before Establishing an SRE Team
mackey0225
3
320
みんなのオブザーバビリティプラットフォームを作ってるんだがパフォーマンスがやばい #mackerelio #srenext
ne_sachirou
0
370
Advanced App Shrinking Techniques
cbeyls
2
150
さきがけから振り返るアーキテクチャ刷新 / Reflecting on the Architectural Renewal from the Vanguard
nrslib
2
770
Folding Cheat Sheet #7
philipschwarz
PRO
0
150

Featured

See All Featured
Fireside Chat
paigeccino
25
2.8k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
224
21k
GitHub's CSS Performance
jonrohan
1026
450k
Done Done
chrislema
179
15k
A Philosophy of Restraint
colly
200
16k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
16
1.6k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
20
7.2k
Embracing the Ebb and Flow
colly
81
4.3k
The Mythical Team-Month
searls
217
43k
Testing 201, or: Great Expectations
jmmastey
33
6.9k
Robots, Beer and Maslow
schacon
PRO
157
8.1k
The Cult of Friendly URLs
andyhume
75
5.9k

Transcript

  1. QUEUES & THE GIANT BEANSTALKD

  2. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM @dcousineau

  3. @croscon

  4. None
  5. None

  6. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM USE

    CASE

  7. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  8. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  9. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  10. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM THEORY

  11. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUE:

    The Data Structure

  12. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM FIFO

  13. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Synchronous

    Computation N seconds m Seconds n+m seconds

  14. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A

    Fetch Z Work Z 5 5 Synchronous Computation

  15. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A

    Fetch Z Work Z 5 10 10 Work 5 Synchronous Computation

  16. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Z

    15 5 10 A Fetch Z Work 5 10 Work 15 Synchronous Computation

  17. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous

    Computation N seconds m Seconds N seconds

  18. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous

    Computation Multi-Threading Multi-Process Memory Thread Thread Thread process process process memory Memory Memory

  19. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race

    Condition A B Fetch Z Work Z 5 5 Work Fetch Z 5

  20. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race

    Condition A B Fetch Z Work Z 5 10 Work 10 Work Fetch Z 5 5

  21. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race

    Condition A B Fetch Z Fetch Z Work Z 5 5 Work 10 Work 15 15 5 10

  22. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking

    A B Fetch Z Z 5 5 Fetch Z A Work

  23. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking

    A B Fetch Z Fetch Z Work Z 5 Work 15 15 15 B 5 A wait wait

  24. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking

    A B Fetch Z Fetch Z Work Z 5 Work 15 15 Work 25 25 15 B 5 A wait wait

  25. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead

    Locking A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10

  26. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead

    Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z

  27. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead

    Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z wait wait wait wait wait wait wait wait wait

  28. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUING

  29. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    1 job 2 worker 1 worker 2

  30. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    1 job 2 job 3 worker 1 worker 2 reserve job 1

  31. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    1 job 2 job 3 worker 1 worker 2 reserve job 1 job 2

  32. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    2 job 3 worker 1 worker 2 Completed Reserve job 3 job 2

  33. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    3 worker 1 worker 2 job 3

  34. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2

  35. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue

    worker worker worker web app cron daemon db

  36. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    input output

  37. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM do

    work update database or save file[s] or new job all data necessary to complete job

  38. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM resize

    image make thumbs Original Location thumbnail[s] image file new Name[s]

  39. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM compute

    stats account Report account id account data

  40. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM add

    to master report account report

  41. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM post

    tweet api response twitter id access tokens tweet text metadata

  42. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM update

    database api response

  43. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM EXAMPLE

  44. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM http://kr.github.io/beanstalkd/

  45. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  46. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  47. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  48. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://github.com/dcousineau/queues-example

  49. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM TIPS

  50. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Either

    Read, or Write, never both

  51. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM If

    you write, design for OVERWRITE

  52. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 worker 1 worker 2 A: ? B: ?

  53. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?

  54. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?

  55. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?

  56. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    25 worker 1 worker 2 B: 25 A: 10 B: 20

  57. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2 A: 10 B: 25

  58. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 worker 1 worker 2 A: ? B: ?

  59. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?

  60. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?

  61. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?

  62. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    25 worker 1 worker 2 A: 10 B: 25 B: 20

  63. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2 A: 10 B: 20 B: 25 Completed Reserve B: 25

  64. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2 A: 10 B: 25

  65. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue

    worker web app cron db

  66. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM scheduler

    Every X minutes, check: “What should be done NOW” worker db web app Assume DB is up to date

  67. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM its

    fine to run a process let it die then resurrect it

  68. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM OR:

    have long running process listen for jobs fork single-run job process

  69. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Keep

    jobs & workers small keep surface area for failures small

  70. THANKS. FOR YOUR ATTENTION DANIEL COUSINEAU // FOLLOW ME :

    @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://joind.in/13443 http://bit.ly/queues-ssp15