Save 37% off PRO during our Black Friday Sale! »

Queues and the beanstalkd

6696617169722009ed1ec8c52496c6da?s=47 Daniel Cousineau
February 06, 2015
Programming
1
420

Queues and the beanstalkd

Given at Sunshine PHP 2015

6696617169722009ed1ec8c52496c6da?s=128

Daniel Cousineau

February 06, 2015
Tweet

More Decks by Daniel Cousineau

See All by Daniel Cousineau
6696617169722009ed1ec8c52496c6da?s=48 dcousineau
1
190
6696617169722009ed1ec8c52496c6da?s=48 dcousineau
0
67
6696617169722009ed1ec8c52496c6da?s=48 dcousineau
0
130
6696617169722009ed1ec8c52496c6da?s=48 dcousineau
2
64
6696617169722009ed1ec8c52496c6da?s=48 dcousineau
1
280
6696617169722009ed1ec8c52496c6da?s=48 dcousineau
0
85
6696617169722009ed1ec8c52496c6da?s=48 dcousineau
2
93
6696617169722009ed1ec8c52496c6da?s=48 dcousineau
0
310
6696617169722009ed1ec8c52496c6da?s=48 dcousineau
4
360

Other Decks in Programming

See All in Programming
198d69383210fbec8c6bea4a35863c9d?s=48 tippy
0
2.3k
6b8e2101579190ad96e747e01c279898?s=48 pauloxnet
1
140
52711e2157a6fed933b0361cc06a6953?s=48 marcelgsantos
3
210
87e6be3b5dcac822e5dfd974ba65045a?s=48 sammy7th
0
230
Ec2fcdc4ea7905b289967a2c4c43e154?s=48 sgeengineer
0
200
52a2813503456def3384b696da178901?s=48 nomadmonad
0
380
9e64d2069945742c818421e884603b44?s=48 ushisantoasobu
1
450
C832fe4470555df3cf7ac6fa2f389261?s=48 tak_iwamoto
0
850
8f84b7d8869ef6005d89b378e8661f7c?s=48 masuda220
PRO
9
4.5k
A10e41b0a61d59f2258d7f6172c33479?s=48 kaityo256
2
110
5059d3f370ad7e1f4d8de4be79ae1a2c?s=48 rvirus0817
0
180
Cf642b7a99e07f86e6e7ec7a3bb94d87?s=48 uemura
0
110

Featured

See All Featured
A60068bce2e73de3a37ca9d2dbe36092?s=48 bryan
105
12k
1177e050db6bafe62885362edf6e3537?s=48 lauravandoore
18
2.5k
5b9fe87ec1faa67a4599782930f45ec9?s=48 sstephenson
149
12k
E195ae45320d9202eaa01c9f1d31a416?s=48 marktimemedia
15
750
9952dfcd5d338f8a8e7175c8a8f65fb5?s=48 wjessup
342
16k
A9a491b0fcbe0fbce3d64063a37add99?s=48 rmw
17
1.2k
E190023b66e2b8aa73a842b106920c93?s=48 zenorocha
301
40k
462dad0dd24c568a32dcdb0ae895ae1b?s=48 rasmusluckow
318
19k
9128d500301ae51524e887bb680f471d?s=48 caitiem20
314
18k
766b9746b59e6f09e280cd33cf4ed419?s=48 skipperchong
14
1.1k
Bfd6b681ec6e8c9bef82ff0521c364f7?s=48 kastner
56
2.1k
Fde6c3a50ca518a909ef35c22c0ee0e4?s=48 destraynor
224
48k

Transcript

  1. QUEUES & THE GIANT BEANSTALKD

  2. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM @dcousineau

  3. @croscon

  4. None
  5. None

  6. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM USE

    CASE

  7. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  8. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  9. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  10. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM THEORY

  11. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUE:

    The Data Structure

  12. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM FIFO

  13. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Synchronous

    Computation N seconds m Seconds n+m seconds

  14. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A

    Fetch Z Work Z 5 5 Synchronous Computation

  15. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A

    Fetch Z Work Z 5 10 10 Work 5 Synchronous Computation

  16. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Z

    15 5 10 A Fetch Z Work 5 10 Work 15 Synchronous Computation

  17. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous

    Computation N seconds m Seconds N seconds

  18. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous

    Computation Multi-Threading Multi-Process Memory Thread Thread Thread process process process memory Memory Memory

  19. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race

    Condition A B Fetch Z Work Z 5 5 Work Fetch Z 5

  20. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race

    Condition A B Fetch Z Work Z 5 10 Work 10 Work Fetch Z 5 5

  21. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race

    Condition A B Fetch Z Fetch Z Work Z 5 5 Work 10 Work 15 15 5 10

  22. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking

    A B Fetch Z Z 5 5 Fetch Z A Work

  23. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking

    A B Fetch Z Fetch Z Work Z 5 Work 15 15 15 B 5 A wait wait

  24. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking

    A B Fetch Z Fetch Z Work Z 5 Work 15 15 Work 25 25 15 B 5 A wait wait

  25. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead

    Locking A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10

  26. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead

    Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z

  27. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead

    Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z wait wait wait wait wait wait wait wait wait

  28. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUING

  29. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    1 job 2 worker 1 worker 2

  30. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    1 job 2 job 3 worker 1 worker 2 reserve job 1

  31. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    1 job 2 job 3 worker 1 worker 2 reserve job 1 job 2

  32. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    2 job 3 worker 1 worker 2 Completed Reserve job 3 job 2

  33. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job

    3 worker 1 worker 2 job 3

  34. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2

  35. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue

    worker worker worker web app cron daemon db

  36. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    input output

  37. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM do

    work update database or save file[s] or new job all data necessary to complete job

  38. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM resize

    image make thumbs Original Location thumbnail[s] image file new Name[s]

  39. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM compute

    stats account Report account id account data

  40. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM add

    to master report account report

  41. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM post

    tweet api response twitter id access tokens tweet text metadata

  42. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM update

    database api response

  43. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM EXAMPLE

  44. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM http://kr.github.io/beanstalkd/

  45. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  46. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  47. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

  48. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://github.com/dcousineau/queues-example

  49. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM TIPS

  50. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Either

    Read, or Write, never both

  51. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM If

    you write, design for OVERWRITE

  52. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 worker 1 worker 2 A: ? B: ?

  53. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?

  54. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?

  55. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?

  56. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    25 worker 1 worker 2 B: 25 A: 10 B: 20

  57. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2 A: 10 B: 25

  58. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 worker 1 worker 2 A: ? B: ?

  59. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?

  60. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:

    10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?

  61. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?

  62. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:

    25 worker 1 worker 2 A: 10 B: 25 B: 20

  63. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2 A: 10 B: 20 B: 25 Completed Reserve B: 25

  64. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker

    1 worker 2 A: 10 B: 25

  65. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue

    worker web app cron db

  66. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM scheduler

    Every X minutes, check: “What should be done NOW” worker db web app Assume DB is up to date

  67. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM its

    fine to run a process let it die then resurrect it

  68. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM OR:

    have long running process listen for jobs fork single-run job process

  69. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Keep

    jobs & workers small keep surface area for failures small

  70. THANKS. FOR YOUR ATTENTION DANIEL COUSINEAU // FOLLOW ME :

    @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://joind.in/13443 http://bit.ly/queues-ssp15