Queues and the beanstalkd

Given at Sunshine PHP 2015

Transcript

1. QUEUES & THE GIANT BEANSTALKD
2. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM @dcousineau
3. @croscon
6. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM USE CASE
7. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
8. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
9. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
10. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM THEORY
11. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUE: The Data Structure
12. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM FIFO
13. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Synchronous Computation N seconds m Seconds n+m seconds
14. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A Fetch Z Work Z 5 5 Synchronous Computation
15. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A Fetch Z Work Z 5 10 10 Work 5 Synchronous Computation
16. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Z 15 5 10 A Fetch Z Work 5 10 Work 15 Synchronous Computation
17. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous Computation N seconds m Seconds N seconds
18. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous Computation Multi-Threading Multi-Process Memory Thread Thread Thread process process process memory Memory Memory
19. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race Condition A B Fetch Z Work Z 5 5 Work Fetch Z 5
20. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race Condition A B Fetch Z Work Z 5 10 Work 10 Work Fetch Z 5 5
21. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race Condition A B Fetch Z Fetch Z Work Z 5 5 Work 10 Work 15 15 5 10
22. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking A B Fetch Z Z 5 5 Fetch Z A Work
23. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking A B Fetch Z Fetch Z Work Z 5 Work 15 15 15 B 5 A wait wait
24. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking A B Fetch Z Fetch Z Work Z 5 Work 15 15 Work 25 25 15 B 5 A wait wait
25. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead Locking A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10
26. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z
27. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z wait wait wait wait wait wait wait wait wait
28. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUING
29. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job 1 job 2 worker 1 worker 2
30. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job 1 job 2 job 3 worker 1 worker 2 reserve job 1
31. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job 1 job 2 job 3 worker 1 worker 2 reserve job 1 job 2
32. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job 2 job 3 worker 1 worker 2 Completed Reserve job 3 job 2
33. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job 3 worker 1 worker 2 job 3
34. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker 1 worker 2
35. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue worker worker worker web app cron daemon db
36. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker input output
37. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM do work update database or save file[s] or new job all data necessary to complete job
38. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM resize image make thumbs Original Location thumbnail[s] image file new Name[s]
39. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM compute stats account Report account id account data
40. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM add to master report account report
41. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM post tweet api response twitter id access tokens tweet text metadata
42. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM update database api response
43. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM EXAMPLE
44. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM http://kr.github.io/beanstalkd/
45. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
46. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
47. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
48. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://github.com/dcousineau/queues-example
49. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM TIPS
50. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Either Read, or Write, never both
51. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM If you write, design for OVERWRITE
52. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A: 10 B: 20 worker 1 worker 2 A: ? B: ?
53. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A: 10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?
54. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A: 10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?
55. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B: 20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?
56. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B: 25 worker 1 worker 2 B: 25 A: 10 B: 20
57. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker 1 worker 2 A: 10 B: 25
58. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A: 10 B: 20 worker 1 worker 2 A: ? B: ?
59. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A: 10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?
60. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A: 10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?
61. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B: 20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?
62. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B: 25 worker 1 worker 2 A: 10 B: 25 B: 20
63. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker 1 worker 2 A: 10 B: 20 B: 25 Completed Reserve B: 25
64. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker 1 worker 2 A: 10 B: 25
65. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue worker web app cron db
66. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM scheduler Every X minutes, check: “What should be done NOW” worker db web app Assume DB is up to date
67. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM its fine to run a process let it die then resurrect it
68. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM OR: have long running process listen for jobs fork single-run job process
69. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Keep jobs & workers small keep surface area for failures small
70. THANKS. FOR YOUR ATTENTION DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://joind.in/13443 http://bit.ly/queues-ssp15