Queues and the beanstalkd

Given at Sunshine PHP 2015

Transcript

1. QUEUES
   & THE GIANT
   BEANSTALKD
   

   View Slide

2. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
   @dcousineau
   

   View Slide

3. @croscon
   

   View Slide

4. View Slide

5. View Slide

6. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
   USE CASE
   

   View Slide

7. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
   

   View Slide

8. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
   

   View Slide

9. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
   

   View Slide

10. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    THEORY
    

    View Slide

11. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    QUEUE: The Data Structure
    

    View Slide

12. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    FIFO
    

    View Slide

13. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Synchronous Computation
    N seconds m Seconds
    n+m seconds
    

    View Slide

14. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    A
    Fetch Z
    Work
    Z
    5
    5
    Synchronous Computation
    

    View Slide

15. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    A
    Fetch Z
    Work
    Z
    5
    10
    10
    Work
    5
    Synchronous Computation
    

    View Slide

16. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Z
    15
    5
    10
    A
    Fetch Z
    Work
    5
    10
    Work
    15
    Synchronous Computation
    

    View Slide

17. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Asynchronous Computation
    N seconds
    m Seconds
    N seconds
    

    View Slide

18. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Asynchronous Computation
    Multi-Threading Multi-Process
    Memory
    Thread Thread
    Thread process process
    process
    memory Memory
    Memory
    

    View Slide

19. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Race Condition
    A B
    Fetch Z
    Work
    Z
    5
    5
    Work
    Fetch Z
    5
    

    View Slide

20. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Race Condition
    A B
    Fetch Z
    Work
    Z
    5
    10
    Work
    10
    Work
    Fetch Z
    5
    5
    

    View Slide

21. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Race Condition
    A B
    Fetch Z
    Fetch Z
    Work
    Z
    5
    5
    Work
    10
    Work
    15 15
    5
    10
    

    View Slide

22. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Locking
    A B
    Fetch Z
    Z
    5
    5
    Fetch Z
    A
    Work
    

    View Slide

23. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Locking
    A B
    Fetch Z
    Fetch Z
    Work
    Z
    5
    Work
    15 15
    15 B
    5
    A
    wait
    wait
    

    View Slide

24. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Locking
    A B
    Fetch Z
    Fetch Z
    Work
    Z
    5
    Work
    15
    15
    Work
    25
    25
    15
    B
    5
    A
    wait
    wait
    

    View Slide

25. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Dead Locking
    A B
    Fetch Z
    Z
    5
    5
    Fetch Y
    A
    Y
    10
    B
    10
    

    View Slide

26. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Dead Locks
    A B
    Fetch Z
    Z
    5
    5
    Fetch Y
    A
    Y
    10
    B
    10
    Fetch Y
    Fetch Z
    

    View Slide

27. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Dead Locks
    A B
    Fetch Z
    Z
    5
    5
    Fetch Y
    A
    Y
    10
    B
    10
    Fetch Y
    Fetch Z
    wait
    wait
    wait
    wait
    wait
    wait
    wait
    wait
    wait
    

    View Slide

28. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    QUEUING
    

    View Slide

29. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    job 1
    job 2
    worker 1
    worker 2
    

    View Slide

30. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    job 1
    job 2
    job 3
    worker 1
    worker 2
    reserve
    job 1
    

    View Slide

31. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    job 1
    job 2
    job 3
    worker 1
    worker 2
    reserve
    job 1
    job 2
    

    View Slide

32. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    job 2
    job 3
    worker 1
    worker 2
    Completed
    Reserve
    job 3
    job 2
    

    View Slide

33. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    job 3
    worker 1
    worker 2
    job 3
    

    View Slide

34. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    worker 1
    worker 2
    

    View Slide

35. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    queue
    worker worker worker
    web app cron daemon
    db
    

    View Slide

36. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    worker
    input output
    

    View Slide

37. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    do work
    update database
    or save file[s]
    or new job
    all data necessary to
    complete job
    

    View Slide

38. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    resize image
    make thumbs
    Original Location
    thumbnail[s]
    image file
    new Name[s]
    

    View Slide

39. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    compute stats account Report
    account id
    account data
    

    View Slide

40. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    add to
    master report
    account report
    

    View Slide

41. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    post tweet api response
    twitter id
    access tokens
    tweet text
    metadata
    

    View Slide

42. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    update database
    api response
    

    View Slide

43. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    EXAMPLE
    

    View Slide

44. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    http://kr.github.io/beanstalkd/
    

    View Slide

45. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    

    View Slide

46. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    

    View Slide

47. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    

    View Slide

48. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    https://github.com/dcousineau/queues-example
    

    View Slide

49. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    TIPS
    

    View Slide

50. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Either Read, or Write, never both
    

    View Slide

51. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    If you write, design for OVERWRITE
    

    View Slide

52. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    A: 10
    B: 20
    worker 1
    worker 2
    A: ?
    B: ?
    

    View Slide

53. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    A: 10
    B: 20
    B: 25
    worker 1
    worker 2
    reserve
    A: 10
    A: ?
    B: ?
    

    View Slide

54. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    A: 10
    B: 20
    B: 25
    worker 1
    worker 2
    reserve
    A: 10
    B: 20
    A: ?
    B: ?
    

    View Slide

55. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    B: 20
    B: 25
    worker 1
    worker 2
    Completed
    Reserve
    B: 25
    B: 20
    A: 10
    B: ?
    

    View Slide

56. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    B: 25
    worker 1
    worker 2
    B: 25
    A: 10
    B: 20
    

    View Slide

57. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    worker 1
    worker 2
    A: 10
    B: 25
    

    View Slide

58. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    A: 10
    B: 20
    worker 1
    worker 2
    A: ?
    B: ?
    

    View Slide

59. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    A: 10
    B: 20
    B: 25
    worker 1
    worker 2
    reserve
    A: 10
    A: ?
    B: ?
    

    View Slide

60. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    A: 10
    B: 20
    B: 25
    worker 1
    worker 2
    reserve
    A: 10
    B: 20
    A: ?
    B: ?
    

    View Slide

61. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    B: 20
    B: 25
    worker 1
    worker 2
    Completed
    Reserve
    B: 25
    B: 20
    A: 10
    B: ?
    

    View Slide

62. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    B: 25
    worker 1
    worker 2
    A: 10
    B: 25
    B: 20
    

    View Slide

63. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    worker 1
    worker 2
    A: 10
    B: 20
    B: 25
    Completed
    Reserve
    B: 25
    

    View Slide

64. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    worker 1
    worker 2
    A: 10
    B: 25
    

    View Slide

65. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    queue
    worker
    web app
    cron
    db
    

    View Slide

66. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    scheduler
    Every X minutes, check:
    “What should be done NOW”
    worker
    db web app
    Assume DB is up to date
    

    View Slide

67. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    its fine to run a process
    let it die
    then resurrect it
    

    View Slide

68. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    OR: have long running process listen for jobs
    fork single-run job process
    

    View Slide

69. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Keep jobs & workers small
    keep surface area for failures small
    

    View Slide

70. THANKS.
    FOR YOUR ATTENTION
    DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    https://joind.in/13443
    http://bit.ly/queues-ssp15
    

    View Slide