$30 off During Our Annual Pro Sale. View Details »

Queues and the beanstalkd

Queues and the beanstalkd

Given at Sunshine PHP 2015

Daniel Cousineau

February 06, 2015
Tweet

More Decks by Daniel Cousineau

Other Decks in Programming

Transcript

  1. QUEUES
    & THE GIANT
    BEANSTALKD

    View Slide

  2. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    @dcousineau

    View Slide

  3. @croscon

    View Slide

  4. View Slide

  5. View Slide

  6. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    USE CASE

    View Slide

  7. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

    View Slide

  8. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

    View Slide

  9. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

    View Slide

  10. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    THEORY

    View Slide

  11. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    QUEUE: The Data Structure

    View Slide

  12. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    FIFO

    View Slide

  13. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Synchronous Computation
    N seconds m Seconds
    n+m seconds

    View Slide

  14. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    A
    Fetch Z
    Work
    Z
    5
    5
    Synchronous Computation

    View Slide

  15. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    A
    Fetch Z
    Work
    Z
    5
    10
    10
    Work
    5
    Synchronous Computation

    View Slide

  16. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Z
    15
    5
    10
    A
    Fetch Z
    Work
    5
    10
    Work
    15
    Synchronous Computation

    View Slide

  17. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Asynchronous Computation
    N seconds
    m Seconds
    N seconds

    View Slide

  18. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Asynchronous Computation
    Multi-Threading Multi-Process
    Memory
    Thread Thread
    Thread process process
    process
    memory Memory
    Memory

    View Slide

  19. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Race Condition
    A B
    Fetch Z
    Work
    Z
    5
    5
    Work
    Fetch Z
    5

    View Slide

  20. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Race Condition
    A B
    Fetch Z
    Work
    Z
    5
    10
    Work
    10
    Work
    Fetch Z
    5
    5

    View Slide

  21. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Race Condition
    A B
    Fetch Z
    Fetch Z
    Work
    Z
    5
    5
    Work
    10
    Work
    15 15
    5
    10

    View Slide

  22. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Locking
    A B
    Fetch Z
    Z
    5
    5
    Fetch Z
    A
    Work

    View Slide

  23. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Locking
    A B
    Fetch Z
    Fetch Z
    Work
    Z
    5
    Work
    15 15
    15 B
    5
    A
    wait
    wait

    View Slide

  24. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Locking
    A B
    Fetch Z
    Fetch Z
    Work
    Z
    5
    Work
    15
    15
    Work
    25
    25
    15
    B
    5
    A
    wait
    wait

    View Slide

  25. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Dead Locking
    A B
    Fetch Z
    Z
    5
    5
    Fetch Y
    A
    Y
    10
    B
    10

    View Slide

  26. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Dead Locks
    A B
    Fetch Z
    Z
    5
    5
    Fetch Y
    A
    Y
    10
    B
    10
    Fetch Y
    Fetch Z

    View Slide

  27. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Dead Locks
    A B
    Fetch Z
    Z
    5
    5
    Fetch Y
    A
    Y
    10
    B
    10
    Fetch Y
    Fetch Z
    wait
    wait
    wait
    wait
    wait
    wait
    wait
    wait
    wait

    View Slide

  28. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    QUEUING

    View Slide

  29. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    job 1
    job 2
    worker 1
    worker 2

    View Slide

  30. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    job 1
    job 2
    job 3
    worker 1
    worker 2
    reserve
    job 1

    View Slide

  31. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    job 1
    job 2
    job 3
    worker 1
    worker 2
    reserve
    job 1
    job 2

    View Slide

  32. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    job 2
    job 3
    worker 1
    worker 2
    Completed
    Reserve
    job 3
    job 2

    View Slide

  33. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    job 3
    worker 1
    worker 2
    job 3

    View Slide

  34. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    worker 1
    worker 2

    View Slide

  35. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    queue
    worker worker worker
    web app cron daemon
    db

    View Slide

  36. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    worker
    input output

    View Slide

  37. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    do work
    update database
    or save file[s]
    or new job
    all data necessary to
    complete job

    View Slide

  38. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    resize image
    make thumbs
    Original Location
    thumbnail[s]
    image file
    new Name[s]

    View Slide

  39. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    compute stats account Report
    account id
    account data

    View Slide

  40. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    add to
    master report
    account report

    View Slide

  41. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    post tweet api response
    twitter id
    access tokens
    tweet text
    metadata

    View Slide

  42. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    update database
    api response

    View Slide

  43. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    EXAMPLE

    View Slide

  44. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    http://kr.github.io/beanstalkd/

    View Slide

  45. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

    View Slide

  46. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

    View Slide

  47. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM

    View Slide

  48. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    https://github.com/dcousineau/queues-example

    View Slide

  49. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    TIPS

    View Slide

  50. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Either Read, or Write, never both

    View Slide

  51. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    If you write, design for OVERWRITE

    View Slide

  52. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    A: 10
    B: 20
    worker 1
    worker 2
    A: ?
    B: ?

    View Slide

  53. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    A: 10
    B: 20
    B: 25
    worker 1
    worker 2
    reserve
    A: 10
    A: ?
    B: ?

    View Slide

  54. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    A: 10
    B: 20
    B: 25
    worker 1
    worker 2
    reserve
    A: 10
    B: 20
    A: ?
    B: ?

    View Slide

  55. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    B: 20
    B: 25
    worker 1
    worker 2
    Completed
    Reserve
    B: 25
    B: 20
    A: 10
    B: ?

    View Slide

  56. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    B: 25
    worker 1
    worker 2
    B: 25
    A: 10
    B: 20

    View Slide

  57. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    worker 1
    worker 2
    A: 10
    B: 25

    View Slide

  58. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    A: 10
    B: 20
    worker 1
    worker 2
    A: ?
    B: ?

    View Slide

  59. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    A: 10
    B: 20
    B: 25
    worker 1
    worker 2
    reserve
    A: 10
    A: ?
    B: ?

    View Slide

  60. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    A: 10
    B: 20
    B: 25
    worker 1
    worker 2
    reserve
    A: 10
    B: 20
    A: ?
    B: ?

    View Slide

  61. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    B: 20
    B: 25
    worker 1
    worker 2
    Completed
    Reserve
    B: 25
    B: 20
    A: 10
    B: ?

    View Slide

  62. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    B: 25
    worker 1
    worker 2
    A: 10
    B: 25
    B: 20

    View Slide

  63. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    worker 1
    worker 2
    A: 10
    B: 20
    B: 25
    Completed
    Reserve
    B: 25

    View Slide

  64. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    worker 1
    worker 2
    A: 10
    B: 25

    View Slide

  65. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    queue
    worker
    web app
    cron
    db

    View Slide

  66. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    scheduler
    Every X minutes, check:
    “What should be done NOW”
    worker
    db web app
    Assume DB is up to date

    View Slide

  67. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    its fine to run a process
    let it die
    then resurrect it

    View Slide

  68. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    OR: have long running process listen for jobs
    fork single-run job process

    View Slide

  69. DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    Keep jobs & workers small
    keep surface area for failures small

    View Slide

  70. THANKS.
    FOR YOUR ATTENTION
    DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
    https://joind.in/13443
    http://bit.ly/queues-ssp15

    View Slide