Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Queues And The Giant beanstalkd

Queues And The Giant beanstalkd

In the modern world of scaling web applications, queues have become a go-to for increasing performance by parallelizing and backgrounding heavy duty tasks. Learn practical usages of queues through beanstalkd.

Given at Feb 2013 DallasPHP meeting and ConFoo 2013

Daniel Cousineau

February 13, 2013
Tweet

More Decks by Daniel Cousineau

Other Decks in Programming

Transcript

  1. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    Queues and the Giant beanstalkd

    View full-size slide

  2. Daniel Cousineau
    Developer at SplashMedia
    @dcousineau

    View full-size slide

  3. use
    Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

    View full-size slide

  4. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

    View full-size slide

  5. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

    View full-size slide

  6. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

    View full-size slide

  7. Theory
    Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

    View full-size slide

  8. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    QUEUE: The Data Structure

    View full-size slide

  9. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    FIFO

    View full-size slide

  10. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    synchronous Computation
    N seconds m Seconds
    n+m seconds

    View full-size slide

  11. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    Synchronous Computation
    A
    Fetch Z
    Work
    Z
    5
    5

    View full-size slide

  12. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    Synchronous Computation
    A
    Fetch Z
    Work
    Z
    5
    10
    10
    Work
    5

    View full-size slide

  13. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    Synchronous Computation
    Z
    15
    5
    10
    A
    Fetch Z
    Work
    5
    10
    Work
    15

    View full-size slide

  14. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    Asynchronous Computation
    N seconds
    m Seconds
    N seconds

    View full-size slide

  15. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    Asynchronous Computation
    Multi-threading Multi-Process
    Memory
    Thread Thread
    Thread process process
    process
    memory Memory
    Memory

    View full-size slide

  16. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    Race Condition
    A B
    Fetch Z
    Work
    Z
    5
    5
    Work
    Fetch Z
    5

    View full-size slide

  17. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    Race Condition
    A B
    Fetch Z
    Work
    Z
    5
    10
    Work
    10
    Work
    Fetch Z
    5
    5

    View full-size slide

  18. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    Race Condition
    A B
    Fetch Z
    Fetch Z
    Work
    Z
    5
    5
    Work
    10
    Work
    15 15
    5
    10

    View full-size slide

  19. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    Locking
    A B
    Fetch Z
    Z
    5
    5
    Fetch Z
    A
    Work

    View full-size slide

  20. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    Locking
    A B
    Fetch Z
    Fetch Z
    Work
    Z
    5
    Work
    15 15
    15 B
    5
    A
    wait
    wait

    View full-size slide

  21. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    Locking
    A B
    Fetch Z
    Fetch Z
    Work
    Z
    5
    Work
    15
    15
    Work
    25
    25
    15
    B
    5
    A
    wait
    wait

    View full-size slide

  22. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    DEAD LOCKING
    A B
    Fetch Z
    Z
    5
    5
    Fetch Y
    A
    Y
    10
    b
    10

    View full-size slide

  23. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    DEAD LOCKS
    A B
    Fetch Z
    Z
    5
    5
    Fetch Y
    A
    Y
    10
    b
    10
    Fetch Y
    Fetch Z

    View full-size slide

  24. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    DEAD LOCKS
    A B
    Fetch Z
    Z
    5
    5
    Fetch Y
    A
    Y
    10
    b
    10
    Fetch Y
    Fetch Z
    wait
    wait
    wait
    wait
    wait
    wait
    wait
    wait
    wait

    View full-size slide

  25. Execution
    Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

    View full-size slide

  26. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    job 1
    job 2
    worker 1
    worker 2

    View full-size slide

  27. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    job 1
    job 2
    job 3
    worker 1
    worker 2
    reserve
    job 1

    View full-size slide

  28. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    job 1
    job 2
    job 3
    worker 1
    worker 2
    reserve
    job 1
    job 2

    View full-size slide

  29. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    job 2
    job 3
    worker 1
    worker 2
    Completed
    Reserve
    job 3
    job 2

    View full-size slide

  30. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    job 3
    worker 1
    worker 2
    job 3

    View full-size slide

  31. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    worker 1
    worker 2

    View full-size slide

  32. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    queue
    worker worker worker
    web app CRON daemon
    db

    View full-size slide

  33. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    Worker
    input output

    View full-size slide

  34. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    do work
    update database
    or save file[s]
    or new job
    all data
    necessary to
    complete job

    View full-size slide

  35. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    resize image
    make thumbs
    Original Location
    thumbnail[s]
    image file
    new Name[s]

    View full-size slide

  36. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    compute
    stats account Report
    account id
    account data

    View full-size slide

  37. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    add to
    master
    report
    account report

    View full-size slide

  38. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    post tweet api response
    twitter id
    access tokens
    tweet text
    metadata

    View full-size slide

  39. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    update
    database
    api response

    View full-size slide

  40. example
    Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

    View full-size slide

  41. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    http://kr.github.com/beanstalkd

    View full-size slide

  42. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

    View full-size slide

  43. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

    View full-size slide

  44. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

    View full-size slide

  45. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    https://github.com/dcousineau/queues-example

    View full-size slide

  46. Special
    Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

    View full-size slide

  47. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    Either Read, or Write, never both

    View full-size slide

  48. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    If you write, design for overwrite

    View full-size slide

  49. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    A: 10
    B: 20
    worker 1
    worker 2
    A: ?
    B: ?

    View full-size slide

  50. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    A: 10
    B: 20
    B: 25
    worker 1
    worker 2
    reserve
    a: 10
    A: ?
    B: ?

    View full-size slide

  51. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    A: 10
    B: 20
    B: 25
    worker 1
    worker 2
    reserve
    a: 10
    B: 20
    A: ?
    B: ?

    View full-size slide

  52. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    B: 20
    B: 25
    worker 1
    worker 2
    Completed
    Reserve
    B: 25
    B: 20
    A: 10
    B: ?

    View full-size slide

  53. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    B: 25
    worker 1
    worker 2
    B: 25
    A: 10
    B: 20

    View full-size slide

  54. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    worker 1
    worker 2
    A: 10
    B: 25

    View full-size slide

  55. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    A: 10
    B: 20
    worker 1
    worker 2
    A: ?
    B: ?

    View full-size slide

  56. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    A: 10
    B: 20
    B: 25
    worker 1
    worker 2
    reserve
    a: 10
    A: ?
    B: ?

    View full-size slide

  57. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    A: 10
    B: 20
    B: 25
    worker 1
    worker 2
    reserve
    a: 10
    B: 20
    A: ?
    B: ?

    View full-size slide

  58. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    worker 1
    worker 2
    B: 25
    B: 20
    A: 10
    B: ?
    A: 10
    B: 20
    B: 25
    STUCK!

    View full-size slide

  59. Q&
    A
    Put your questions
    Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

    View full-size slide

  60. THANKS.
    FOR YOUR ATTENTION
    Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
    http://joind.in/7873

    View full-size slide