Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Queues And The Giant beanstalkd

Queues And The Giant beanstalkd

In the modern world of scaling web applications, queues have become a go-to for increasing performance by parallelizing and backgrounding heavy duty tasks. Learn practical usages of queues through beanstalkd.

Given at Feb 2013 DallasPHP meeting and ConFoo 2013

6696617169722009ed1ec8c52496c6da?s=128

Daniel Cousineau

February 13, 2013
Tweet

Transcript

  1. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Queues

    and the Giant beanstalkd
  2. Daniel Cousineau Developer at SplashMedia @dcousineau

  3. use Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

  4. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

  5. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

  6. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

  7. Theory Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

  8. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com QUEUE:

    The Data Structure
  9. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com FIFO

  10. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com synchronous

    Computation N seconds m Seconds n+m seconds
  11. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Synchronous

    Computation A Fetch Z Work Z 5 5
  12. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Synchronous

    Computation A Fetch Z Work Z 5 10 10 Work 5
  13. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Synchronous

    Computation Z 15 5 10 A Fetch Z Work 5 10 Work 15
  14. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Asynchronous

    Computation N seconds m Seconds N seconds
  15. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Asynchronous

    Computation Multi-threading Multi-Process Memory Thread Thread Thread process process process memory Memory Memory
  16. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Race

    Condition A B Fetch Z Work Z 5 5 Work Fetch Z 5
  17. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Race

    Condition A B Fetch Z Work Z 5 10 Work 10 Work Fetch Z 5 5
  18. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Race

    Condition A B Fetch Z Fetch Z Work Z 5 5 Work 10 Work 15 15 5 10
  19. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Locking

    A B Fetch Z Z 5 5 Fetch Z A Work
  20. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Locking

    A B Fetch Z Fetch Z Work Z 5 Work 15 15 15 B 5 A wait wait
  21. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Locking

    A B Fetch Z Fetch Z Work Z 5 Work 15 15 Work 25 25 15 B 5 A wait wait
  22. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com DEAD

    LOCKING A B Fetch Z Z 5 5 Fetch Y A Y 10 b 10
  23. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com DEAD

    LOCKS A B Fetch Z Z 5 5 Fetch Y A Y 10 b 10 Fetch Y Fetch Z
  24. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com DEAD

    LOCKS A B Fetch Z Z 5 5 Fetch Y A Y 10 b 10 Fetch Y Fetch Z wait wait wait wait wait wait wait wait wait
  25. Execution Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

  26. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com job

    1 job 2 worker 1 worker 2
  27. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com job

    1 job 2 job 3 worker 1 worker 2 reserve job 1
  28. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com job

    1 job 2 job 3 worker 1 worker 2 reserve job 1 job 2
  29. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com job

    2 job 3 worker 1 worker 2 Completed Reserve job 3 job 2
  30. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com job

    3 worker 1 worker 2 job 3
  31. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com worker

    1 worker 2
  32. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com queue

    worker worker worker web app CRON daemon db
  33. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Worker

    input output
  34. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com do

    work update database or save file[s] or new job all data necessary to complete job
  35. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com resize

    image make thumbs Original Location thumbnail[s] image file new Name[s]
  36. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com compute

    stats account Report account id account data
  37. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com add

    to master report account report
  38. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com post

    tweet api response twitter id access tokens tweet text metadata
  39. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com update

    database api response
  40. example Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

  41. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com http://kr.github.com/beanstalkd

  42. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

  43. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

  44. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

  45. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com https://github.com/dcousineau/queues-example

  46. Special Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com

  47. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Either

    Read, or Write, never both
  48. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com If

    you write, design for overwrite
  49. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com A:

    10 B: 20 worker 1 worker 2 A: ? B: ?
  50. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com A:

    10 B: 20 B: 25 worker 1 worker 2 reserve a: 10 A: ? B: ?
  51. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com A:

    10 B: 20 B: 25 worker 1 worker 2 reserve a: 10 B: 20 A: ? B: ?
  52. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com B:

    20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?
  53. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com B:

    25 worker 1 worker 2 B: 25 A: 10 B: 20
  54. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com worker

    1 worker 2 A: 10 B: 25
  55. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com A:

    10 B: 20 worker 1 worker 2 A: ? B: ?
  56. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com A:

    10 B: 20 B: 25 worker 1 worker 2 reserve a: 10 A: ? B: ?
  57. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com A:

    10 B: 20 B: 25 worker 1 worker 2 reserve a: 10 B: 20 A: ? B: ?
  58. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com worker

    1 worker 2 B: 25 B: 20 A: 10 B: ? A: 10 B: 20 B: 25 STUCK!
  59. Q& A Put your questions Daniel Cousineau // follow me

    : @dcousineau or http://dcousineau.com
  60. THANKS. FOR YOUR ATTENTION Daniel Cousineau // follow me :

    @dcousineau or http://dcousineau.com http://joind.in/7873