Rails Underground 2009 RabbitMQ

Rails Underground 2009 RabbitMQ

Ada486a7a6724ac155c537674f6942ae?s=128

hungryblank

April 24, 2012
Tweet

Transcript

  1. “Divide and conquer riding rabbits and trading gems” - a

    tale about distributed programming - Paolo Negri @hungryblank • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbit_starter
  2. Resources • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbit_starter

  3. About me Time GNU/Linux - Dbs “systems” Perl Python Ruby

    PHP
  4. Summary:

  5. Distributed Concurrent Programming

  6. http://www.flickr.com/photos/myxi/448253580 rabbitMQ

  7. Control

  8. The problem Given a dictionary of 1.000.000 search phrases, compare

    for each one the first page of results on google.com and bing.com
  9. Don’t do it. It’s a violation of terms of service!

    (Sorry, I needed an example that required little or no explanation)
  10. Comparison

  11. The whole process

  12. Great! but... • Fetching, parsing and analyzing 2.000.000 pages of

    results will take a long time • we want to collect the statistics in a limited time and this sequential approach is not quick enough
  13. Distributed Computing Is a method of solving computational problem by

    dividing the problem into many tasks run simultaneously on many hardware or software systems (Wikipedia)
  14. Map Reduce "Map" step the master node takes the input,

    chops it up into smaller sub-problems, and distributes those to worker nodes. (Wikipedia)
  15. Problems: •How many nodes? •How many workers? •Distribution mechanism to

    feed the workers?
  16. What about queuing? • the master node takes the input,

    chops it up into smaller sub-problems, and publishes them in a queue • workers independently consume the content of the queue
  17. Here comes • RabbitMQ is an implementation of AMQP, the

    emerging standard for high performance enterprise messaging • It’s opensource • Written in Erlang
  18. Erlang? • general-purpose concurrent programming language designed by Ericsson •

    first version written by J. Armstrong in 1986 • distributed • fault tolerant • soft real time • high availability
  19. Erlang - is coming back • Projects • CouchDB -

    RESTful document storage • WebMachine - REST toolkit • Nitrogen - web framework • Mochiweb - web framework • Ruby shops using it • 37 Signal - Campfire • Heroku
  20. + Erlang It’s messages all the way down

  21. Install it •sudo apt-get install rabbitmq •sudo gem install tmm1-amqp

    Note: rabbitMQ must be v1.6.0 and amqp gem v 0.6.4 to follow the code in the slides
  22. Do it! - master node

  23. Do it! - worker node

  24. Get for free • Decoupling master/worker • Workers take care

    of feeding themselves • Flexible number of workers
  25. Behind the scenes msg A Queue Worker1 Worker2 Worker3 Master

    Exchange
  26. Behind the scenes Queue Worker1 Worker2 Worker3 Master Exchange msg

    A
  27. Behind the scenes Queue Worker1 Worker2 Worker3 Master Exchange msg

    A
  28. Behind the scenes Queue Worker1 Worker2 Worker3 Master Exchange msg

    A
  29. RabbitMQ • Multiple exchanges (and multiple types of exchange) •

    Multiple queues • Queues are connected by bindings to exchanges • Exchanges route messages to queues
  30. RabbitMQ • Exchanges and queues have names • BUT direct

    exchanges created implicitly are not public and don’t have name • Queues and messages are resident in RAM and can be persisted on disk (at a performance price)
  31. What and where RabbitMQ (Erlang) TCP/IP Master (ruby) Worker (ruby)

    Worker (ruby) Queue Exchange Worker (ruby)
  32. Problem #1 If a worker has a problem we might

    lose one or more messages http://www.flickr.com/photos/danzen/2288625136/
  33. Solution - ACK in worker

  34. Acknoledgement • messages - 1 or more depending by prefetch

    settings - are passed to the client • the client processes and acknowledges the messages one by one • if the client connection closes => unacknowledged messages go back in the queue
  35. Problem #2 No convenient way to control the workers http://www.flickr.com/photos/streetfly_jz/2770586821/

  36. System queue - worker

  37. System queue - control • save it as system_command.rb •

    ruby system_command.rb halt
  38. System queue Queue1 Queue3 Queue2 Worker1 Worker2 Worker3 Control script

    Exchange msg A
  39. System queue Queue1 Queue3 Queue2 Worker1 Worker2 Worker3 Control script

    Exchange msg A
  40. System queue Queue1 Queue3 Queue2 Worker1 Worker2 Worker3 Control script

    Exchange msg A msg A msg A
  41. EventMachine

  42. EventMachine • Non blocking IO and lightweight concurrency • eliminate

    the complexities of high- performance threaded network programming Is an implementation of Reactor Pattern
  43. without EM code network operation use network operation result Free

    code network operation use network operation result Callback Free Free Time with EM Ruby process Ruby process
  44. EventMachine amqp gem is built on top of EventMachine =>

    you’re in a context where you can leverage concurrent programming
  45. EM - Deferrables “The Deferrable pattern allows you to specify

    any number of Ruby code blocks that will be executed at some future time when the status of the Deferrable object changes “
  46. EM - Deferrables

  47. EM - Deferrables

  48. EM - Deferrables

  49. EM - Deferrables

  50. Deferrables GooglePage BingPage without deferrables with deferrables Time GooglePage BingPage

  51. Problem #3 How many of them? what are they doing?

    http://www.flickr.com/photos/philocrites/341850461/
  52. Heartbeat - worker

  53. Heartbeat monitor

  54. Heartbeat queue msg A Queue Worker1 Worker2 Worker3 Monitor Exchange

    msg B
  55. Heartbeat queue Queue Worker1 Worker2 Worker3 Monitor Exchange msg A

    msg B
  56. Heartbeat queue Queue Worker1 Worker2 Worker3 Monitor Exchange msg A

    msg B
  57. Clustering RabbitMQ - node C RabbitMQ - node A RabbitMQ

    - node B TCP/IP TCP/IP TCP/IP
  58. async vs sync • famoseagle/carrot • celldee/bunny Syncronous clients on

    github
  59. Rabbits on github • danielsdeleo/moqueue Tests/stubbing • Check out ezmobius/nanite

    “self assembling fabric of ruby daemons” • Keep an eye on tonyg/rabbithub implementation of pubsubhubbub (PubSub over REST) • auser/alice web app to monitor RabbitMQ
  60. More rabbits on github • tmm1/em-spec • eventmachine/eventmachine • tmm1/amqp

    • macournoyer/thin
  61. Q&A ?

  62. Thanks! (Thanks Mark!) Paolo Negri / hungryblank