Rails Underground 2009 RabbitMQ

Transcript

1. “Divide and conquer riding rabbits and trading gems” - a tale about distributed programming - Paolo Negri @hungryblank • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbit_starter
2. Resources • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbit_starter
3. About me Time GNU/Linux - Dbs “systems” Perl Python Ruby PHP
4. Summary:
5. Distributed Concurrent Programming
6. http://www.flickr.com/photos/myxi/448253580 rabbitMQ
7. Control
8. The problem Given a dictionary of 1.000.000 search phrases, compare for each one the first page of results on google.com and bing.com
9. Don’t do it. It’s a violation of terms of service! (Sorry, I needed an example that required little or no explanation)
10. Comparison
11. The whole process
12. Great! but... • Fetching, parsing and analyzing 2.000.000 pages of results will take a long time • we want to collect the statistics in a limited time and this sequential approach is not quick enough
13. Distributed Computing Is a method of solving computational problem by dividing the problem into many tasks run simultaneously on many hardware or software systems (Wikipedia)
14. Map Reduce "Map" step the master node takes the input, chops it up into smaller sub-problems, and distributes those to worker nodes. (Wikipedia)
15. Problems: •How many nodes? •How many workers? •Distribution mechanism to feed the workers?
16. What about queuing? • the master node takes the input, chops it up into smaller sub-problems, and publishes them in a queue • workers independently consume the content of the queue
17. Here comes • RabbitMQ is an implementation of AMQP, the emerging standard for high performance enterprise messaging • It’s opensource • Written in Erlang
18. Erlang? • general-purpose concurrent programming language designed by Ericsson • first version written by J. Armstrong in 1986 • distributed • fault tolerant • soft real time • high availability
19. Erlang - is coming back • Projects • CouchDB - RESTful document storage • WebMachine - REST toolkit • Nitrogen - web framework • Mochiweb - web framework • Ruby shops using it • 37 Signal - Campfire • Heroku
20. + Erlang It’s messages all the way down
21. Install it •sudo apt-get install rabbitmq •sudo gem install tmm1-amqp Note: rabbitMQ must be v1.6.0 and amqp gem v 0.6.4 to follow the code in the slides
22. Do it! - master node
23. Do it! - worker node
24. Get for free • Decoupling master/worker • Workers take care of feeding themselves • Flexible number of workers
25. Behind the scenes msg A Queue Worker1 Worker2 Worker3 Master Exchange
26. Behind the scenes Queue Worker1 Worker2 Worker3 Master Exchange msg A
27. Behind the scenes Queue Worker1 Worker2 Worker3 Master Exchange msg A
28. Behind the scenes Queue Worker1 Worker2 Worker3 Master Exchange msg A
29. RabbitMQ • Multiple exchanges (and multiple types of exchange) • Multiple queues • Queues are connected by bindings to exchanges • Exchanges route messages to queues
30. RabbitMQ • Exchanges and queues have names • BUT direct exchanges created implicitly are not public and don’t have name • Queues and messages are resident in RAM and can be persisted on disk (at a performance price)
31. What and where RabbitMQ (Erlang) TCP/IP Master (ruby) Worker (ruby) Worker (ruby) Queue Exchange Worker (ruby)
32. Problem #1 If a worker has a problem we might lose one or more messages http://www.flickr.com/photos/danzen/2288625136/
33. Solution - ACK in worker
34. Acknoledgement • messages - 1 or more depending by prefetch settings - are passed to the client • the client processes and acknowledges the messages one by one • if the client connection closes => unacknowledged messages go back in the queue
35. Problem #2 No convenient way to control the workers http://www.flickr.com/photos/streetfly_jz/2770586821/
36. System queue - worker
37. System queue - control • save it as system_command.rb • ruby system_command.rb halt
38. System queue Queue1 Queue3 Queue2 Worker1 Worker2 Worker3 Control script Exchange msg A
39. System queue Queue1 Queue3 Queue2 Worker1 Worker2 Worker3 Control script Exchange msg A
40. System queue Queue1 Queue3 Queue2 Worker1 Worker2 Worker3 Control script Exchange msg A msg A msg A
41. EventMachine
42. EventMachine • Non blocking IO and lightweight concurrency • eliminate the complexities of high- performance threaded network programming Is an implementation of Reactor Pattern
43. without EM code network operation use network operation result Free code network operation use network operation result Callback Free Free Time with EM Ruby process Ruby process
44. EventMachine amqp gem is built on top of EventMachine => you’re in a context where you can leverage concurrent programming
45. EM - Deferrables “The Deferrable pattern allows you to specify any number of Ruby code blocks that will be executed at some future time when the status of the Deferrable object changes “
46. EM - Deferrables
47. EM - Deferrables
48. EM - Deferrables
49. EM - Deferrables
50. Deferrables GooglePage BingPage without deferrables with deferrables Time GooglePage BingPage
51. Problem #3 How many of them? what are they doing? http://www.flickr.com/photos/philocrites/341850461/
52. Heartbeat - worker
53. Heartbeat monitor
54. Heartbeat queue msg A Queue Worker1 Worker2 Worker3 Monitor Exchange msg B
55. Heartbeat queue Queue Worker1 Worker2 Worker3 Monitor Exchange msg A msg B
56. Heartbeat queue Queue Worker1 Worker2 Worker3 Monitor Exchange msg A msg B
57. Clustering RabbitMQ - node C RabbitMQ - node A RabbitMQ - node B TCP/IP TCP/IP TCP/IP
58. async vs sync • famoseagle/carrot • celldee/bunny Syncronous clients on github
59. Rabbits on github • danielsdeleo/moqueue Tests/stubbing • Check out ezmobius/nanite “self assembling fabric of ruby daemons” • Keep an eye on tonyg/rabbithub implementation of pubsubhubbub (PubSub over REST) • auser/alice web app to monitor RabbitMQ
60. More rabbits on github • tmm1/em-spec • eventmachine/eventmachine • tmm1/amqp • macournoyer/thin
61. Q&A ?
62. Thanks! (Thanks Mark!) Paolo Negri / hungryblank