Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Railsconf 2009 Rabbitmq
Search
hungryblank
April 24, 2012
Programming
1
140
Railsconf 2009 Rabbitmq
hungryblank
April 24, 2012
Tweet
Share
More Decks by hungryblank
See All by hungryblank
Rails Underground 2009 RabbitMQ
hungryblank
3
750
Other Decks in Programming
See All in Programming
TypeScriptでDXを上げろ! Hono編
yusukebe
3
860
SwiftでMCPサーバーを作ろう!
giginet
PRO
2
190
リバースエンジニアリング新時代へ! GhidraとClaude DesktopをMCPで繋ぐ/findy202507
tkmru
4
1.2k
The Niche of CDK Grant オブジェクトって何者?/the-niche-of-cdk-what-isgrant-object
hassaku63
1
690
202507_ADKで始めるエージェント開発の基本 〜デモを通じて紹介〜(奥田りさ)
risatube
PRO
3
540
AIともっと楽するE2Eテスト
myohei
9
3.1k
20250708_JAWS_opscdk
takuyay0ne
2
150
Googleの新しいコーディングAIエージェントJulesを使ってみた
tonionagauzzi
0
120
AI Ramen Fight
yusukebe
0
110
脱Riverpod?fqueryで考える、TanStack Queryライクなアーキテクチャの可能性
ostk0069
0
560
初学者でも今すぐできる、Claude Codeの生産性を10倍上げるTips
s4yuba
16
13k
状態遷移図を書こう / Sequence Chart vs State Diagram
orgachem
PRO
3
250
Featured
See All Featured
Fireside Chat
paigeccino
37
3.5k
Optimising Largest Contentful Paint
csswizardry
37
3.3k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.9k
Site-Speed That Sticks
csswizardry
10
720
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
181
54k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
44
2.4k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
How to train your dragon (web standard)
notwaldorf
96
6.1k
Designing for humans not robots
tammielis
253
25k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Transcript
%w(map reduce).first A Tale About Rabbits, Latency and Slim Crontabs
Paolo Negri thanks to: www.autoscout24.de
Summary:
Map http://www.matthiasdittrich.com/projekte/dliste/visualisations/index.html
http://www.flickr.com/photos/myxi/448253580 rabbitMQ
crontab diet http://www.flickr.com/photos/tim_norris/2600843131/
Map Reduce “Programming model for processing and generating large data
sets” (Google)
Map Reduce "Map" step the master node takes the input,
chops it up into smaller sub-problems, and distributes those to worker nodes. (Wikipedia)
The problem Invoicing our clients
Is it as simple as... clients.map do |client| client.invoice end
No! •distributed •concurrent Because the process is:
Problems: •How many nodes? •How many workers? •Distribution mechanism to
feed the workers?
What about queuing? • the master node takes the input,
chops it up into smaller sub-problems, and publishes them in a queue • workers independently consume the content of the queue
Here comes • RabbitMQ is an implementation of AMQP, the
emerging standard for high performance enterprise messaging • It’s opensource • Can be used to manage queues • Written in Erlang
Erlang? • Erlang is a general-purpose concurrent programming language designed
by Ericsson • distributed • fault tolerant • soft real time • high availability
Install it •sudo apt-get install rabbitmq •sudo gem install tmm1-amqp
Do it - master node
Use it - worker node
What and where RabbitMQ (Erlang) TCP/IP Master (ruby) Worker (ruby)
Worker (ruby)
Get for free • Decoupling master/worker • Workers take care
of feeding themselves • Flexible number of workers
Get for free • RabbitMQ can be clustered • Support
of message acknowledgement • Queues can be persisted on disk (at a price) • low latency
Queue • Is an actual entity • has a name
• can be inspected and managed
EventMachine
EventMachine • Non blocking IO and lightweight concurrency • eliminate
the complexities of high- performance threaded network programming Is an implementation of Reactor Pattern
EventMachine
EventMachine amqp gem is built on EventMachine => you’re in
a context where you can leverage concurrent programming
EM - Deferrables
EM - Deferrables “The Deferrable pattern allows you to specify
any number of Ruby code blocks that will be executed at some future time when the status of the Deferrable object changes “
EM - Deferrables
EM - Deferrables
Deferrables ClientStat Arrears without deferrables with deferrables Time ClientStat Arrears
Achieved so far • Easy distribution of tasks • Architecture
that supports arbitrary number of workers (and masters) • Concurrency within the single worker
More rabbits Analogy with email system
Multicasting - producer
Multicasting - consumer
Multicasting Exchange msg A Queue1 Queue3 Queue2 Cons1 Cons2 Cons3
Publisher
Multicasting Exchange Queue1 Queue3 Queue2 msg A msg A msg
A Cons1 Cons2 Cons3 Publisher
Not only queues then • communication across hosts, heterogeneous systems
• low latency • clustering Use messages distribution to build the nervous system of your app
Where to start? crontab -l 5 * * * *
bin/do_the_quick_thing.rb 0 2 * * * bin/do_the_scary_thing.rb
Cron • Simple • Reliable • No maintenance • Status
is not explicit • Locking? • Shot and forget
Queue • Distributed easily • Reliable • Can be inspected
• Add/decrease workers •Makes you think! • Adds more complexity
On github - Projects • eventmachine/eventmachine • tmm1/amqp • macournoyer/thin
• famoseagle/carrot • celldee/bunny • ezmobius/nanite
Q&A ?
Thanks! Paolo Negri / hungryblank