RabbitMQ Trong Dao Le tdaole@quantcast.com Software Engineer @ Quantcast

Why would you use a Message System? ¤  Loose Coupling ¤  Automatic performance scaling ¤  Scaling through configuration changes in the exchange bindings (splitting queues in topic exchanges (later in AMQP part)) ¤  Various message patterns: ¤  Request/Reply (RPC) ¤  Fire and forget ¤  Publish/Subscribe ¤  Auto-retry behavior (timeouts and transaction)

Loose coupling ¤  Sender is not responsible for issues such as: ¤  Is the receiver up? ¤  Could the receiver consume the message? ¤  Failure of delivery ¤  Instead of thinking about interface of function calls, think about the data and the flow of data ¤  Handling failure of response gracefully (timeouts) which doesn’t overload a system

Automatic performance scaling ¤  Deploy more servers listening to the same set of queues to handle more requests ¤  Multiple requesters won’t bring down your server (they have to bring down your broker, which is unlikely) ¤  No configuration files on the requesters side (Zookeeper can solve this, but it’s another system to maintain)

Our use case ¤  Accept a template and send text message to all the numbers with best effort ¤  Original approach: ¤  Insert into DB ¤  Cron job to read from DB and send message ¤  All using JTA!! (dreadful, but required as the accounts and the messages are on different DBs) ¤  Doesn’t scale well (max 500 messages, JTA timeout at 60s)

New simplified approach Catchs: 1/ DB contention as multiple requests to deduct the same account come at the same time. 2/ Multiple similar DB insertions are not batched. Solved with scheduled tasks by Dead-letter exchanges

Advantages of the new approach ¤  Distribute the work load of workers evenly ¤  Automatic retry if the transaction fail (by requeuing the message) ¤  Send message concurrently and faster

Migration from ActiveMQ to RabbitMQ ¤  Our chat system was using ActiveMQ (JMS) with scheduled delay to check if a message was delivered ¤  The scheduled delay was a fixed amount ¤  Migration to ActiveMQ is easy with dead-letter exchanges ¤  Using Spring, both ActiveMQ and RabbitMQ listeners are POJOs (specify the handling method in the configuration) ¤  Mostly configuration changes for listeners and type changes for senders.

AMQP vs JMS AMQP JMS 4 (5) models: direct, fanout, topic, header, (system) 2 models: 1-to-1 and pub/sub Binary data 5 different data types Clients send to exchanges, exchanges forward to queues Clients send directly to queues or topics Language agnostic Only support JVM languages Define a wire-level protocol standard Define a set of APIs

AMQP Ex Ex P P P C C C Producers Consumers Channels Channels Exchanges Queues Bindings

Direct Exchange

Fanout Exchange

Topic Exchange Ex D.E.A N.S.A C.I.A key="#.A" key=“#.I.#" F.B.I key="#" C.I.A N.S.A D.E.A F.B.I C.I.A D.E.A N.S.A C.I.A F.B.I

Other use cases ¤  Schedule task using dead-letter exchanges ¤  Distributed logging using topic exchanges ¤  Intermittent failure resistant workers (unsubscribe and message reject) ¤  Complicated event triggering (award the 1 million transaction)

Why use Spring-AMQP? ¤  The RabbitMQ Java Client has some drawbacks (just like JDBC) ¤  No channel pooling ¤  No retry ¤  Ugly listener code using while-loop ¤  Boiler plate codes

How to use RabbitMQ with Spring

RabbitMQ extension features ¤  Publisher confirms ¤  Consumer priorities ¤  Fast reply-to for RPC ¤  Exchange to exchange bindings ¤  Per-queue message TTL, per- message TTL ¤  Queue TTL ¤  Dead-letter exchanges ¤  Length limit

RabbitMQ vs ActiveMQ RabbitMQ ActiveMQ AMQP (0-8 and 0-9-1) AMQP (1.0) STOMP STOMP REST RSS/ATOM

HA: RabbitMQ vs ActiveMQ RabbitMQ ActiveMQ Master + Slaves Master + Slaves Dynamic clustering: Nodes can join/leave None that I know of Full replication (mirrored queues) or full visibility (clustering) Networks of brokers

Q/A

Looking for… ¤  Website Operations Engineer ¤  Data Operations Engineer Ops works + Develop tools to manage the data we have. Scala + Java developers, who has a knack for Ops.