Slide 1

Slide 1 text

RabbitMQ Trong Dao Le [email protected] Software Engineer @ Quantcast

Slide 2

Slide 2 text

Why would you use a Message System? ¤  Loose Coupling ¤  Automatic performance scaling ¤  Scaling through configuration changes in the exchange bindings (splitting queues in topic exchanges (later in AMQP part)) ¤  Various message patterns: ¤  Request/Reply (RPC) ¤  Fire and forget ¤  Publish/Subscribe ¤  Auto-retry behavior (timeouts and transaction)

Slide 3

Slide 3 text

Loose coupling ¤  Sender is not responsible for issues such as: ¤  Is the receiver up? ¤  Could the receiver consume the message? ¤  Failure of delivery ¤  Instead of thinking about interface of function calls, think about the data and the flow of data ¤  Handling failure of response gracefully (timeouts) which doesn’t overload a system

Slide 4

Slide 4 text

Automatic performance scaling ¤  Deploy more servers listening to the same set of queues to handle more requests ¤  Multiple requesters won’t bring down your server (they have to bring down your broker, which is unlikely) ¤  No configuration files on the requesters side (Zookeeper can solve this, but it’s another system to maintain)

Slide 5

Slide 5 text

Our use case ¤  Accept a template and send text message to all the numbers with best effort ¤  Original approach: ¤  Insert into DB ¤  Cron job to read from DB and send message ¤  All using JTA!! (dreadful, but required as the accounts and the messages are on different DBs) ¤  Doesn’t scale well (max 500 messages, JTA timeout at 60s)

Slide 6

Slide 6 text

New simplified approach Catchs: 1/ DB contention as multiple requests to deduct the same account come at the same time. 2/ Multiple similar DB insertions are not batched. Solved with scheduled tasks by Dead-letter exchanges

Slide 7

Slide 7 text

Advantages of the new approach ¤  Distribute the work load of workers evenly ¤  Automatic retry if the transaction fail (by requeuing the message) ¤  Send message concurrently and faster

Slide 8

Slide 8 text

Migration from ActiveMQ to RabbitMQ ¤  Our chat system was using ActiveMQ (JMS) with scheduled delay to check if a message was delivered ¤  The scheduled delay was a fixed amount ¤  Migration to ActiveMQ is easy with dead-letter exchanges ¤  Using Spring, both ActiveMQ and RabbitMQ listeners are POJOs (specify the handling method in the configuration) ¤  Mostly configuration changes for listeners and type changes for senders.

Slide 9

Slide 9 text

AMQP vs JMS AMQP JMS 4 (5) models: direct, fanout, topic, header, (system) 2 models: 1-to-1 and pub/sub Binary data 5 different data types Clients send to exchanges, exchanges forward to queues Clients send directly to queues or topics Language agnostic Only support JVM languages Define a wire-level protocol standard Define a set of APIs

Slide 10

Slide 10 text

AMQP Ex Ex P P P C C C Producers Consumers Channels Channels Exchanges Queues Bindings

Slide 11

Slide 11 text

Direct Exchange

Slide 12

Slide 12 text

Fanout Exchange

Slide 13

Slide 13 text

Topic Exchange Ex D.E.A N.S.A C.I.A key="#.A" key=“#.I.#" F.B.I key="#" C.I.A N.S.A D.E.A F.B.I C.I.A D.E.A N.S.A C.I.A F.B.I

Slide 14

Slide 14 text

Other use cases ¤  Schedule task using dead-letter exchanges ¤  Distributed logging using topic exchanges ¤  Intermittent failure resistant workers (unsubscribe and message reject) ¤  Complicated event triggering (award the 1 million transaction)

Slide 15

Slide 15 text

Why use Spring-AMQP? ¤  The RabbitMQ Java Client has some drawbacks (just like JDBC) ¤  No channel pooling ¤  No retry ¤  Ugly listener code using while-loop ¤  Boiler plate codes

Slide 16

Slide 16 text

How to use RabbitMQ with Spring

Slide 17

Slide 17 text

RabbitMQ extension features ¤  Publisher confirms ¤  Consumer priorities ¤  Fast reply-to for RPC ¤  Exchange to exchange bindings ¤  Per-queue message TTL, per- message TTL ¤  Queue TTL ¤  Dead-letter exchanges ¤  Length limit

Slide 18

Slide 18 text

RabbitMQ vs ActiveMQ RabbitMQ ActiveMQ AMQP (0-8 and 0-9-1) AMQP (1.0) STOMP STOMP REST RSS/ATOM

Slide 19

Slide 19 text

HA: RabbitMQ vs ActiveMQ RabbitMQ ActiveMQ Master + Slaves Master + Slaves Dynamic clustering: Nodes can join/leave None that I know of Full replication (mirrored queues) or full visibility (clustering) Networks of brokers

Slide 20

Slide 20 text

Q/A

Slide 21

Slide 21 text

Looking for… ¤  Website Operations Engineer ¤  Data Operations Engineer Ops works + Develop tools to manage the data we have. Scala + Java developers, who has a knack for Ops.