Tweet
Tweet

Slide 1

Slide 1 text

Integrating PostgreSQL with RabbitMQ Gavin M. Roy April 4th, 2014

Slide 2

Slide 2 text

About Me VP of Architecture
 AWeber Communications ! Blog: http://gavinroy.com Twitter: @Crad GitHub: https://github.com/gmr ! RabbitMQ in Depth (MEAP) http://manning.com/roy

Slide 3

Slide 3 text

RabbitMQ Primer

Slide 4

Slide 4 text

About RabbitMQ • Message Oriented Middleware • Primary identity is defined by AMQP 0-9-1 • Supports other protocols with Plugins:
 
 AMQP 1.0, HTTP, MQTT, STOMP, XMPP, UDP, WebStomp, and more • Written in Erlang/OTP, is Open Source (MPL), and is developed/maintained by Pivotal

Slide 5

Slide 5 text

Why RabbitMQ? • Create loosely coupled architectures • Decouple database write operations • Communicate across application platforms • Tap into pre-existing message flow for new purposes • Scale-out clustering for growth, throughput & HA • Federation for WAN latencies & network partitions

Slide 6

Slide 6 text

Native AMQP Clients C Clojure Cobol Common Lisp Delphi Erlang Go Groovy Haskell Java JavaScript .NET OCaml Perl PHP Python Ruby Scala

Slide 7

Slide 7 text

Who Uses It? Agora Games Chef Google AdMob Instagram MeetMe Mercado Libre Mozilla Nasa New York Times NSF Openstack Reddit

Slide 8

Slide 8 text

RabbitMQ Concepts

Slide 9

Slide 9 text

AMQP Message Body Properties

Slide 10

Slide 10 text

Message Properties app-id content-encoding content-type correlation-id delivery-mode expiration headers message-id priority reply-to timestamp type user-id Body Properties

Slide 11

Slide 11 text

Publishers and Consumers C P

Slide 12

Slide 12 text

Multiple Publishers P C P P

Slide 13

Slide 13 text

Multiple Consumers P C C C

Slide 14

Slide 14 text

AMQ Model Exchange
 defines routing behavior Queue
 stores messages in memory and optionally on disk Binding
 defines relationship between exchanges and queues X Queue Binding

Slide 15

Slide 15 text

Routing Keys • Provided when publishing a message • Compared against binding keys by exchanges • Example uses of a Routing Key: • Connotate the type of message • Designate the destination of a message • Categorize the content in the message

Slide 16

Slide 16 text

Built-In Exchange Types Direct
 String matching on Routing Key Fanout
 No routing key, messages sent to all bound queues Topic
 Pattern matching in Routing Key Headers
 No routing key, string matching in the message headers property

Slide 17

Slide 17 text

Topic Exchange Binding Keys namespace.delimited.keys #
 Receive all messages namespace.#
 Receive all messages in the namespace namespace.delimited.*
 Receive all namespace.delimited messages namespace.*.keys
 Receive all namespace messages ending in keys

Slide 18

Slide 18 text

Example Exchange Plugins Consistent Hashing
 Distribute messages via routing key hashed value Pulse
 Publish internal RabbitMQ metrics via AMQP Random
 Distribute messages across all bound queues randomly Recent History
 Stores last 20 messages to any bound queue Riak Storage 
 Store received messages in Riak" Script Exchange
 Calls out to external scripts for routing behavior

Slide 19

Slide 19 text

Exchange to Exchange Bindings X X X Queue Queue Queue

Slide 20

Slide 20 text

to RabbitMQ

Slide 21

Slide 21 text

pg_amqp • PostgreSQL Extension • User-defined functions to publish via AMQP • Developed by OmnTI • Invoke from user-defined function • Available on PGXN

Slide 22

Slide 22 text

Using pg_amqp • Has table of AMQP broker connections • Publishes body only AMQP messages • Transactional publishing via amqp.publish • Non-transactional via amqp.autonomous_publish • Needs some love for additional features

Slide 23

Slide 23 text

https://github.com/gmr/On-Rabbits-and-Elephants Example: SELECT amqp.publish(broker_id, 'name', 'routing-key', 'message'); Publishing a Message

Slide 24

Slide 24 text

PgSQL Listen Exchange • Exchange that issues LISTEN and publishes received notifications to bound queues • When routing messages, bindings are checked for the routing key to match the NOTIFY channel • Unobtrusive to PostgreSQL environment, native constructs for publishing messages • Requires PostgreSQL 9.0 and greater • Body only messages*

Slide 25

Slide 25 text

Sending a Notification psql (9.3.2) Type "help" for help. ! postgres=# \timing Timing is on. postgres=# NOTIFY channel_name, 
 ‘Test notification'; NOTIFY Time: 0.160 ms

Slide 26

Slide 26 text

Receiving a Notification psql (9.3.2) Type "help" for help. ! postgres=# LISTEN channel_name; ! (after NOTIFY issued) ! Asynchronous notification "channel_name" with payload "Test notification" received from server process with PID 16749.

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

Example Notification Message

Slide 29

Slide 29 text

to PostgreSQL

Slide 30

Slide 30 text

No direct integrations yet

Slide 31

Slide 31 text

Chapter 14 Creating a pgsql-storage RabbitMQ plugin

Slide 32

Slide 32 text

Other ways? via Consumer apps in the language of your choice via Apache Flume with RabbitMQ Source, PostgreSQL Sink

Slide 33

Slide 33 text

Questions? rabbitmq-pgsql-listen-exchange
 https://github.com/aweber/rabbitmq-pgsql-listen-exchange pg_ampq
 http://pgxn.org/dist/pg_amqp/ RabbitMQ in Depth 
 44% off code pgcf (includes other books as well)
 http://manning.com/roy/ Follow me on Twitter
 @Crad