Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Integrating PostgreSQL with RabbitMQ

Integrating PostgreSQL with RabbitMQ

A talk given at PgConf NYC 2014 about RabbitMQ and PostgreSQL.

Gavin M. Roy

April 04, 2014
Tweet

More Decks by Gavin M. Roy

Other Decks in Technology

Transcript

  1. About Me VP of Architecture
 AWeber Communications ! Blog: http://gavinroy.com

    Twitter: @Crad GitHub: https://github.com/gmr ! RabbitMQ in Depth (MEAP) http://manning.com/roy
  2. About RabbitMQ • Message Oriented Middleware • Primary identity is

    defined by AMQP 0-9-1 • Supports other protocols with Plugins:
 
 AMQP 1.0, HTTP, MQTT, STOMP, XMPP, UDP, WebStomp, and more • Written in Erlang/OTP, is Open Source (MPL), and is developed/maintained by Pivotal
  3. Why RabbitMQ? • Create loosely coupled architectures • Decouple database

    write operations • Communicate across application platforms • Tap into pre-existing message flow for new purposes • Scale-out clustering for growth, throughput & HA • Federation for WAN latencies & network partitions
  4. Native AMQP Clients C Clojure Cobol Common Lisp Delphi Erlang

    Go Groovy Haskell Java JavaScript .NET OCaml Perl PHP Python Ruby Scala
  5. Who Uses It? Agora Games Chef Google AdMob Instagram MeetMe

    Mercado Libre Mozilla Nasa New York Times NSF Openstack Reddit
  6. AMQ Model Exchange
 defines routing behavior Queue
 stores messages in

    memory and optionally on disk Binding
 defines relationship between exchanges and queues X Queue Binding
  7. Routing Keys • Provided when publishing a message • Compared

    against binding keys by exchanges • Example uses of a Routing Key: • Connotate the type of message • Designate the destination of a message • Categorize the content in the message
  8. Built-In Exchange Types Direct
 String matching on Routing Key Fanout


    No routing key, messages sent to all bound queues Topic
 Pattern matching in Routing Key Headers
 No routing key, string matching in the message headers property
  9. Topic Exchange Binding Keys namespace.delimited.keys #
 Receive all messages namespace.#


    Receive all messages in the namespace namespace.delimited.*
 Receive all namespace.delimited messages namespace.*.keys
 Receive all namespace messages ending in keys
  10. Example Exchange Plugins Consistent Hashing
 Distribute messages via routing key

    hashed value Pulse
 Publish internal RabbitMQ metrics via AMQP Random
 Distribute messages across all bound queues randomly Recent History
 Stores last 20 messages to any bound queue Riak Storage 
 Store received messages in Riak" Script Exchange
 Calls out to external scripts for routing behavior
  11. pg_amqp • PostgreSQL Extension • User-defined functions to publish via

    AMQP • Developed by OmnTI • Invoke from user-defined function • Available on PGXN
  12. Using pg_amqp • Has table of AMQP broker connections •

    Publishes body only AMQP messages • Transactional publishing via amqp.publish • Non-transactional via amqp.autonomous_publish • Needs some love for additional features
  13. PgSQL Listen Exchange • Exchange that issues LISTEN and publishes

    received notifications to bound queues • When routing messages, bindings are checked for the routing key to match the NOTIFY channel • Unobtrusive to PostgreSQL environment, native constructs for publishing messages • Requires PostgreSQL 9.0 and greater • Body only messages*
  14. Sending a Notification psql (9.3.2) Type "help" for help. !

    postgres=# \timing Timing is on. postgres=# NOTIFY channel_name, 
 ‘Test notification'; NOTIFY Time: 0.160 ms
  15. Receiving a Notification psql (9.3.2) Type "help" for help. !

    postgres=# LISTEN channel_name; ! (after NOTIFY issued) ! Asynchronous notification "channel_name" with payload "Test notification" received from server process with PID 16749.
  16. Other ways? via Consumer apps in the language of your

    choice via Apache Flume with RabbitMQ Source, PostgreSQL Sink