RabbitMQ, Event-Driven Architectures, and Data Warehousing

Slides from my Data Philly talk from March 19th

Transcript

1. Data Philly, March 2015 RabbitMQ, 
 Event-Driven Architectures, and Data Warehousing Gavin M. Roy VP of Architecture AWeber Communications Twitter: @Crad
2. About Me VP of Architecture
 AWeber Communications Blame me for pika, rabbitpy, pamqp, and a handful of RabbitMQ plugins Blog: https://gavinroy.com Github: https://github.com/gmr Book: http://manning.com/roy
3. About RabbitMQ ❖ Open Source (MPL) ❖ Written in Erlang/OTP ❖ Developed/Maintained by Pivotal ❖ Multi-Protocol
 AMQP 0-9-1 & 1.0, MQTT, STOMP, XMPP, HTTP, Web-STOMP & More ❖ Roots in AMQP 0-8/0-9-1
4. Why Use RabbitMQ? ❖ Create loosely-coupled applications ❖ Communicate across applications or platforms ❖ Tap into pre-existing message flows for new purposes ❖ Scale-out clustering for growth, throughput, and HA ❖ Federation for WAN latencies and network partitions ❖ Extensible plugin-in architecture
5. Who Uses It? Agora Games Chef Google AdMob Instagram Openstack Mercado Libre Mozilla NASA New York Times National Science Foundation Ocean Observatory Initiative Rapportive Reddit Soundcloud (and many more)
6. RabbitMQ @ the OOI ❖ Network of over 700 sensors deployed off 6 coastlines ❖ RabbitMQ is the centerpiece of their Integrated Observatory Network ❖ Data is made available through “virtual observatories” in CSV and MATLAB formats For more information visit http://goo.gl/F48Ei
7. Advanced Message Queueing Protocol ❖ Open Standard ❖ Platform and Vendor Neutral ❖ Multiple Versions
8. AMQP Timeline 2003 2005 2006 2008 2011 Initial Specification Working Group formed AMQP 0-8 AMQP 0-9-1 OASIS AMQP 1.0
9. AMQP Working Group ❖ JPMorgan Chase ❖ Cisco Systems ❖ IONA Technologies ❖ iMatrix ❖ RedHat ❖ TWIST ❖ Bank of America ❖ Barclays ❖ Credit Suisse ❖ Deutsche Börse Systems ❖ Goldman Sachs ❖ HCL Technologies ❖ Progress Software ❖ IIT Software ❖ INETCO Systems Ltd. ❖ Informatica Corporation ❖ Microsoft Corporation ❖ my-Channels ❖ Novell ❖ Solace Systems ❖ Tervala, Inc. ❖ VMWare ❖ WSO2
10. Competing AMQP Standards ≠ 0-8 & 0-9-1 1.0
11. AMQP 0-9-1 ❖ Currently has wider support than AMQP 1.0 ❖ Multiple Broker Implementations: 
 RabbitMQ, Apache Qpid, and SwiftMQ to name a few ❖ Specifies a Model and Protocol
12. Native AMQP 0-8/0-9-1 Clients C Clojure Cobol Common Lisp Delphi Erlang Go Groovy Haskell Java JavaScript .NET OCaml Perl PHP Python Ruby Scala
13. ❖ Exchange
 Receives and route messages ❖ Message Queue
 Stores messages until they can be
 consumed ❖ Binding
 Defines the relationship between an Exchange and Queue and provides routing criteria Advanced Message Queueing Model X Exchange Binding Message Queue
14. Routing Keys ❖ Provided when publishing a message ❖ Compared against binding keys by exchanges ❖ Ideally provide context to the message: ❖ Connote the type of the message ❖ Categorize the content in the message ❖ Specify the type of consumer that should receive it
15. Advanced Message Queueing Protocol ❖ Compact, binary frame format wire protocol ❖ Bi-directional RPC ❖ Commands consist of Classes and Methods: ❖ Example Request:
 
 Queue.Declare(name=“foo”) ❖ Response:
 
 Queue.DeclareOk(messages=0, consumers=0)
16. Common AMQP Terms ❖ Broker
 A server that implements AMQP ❖ Producer or Publisher
 A client application that sends messages to a broker ❖ Consumer
 A client application that reads messages from a queue
17. Publishers and Consumers C P
18. Multiple Publishers C P P P
19. Multiple Consumers C C C P
20. Event-Driven Architectures and Data Warehousing
21. Tightly-Coupled Applications Application DB
22. Decoupling Database Writes Application C DB
23. The Event Firehose ❖ Evolution of decoupled applications to an event-driven architecture ❖ Events should: ❖ Be semantically derived ❖ Contain all of the data ❖ Contain context
24. Multi-Purposed Event Stream Application DB Cloud Stats / Monitoring / etc
25. Apache Flume "Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data."
26. Flume Concepts HDFS Source Sink Channel
27. Connecting RabbitMQ to Flume ❖ rabbitmq-flume-plugin - Source & Sink
 https://github.com/aweber/rabbitmq-flume-plugin ❖ flume-ng-rabbitmq - Source & Sink
 https://github.com/jcustenborder/flume-ng-rabbitmq ❖ flume-rabbitmq-sink - Sink
 https://github.com/kenshoo/flume-rabbitmq-sink
28. Using Flume to Stream to HDFS Application HDFS Flume DB C
29. RabbitMQ & Flume at AWeber ❖ Data is linefeed delimited JSON ❖ Uploaded to S3 ❖ S3 support is sketchy: uploads fail, can’t retry ❖ Deprecating Flume ❖ Moving towards Avro datums ❖ Working on a new app: estuary
30. Questions? 42% discount @ manning.com Use code ctwdataphiug