Sprockets
a RabbitMQ Consumer Framework in Python
Gavin M. Roy
PhillyPUG May 2013
Slide 2
Slide 2 text
What is RabbitMQ?
An open source, multi-platform, and
scalable message broker
Slide 3
Slide 3 text
The Problem?
Slide 4
Slide 4 text
Application DB
In a tightly coupled application,
database writes are communicated
directly with the database
Slide 5
Slide 5 text
Application C DB
In a loosely coupled application, the application
publishes a message with the data to RabbitMQ
RabbitMQ delivers the message to
subscribed consumer applications
The consumer application communicates
the database writes to the database as it
receives each message
Slide 6
Slide 6 text
Event Driven
Messaging
Architectures
Slide 7
Slide 7 text
P
Publishers publish messages
C
Consumers consume messages
Slide 8
Slide 8 text
Application
DB
RabbitMQ is now delivering the data
to two consumers instead of one
C
A new consumer can deliver the same
data to a 3rd-party cloud based service
The original consumer still manages the
database writes for the application
C
Cloud
Service
Nothing has changed with the
application, it is still publishing the
data to RabbitMQ the same way
Slide 9
Slide 9 text
More complex
Slide 10
Slide 10 text
Messaging System
P
Crons
C
Email
Generation
C
Event
Storage
C
Event
Counter
P
P
eCelerity Log Processing
Postfix Feedback Loop
Processing
C
Opt-Out
Processing
P
Website
Events
P
Click & Open
Tracking
C
Push Notification
Generation
C
SMS
Generation
C
Bounce
Processing
C
Site Message
Generation
P
SMS Service
Events
Slide 11
Slide 11 text
Writing Consumers
Slide 12
Slide 12 text
Consumer Example
• Pika Async Example
• 148 Lines of Code
without docstrings
• No actual business
logic
https://pika.readthedocs.org/en/latest/examples/asynchronous_consumer_example.html
• How many consumers per queue?
• Consistent way to configure applications?
• Connections, logging, error control,
reconnections, etc
• Business logic specific configuration
• Controlling process?
• Stats reporting?
Slide 19
Slide 19 text
Sprockets
Slide 20
Slide 20 text
Born out of
rejected
https://github.com/gmr/rejected
Slide 21
Slide 21 text
Rejected
• Since 2009
• v3.2.11
• Over 50 internal consumer applications
• Small userbase
• 6 forks, 29 stars on Github
• 5,567 downloads from pypi
• No meaningful documentation
Slide 22
Slide 22 text
It’s like rejected...
but easier.
Slide 23
Slide 23 text
Consumer Example
Slide 24
Slide 24 text
RPC Example
Slide 25
Slide 25 text
It’s like rejected...
but with documentation
https://sprockets.rtfd.org
Slide 26
Slide 26 text
Sprockets
• Is a consumer framework and runtime
• Is easy to use
• Has consumer class mixins to speed
development
• Has dynamic QoS for throughput
optimization
• Has a web-based management UI & API
https://github.com/gmr/sprockets