Sprockets

A lighting talk I gave at PhillyPUG on Sprockets

Transcript

1. Sprockets a RabbitMQ Consumer Framework in Python Gavin M. Roy PhillyPUG May 2013
2. What is RabbitMQ? An open source, multi-platform, and scalable message broker
3. The Problem?
4. Application DB In a tightly coupled application, database writes are communicated directly with the database
5. Application C DB In a loosely coupled application, the application publishes a message with the data to RabbitMQ RabbitMQ delivers the message to subscribed consumer applications The consumer application communicates the database writes to the database as it receives each message
6. Event Driven Messaging Architectures
7. P Publishers publish messages C Consumers consume messages
8. Application DB RabbitMQ is now delivering the data to two consumers instead of one C A new consumer can deliver the same data to a 3rd-party cloud based service The original consumer still manages the database writes for the application C Cloud Service Nothing has changed with the application, it is still publishing the data to RabbitMQ the same way
9. More complex
10. Messaging System P Crons C Email Generation C Event Storage C Event Counter P P eCelerity Log Processing Postfix Feedback Loop Processing C Opt-Out Processing P Website Events P Click & Open Tracking C Push Notification Generation C SMS Generation C Bounce Processing C Site Message Generation P SMS Service Events
11. Writing Consumers
12. Consumer Example • Pika Async Example • 148 Lines of Code without docstrings • No actual business logic https://pika.readthedocs.org/en/latest/examples/asynchronous_consumer_example.html
13. Good Consumers?
14. Should be easy to write
15. Should be well behaved
16. • Processing Failures? • Consistent coding style and consumer behaviors? • Performance tweaks and throughput? • Disconnects / Reconnects? • Unhandled exceptions?
17. Should be easy to manage
18. • How many consumers per queue? • Consistent way to configure applications? • Connections, logging, error control, reconnections, etc • Business logic specific configuration • Controlling process? • Stats reporting?
19. Sprockets
20. Born out of rejected https://github.com/gmr/rejected
21. Rejected • Since 2009 • v3.2.11 • Over 50 internal consumer applications • Small userbase • 6 forks, 29 stars on Github • 5,567 downloads from pypi • No meaningful documentation
22. It’s like rejected... but easier.
23. Consumer Example
24. RPC Example
25. It’s like rejected... but with documentation https://sprockets.rtfd.org
26. Sprockets • Is a consumer framework and runtime • Is easy to use • Has consumer class mixins to speed development • Has dynamic QoS for throughput optimization • Has a web-based management UI & API https://github.com/gmr/sprockets