RabbitMQ and Hutch

Talk I gave at LRUG about RabbitMQ, how we're using it at GoCardless and the some tools we built - Hutch and AMQPC

Hutch is an opinionated library for RabbitMQ which makes using it for comms in SOA super simple

Transcript

1. RabbitMQ & Hutch Pete Hamilton - GoCardless
2. About Me • Developer at GoCardless • CompSci Student at ICL • Github: https://github.com/petehamilton • Blog: pete-hamilton.co.uk • Twitter: @peterejhamilton
3. SOA Communication A (very) brief overview Awesome Service
4. SOA Communication A (very) brief overview Awesomer Services
5. SOA Communication A (very) brief overview Awesomer Services
6. SOA Communication A (very) brief overview ? Client - Server? Message Queues?
7. RabbitMQ
8. RabbitMQ Why? • Nice Learning Curve • Reliable • Active Community • Polyglot • Scalable • Proven itself in battle production
9. RabbitMQ Why? • Payments are pretty serious stuff • Async = Potential problems • Network Failure? • Consumer Crashes? • Duplication/redelivery? • RabbitMQ helps with confirms, acknowledgements, mandatory flags
10. RabbitMQ Routing P X C1 C2 Q1 Q2
11. Fanout Exchange RabbitMQ Routing P X C1 C2 Q1 Q2
12. Direct Exchange RabbitMQ Routing P X C1 C2 type=direct firefly star_trek big_bang_theory Q1 Q2
13. Topic Exchange RabbitMQ Routing P X C1 C2 type=topic tv.scifi.* tv.comedy.* C3 tv.# Q1 Q2 Q3 Sample Message Routing Keys: - tv.comedy.big_bang_theory - tv.scifi.firefly - tv.scifi.star_trek
14. RabbitMQ • Very Flexible, but... • Which data structure do you use? • How do you serialise your data? • Which fields do you require in your messages? • How should messages be routed? fan out? direct? topically? • How do new nodes set themselves up? • What form should “Boiler Plate” code take?
15. A system for processing RabbitMQ messages in Ruby
16. • Our Conventions (Warning: Opinions!) • Topic Exchange Routing • Serialisation w/ JSON • Message IDs and Timestamps • Simple to use • Configurable • Specific ruby files or full Rails apps
17. Publisher
18. Consumer
22. https://github.com/gocardless/hutch
23. RabbitMQ Scaling • Good guide to clustering online • Potential for issues with multiple nodes
24. RabbitMQ Scaling • Good guide to clustering online • Potential for issues with multiple nodes S S M C1
25. RabbitMQ Scaling • Good guide to clustering online • Potential for issues with multiple nodes S S M x C1
26. RabbitMQ Scaling • Good guide to clustering online • Potential for issues with multiple nodes S S M x C1
27. RabbitMQ Scaling • Good guide to clustering online • Potential for issues with multiple nodes S S M C1
28. RabbitMQ Scaling • Good guide to clustering online • Potential for issues with multiple nodes S S M C1 x
29. RabbitMQ Scaling • Good guide to clustering online • Potential for issues with multiple nodes M S M C1 x
30. RabbitMQ Scaling • Good guide to clustering online • Potential for issues with multiple nodes • Clients exist for load testing, but... • We wanted ++ parallelism • Built “AMQPC”
31. AMQPC • Written in Go • Producer & Consumer modes • Traffic Modes • One off burst • Intermittent burst • Consistent load
32. AMQPC
33. https://github.com/gocardless/amqpc
34. Useful Links • Hutch - www.github.com/gocardless/hutch • AMQPC - www.github.com/gocardless/amqpc • RabbitMQ - http://www.rabbitmq.com/getstarted.html
35. Questions?