RabbitMQ and Node.js

RabbitMQ and Node.js

1. Event processing with Node.js and RabbitMQ by Zlatko Duric
2. Disclaimer (AKA about me) • Experience – Node.js? Lots – NoSQL? Lots. – Messaging? Little • Backend Developer at ThinxNet.com (http://thinxnet.com/#jobs)
3. Connected cars? • Hardware device • Various sensors – lots of data • “Cloud” processing
4. Ready to serve. How? • Event collection and processing • Webapp, mobile app • Backoffice and support apps, notification services etc. • Reporting, third party software integration etc • REST approach: – Lots of requests – Parsing of data is blocking – Avoid choking devices – Communication overhead – Error management
5. What service? Microservice! • Redis PUBSUB • ZeroMQ, MQTT, AMQP • RabbitMQ, ActiveMQ
6. About RabbitMQ • Messaging broker based on AMQP protocol • Platform/language independent • Extensible (Erlang) • Persistent message queues • Node.js drivers (node-amqp, rabbit.js...)
7. Hello world
8. Exchanges and topics • Publisher and subscriber. Simple as that. • Get message, ack or reject (nack) it. • Subscribers listen on a queue • Publishers talk to an exchange • Exchange routes to queues based on topics
9. Example
10. Channels and connections • 1 node process = 1 connection • Many channels, share same connection
11. PUBSUB • Simple PUBSUB, as used elsewhere • All listeners get the same message • Suboptimal
12. Example
13. PUSH-PULL/WORKER • Round-robin distribution • Worker socket.ack() is explicit (rabbit.js driver) • Prefetch sets the number of msgs received • Problem: concurrent/sequential modification?
14. Example
15. Content-based routing • Erlang extensions? Not standard. • App-side: – Keep internal mini-queue – Do not ack a msg if it's not first on local queue – Hold internal messages based on internal “topic”, ie “user.<ID>.save” - hold msgs for that user – Item.update() instead of item.save() • Gets nasty fast, should minimize these scenarios
16. Example
17. REST API with MQ • REQUEST/REPLY for immediate responses • Other patterns for direct access – REST API just forwards the message – replies will go by other route • Pub/sub is easy!
18. Example
19. Error handling • Uncaught exceptions are covered • Retry logic is simplified – push incomplete messages to a 'fixer' queue, requeue failed jobs or when no resources • Dedicated error handling
20. Example
21. Error handling • Uncaught exceptions are covered • Retry logic is simplified – push incomplete messages to a 'fixer' queue, requeue failed jobs or when no resources • Dedicated error handling
22. Testing • No good known mocks, please help • RabbitMQ always on on Jenkins machine • Test handlers directly
23. Buzzwords index • Cloud • Connected car • Microservice • Patterns :)
24. Info sources • https://www.rabbitmq.com/ • http://www.squaremobius.net/rabbit.js/ • http://www.manning.com/videla/
25. Note to self • Thank the audience for patience • Invite audience for a discussion and questions