Workers, Queues and Data

Designing data flows, queue architectures and reliable worker scripts, using RabbitMQ as the example

Transcript

1. Workers, Queues and Data Lorna Mitchell, IBM OSCON 2017
2. Introducing Queues Use queues to: • asynchronously process tasks in your application • e.g. sending email, processing uploads • provide loose coupling points • e.g. delegate heavy tasks, split ownership/technology • enable parts of the system to scale appropriately • e.g. an event in one system causes a thousand other actions @lornajane
3. Queue Tools A selection of queue-ish tools • RabbitMQ* http://www.rabbitmq.com/ • Gearman http://gearman.org/ • Beanstalkd http://kr.github.io/beanstalkd/ • Kafka https://kafka.apache.org/ • Redis https://redis.io/ * RabbitMQ is used in today's examples @lornajane
4. See also: microservices, serverless @lornajane
5. Example: Webhooks Queue gets messages containing: • data • endpoints App sends webhooks @lornajane
6. Framing The Problem @lornajane
7. Framing The Problem • What needs to be done? • Is that one task or a series of steps? Turn the steps into a flow diagram @lornajane
8. Framing The Problem @lornajane
9. Data Hygiene "A swarm of small standalone apps all talking to the same database is simply an HTTP-enabled monolith" -- Lorna Mitchell, 2016 @lornajane
10. Data Hygiene • Where does the data go from? Where does the data go to? • How should that data be structured/formatted? • Which component "owns" access to each data store? Add data movement to diagram. Include system inputs/outputs @lornajane
11. Framing the Problem @lornajane
12. Framing the Problem @lornajane
13. Now really draw the diagram @lornajane
14. Now really draw the diagram (Then ask another human if the naming makes sense) @lornajane
15. Designing Queues @lornajane
16. Designing Queues Some vocabulary • exchange: where to send the messages to • queue: where messages wait to be processed • message: the data to be processed • binding key: rules for which messages go into this queue • routing key: message route information @lornajane
17. Designing Queues Queues can: • have wildcards in binding keys • be durable (messages have their own durability) • have a maximum length • be configured with a "dead letter exchange" If producer needs feedback, create another queue to put status updates into @lornajane
18. Designing Queues @lornajane
19. Designing Queues @lornajane
20. Expect Failure Sometimes, things go wrong! 1. Decide if you care 2. React appropriately Blog post about failure (I'm an expert): http://lrnja.net/rabbitfail @lornajane
21. Creating Workers @lornajane
22. Creating Workers Workers are disposable! • if things go wrong, exit • separate tool to monitor/restart as needed • beware long-running process hazards • everything processed "at least once" (but maybe more than once, and in any order...) @lornajane
23. Independent Workers @lornajane
24. Independent Workers For best results: • awesome, aggregated logging • monitoring: queue size, worker uptime, processing time • miniminum viable dependencies @lornajane
25. Deploying Workers As many workers as you need, per task - and this may change Workers can be multi-skilled, one worker for many occasional tasks If in design doubt: more queues @lornajane
26. Workers, Queues, Data - and your App @lornajane
27. Queues in Your Applications Queues are useful in either new or existing systems • they introduce resilience • enable migrations/refactoring • improve performance and scalability • (see also: server costs) @lornajane
28. Resources • Please leave me some feedback! • Try RabbitMQ on Bluemix (free trial) http://bluemix.com • Email me: lorna.mitchell@uk.ibm.com • Website: https://lornajane.net Further reading: • Failure handling: http://lrnja.net/rabbitfail • Webhooks example: http://lrnja.net/2iu4iBQ @lornajane