Slide 1

Slide 1 text

Workers, Queues and Data Lorna Mitchell, IBM OSCON 2017

Slide 2

Slide 2 text

Introducing Queues Use queues to: • asynchronously process tasks in your application • e.g. sending email, processing uploads • provide loose coupling points • e.g. delegate heavy tasks, split ownership/technology • enable parts of the system to scale appropriately • e.g. an event in one system causes a thousand other actions @lornajane

Slide 3

Slide 3 text

Queue Tools A selection of queue-ish tools • RabbitMQ* http://www.rabbitmq.com/ • Gearman http://gearman.org/ • Beanstalkd http://kr.github.io/beanstalkd/ • Kafka https://kafka.apache.org/ • Redis https://redis.io/ * RabbitMQ is used in today's examples @lornajane

Slide 4

Slide 4 text

See also: microservices, serverless @lornajane

Slide 5

Slide 5 text

Example: Webhooks Queue gets messages containing: • data • endpoints App sends webhooks @lornajane

Slide 6

Slide 6 text

Framing The Problem @lornajane

Slide 7

Slide 7 text

Framing The Problem • What needs to be done? • Is that one task or a series of steps? Turn the steps into a flow diagram @lornajane

Slide 8

Slide 8 text

Framing The Problem @lornajane

Slide 9

Slide 9 text

Data Hygiene "A swarm of small standalone apps all talking to the same database is simply an HTTP-enabled monolith" -- Lorna Mitchell, 2016 @lornajane

Slide 10

Slide 10 text

Data Hygiene • Where does the data go from? Where does the data go to? • How should that data be structured/formatted? • Which component "owns" access to each data store? Add data movement to diagram. Include system inputs/outputs @lornajane

Slide 11

Slide 11 text

Framing the Problem @lornajane

Slide 12

Slide 12 text

Framing the Problem @lornajane

Slide 13

Slide 13 text

Now really draw the diagram @lornajane

Slide 14

Slide 14 text

Now really draw the diagram (Then ask another human if the naming makes sense) @lornajane

Slide 15

Slide 15 text

Designing Queues @lornajane

Slide 16

Slide 16 text

Designing Queues Some vocabulary • exchange: where to send the messages to • queue: where messages wait to be processed • message: the data to be processed • binding key: rules for which messages go into this queue • routing key: message route information @lornajane

Slide 17

Slide 17 text

Designing Queues Queues can: • have wildcards in binding keys • be durable (messages have their own durability) • have a maximum length • be configured with a "dead letter exchange" If producer needs feedback, create another queue to put status updates into @lornajane

Slide 18

Slide 18 text

Designing Queues @lornajane

Slide 19

Slide 19 text

Designing Queues @lornajane

Slide 20

Slide 20 text

Expect Failure Sometimes, things go wrong! 1. Decide if you care 2. React appropriately Blog post about failure (I'm an expert): http://lrnja.net/rabbitfail @lornajane

Slide 21

Slide 21 text

Creating Workers @lornajane

Slide 22

Slide 22 text

Creating Workers Workers are disposable! • if things go wrong, exit • separate tool to monitor/restart as needed • beware long-running process hazards • everything processed "at least once" (but maybe more than once, and in any order...) @lornajane

Slide 23

Slide 23 text

Independent Workers @lornajane

Slide 24

Slide 24 text

Independent Workers For best results: • awesome, aggregated logging • monitoring: queue size, worker uptime, processing time • miniminum viable dependencies @lornajane

Slide 25

Slide 25 text

Deploying Workers As many workers as you need, per task - and this may change Workers can be multi-skilled, one worker for many occasional tasks If in design doubt: more queues @lornajane

Slide 26

Slide 26 text

Workers, Queues, Data - and your App @lornajane

Slide 27

Slide 27 text

Queues in Your Applications Queues are useful in either new or existing systems • they introduce resilience • enable migrations/refactoring • improve performance and scalability • (see also: server costs) @lornajane

Slide 28

Slide 28 text

Resources • Please leave me some feedback! • Try RabbitMQ on Bluemix (free trial) http://bluemix.com • Email me: [email protected] • Website: https://lornajane.net Further reading: • Failure handling: http://lrnja.net/rabbitfail • Webhooks example: http://lrnja.net/2iu4iBQ @lornajane