Queues with RabbitMQ

@lornajane QUEUES WITH RABBITMQ Lorna Mitchell, IBM Cloud Data Services
PHPNW, September 2016 (slides are available: ) http://lornajane.net/resources 1

@lornajane WHY USE A QUEUE? loose coupling for scalability interaction
between technology stacks asynchronous processing event-driven patterns 2

@lornajane WHY USE RABBITMQ? Homepage: Open source, uses AMQP standard.
Scalable, robust and written in Erlang. Alternatives could be: Gearman Beanstalkd A lot of the theory applies regardless https://www.rabbitmq.com/ 3

@lornajane GETTING TO KNOW RABBITMQ 4

@lornajane RABBITMQ CHANNELS When we want to talk to RabbitMQ,
we establish a channel Channels allow multiple connections between servers over a single TCP connection, but each is isolated from the others. 5

@lornajane DECLARING THE QUEUE We declare a queue before we
write to it. if it doesn't exist, it is created if it does exist, cool to check if a queue exists but not create it, use passive mode Our example uses the default exchange, more on exchanges in a bit 6

@lornajane SIMPLE (BUT REAL) EXAMPLE Use queues to handle incoming
webhooks 7

@lornajane PHP AND AMQP Try the php-amqplib/php-amqplib package from Composer
8

@lornajane HANDLE WEBHOOKS <?php require "vendor/autoload.php"; $input = file_get_contents("php://input"); $rabbit
= new PhpAmqpLib\Connection\AMQPStreamConnection( 'localhost', 5672, 'guest', 'guest'); $channel = $rabbit->channel(); $channel->queue_declare('pushes', false, true, false, false); $message = new PhpAmqpLib\Message\AMQPMessage( $input, ["delivery_mode" => 2]); $channel->basic_publish($message, '', 'pushes'); 9

@lornajane RABBITMQ-MANAGEMENT PLUGIN 10

@lornajane CONSUMING WITH PHP <?php require "vendor/autoload.php"; $rabbit = new
PhpAmqpLib\Connection\AMQPStreamConnection( 'localhost', 5672, 'guest', 'guest'); $channel = $rabbit->channel(); $channel->queue_declare('pushes', false, true, false, false); echo "waiting for jobs ...\n"; 11

@lornajane CONSUMING WITH PHP $process = function ($message) { $data
= json_decode($message->getBody(), true); // do the actual processing here ... $message->delivery_info['channel'] ->basic_ack($message->delivery_info['delivery_tag']); }; $channel->basic_consume('pushes', '', false, false, false, false, $process); while(count($channel->callbacks)) { $channel->wait(); } 12

@lornajane WRITING WORKERS Workers are long-running processes, beware scope issues
and memory leaks. There can be many workers attached to one queue, each one independent. They need to be robust and handle failures. 13

@lornajane COMPLETED MESSAGES acknowledge when messages are processed successfully can
acknowledge failure reject the message optionally: requeue 14

@lornajane HANDLING FAILURE: DEAD LETTER QUEUES reject without requeue exceed
TTL queue length exceeded 15

@lornajane HANDLING FAILURE: RETRIES Implement your own logic to handle
retries Create a new message with: all the existing message contents plus some metadata such as retry count or backoff time 16

@lornajane RABBITMQ EXCHANGES Exchanges are the routing logic of RabbitMQ
Messages go to exchanges and the exchanges put them into the correct queues for storage (our first example used the empty-named default exchange) 17

@lornajane TYPES OF EXCHANGE Direct: a given routing key puts
messages onto the matching queue(s) Topic: queues are bound by key, and messages are routed to as many queues as their routing key matches Fanout: messages go to all queues bound to this exchange 18

@lornajane REAL WORLD EXAMPLE An application needs images resized and
sometimes modified 19

@lornajane IMAGE RESIZING APPLICATION 20

@lornajane DECLARE EXCHANGE, BIND QUEUE IN PHP <?php require "vendor/autoload.php";
$rabbit = new PhpAmqpLib\Connection\AMQPStreamConnection( 'localhost', 5672, 'guest', 'guest'); $channel = $rabbit->channel(); $channel->exchange_declare('images', 'topic', false, true, false); $channel->queue_declare('triagev1', false, true, false, false); $channel->queue_bind('triagev1', 'images', 'image.v1.triage'); 21

@lornajane EXAMPLE MESSAGE STRUCTURE { "url": "https://www.gravatar.com/avatar/f6bb323eb6b2ad7f5c15f887", "badge": "pro", "sizes":
[ [ 100, 100 ], [ 250, 250 ] ] } 22

@lornajane FEEDBACK MECHANISMS Rabbit is fire-and-forget; work is delegated Common
pattern: return queue to put updates into for the original producer then to consume. 27

@lornajane QUEUES WITH RABBITMQ 28

@lornajane QUEUES WITH RABBITMQ Queues are awesome for scalability and
robustness RabbitMQ is open source, lightweight and fast Queues help us meet the requirements for modern applications 29

@lornajane THANKS! Feedback: Code: Managed RabbitMQ: Contact: [email protected] @lornajane https://joind.in/talk/4d555
https://github.com/lornajane/queues-with- rabbitmq https://compose.com 30

Queues with RabbitMQ

Queues with RabbitMQ

Lorna Mitchell

More Decks by Lorna Mitchell

Other Decks in Technology

Featured

Transcript

@lornajane QUEUES WITH RABBITMQ Lorna Mitchell, IBM Cloud Data Services

@lornajane WHY USE A QUEUE? loose coupling for scalability interaction

@lornajane WHY USE RABBITMQ? Homepage: Open source, uses AMQP standard.

@lornajane GETTING TO KNOW RABBITMQ 4

@lornajane RABBITMQ CHANNELS When we want to talk to RabbitMQ,

@lornajane DECLARING THE QUEUE We declare a queue before we

@lornajane SIMPLE (BUT REAL) EXAMPLE Use queues to handle incoming

@lornajane PHP AND AMQP Try the php-amqplib/php-amqplib package from Composer

@lornajane HANDLE WEBHOOKS <?php require "vendor/autoload.php"; $input = file_get_contents("php://input"); $rabbit

@lornajane RABBITMQ-MANAGEMENT PLUGIN 10

@lornajane CONSUMING WITH PHP <?php require "vendor/autoload.php"; $rabbit = new

@lornajane CONSUMING WITH PHP $process = function ($message) { $data

@lornajane WRITING WORKERS Workers are long-running processes, beware scope issues

@lornajane COMPLETED MESSAGES acknowledge when messages are processed successfully can

@lornajane HANDLING FAILURE: DEAD LETTER QUEUES reject without requeue exceed

@lornajane HANDLING FAILURE: RETRIES Implement your own logic to handle

@lornajane RABBITMQ EXCHANGES Exchanges are the routing logic of RabbitMQ

@lornajane TYPES OF EXCHANGE Direct: a given routing key puts

@lornajane REAL WORLD EXAMPLE An application needs images resized and

@lornajane IMAGE RESIZING APPLICATION 20

@lornajane DECLARE EXCHANGE, BIND QUEUE IN PHP <?php require "vendor/autoload.php";

@lornajane EXAMPLE MESSAGE STRUCTURE { "url": "https://www.gravatar.com/avatar/f6bb323eb6b2ad7f5c15f887", "badge": "pro", "sizes":

@lornajane IMAGE RESIZING APPLICATION 23

@lornajane IMAGE RESIZING APPLICATION 24

@lornajane IMAGE RESIZING APPLICATION 25

@lornajane IMAGE RESIZING APPLICATION 26

@lornajane FEEDBACK MECHANISMS Rabbit is fire-and-forget; work is delegated Common

@lornajane QUEUES WITH RABBITMQ 28

@lornajane QUEUES WITH RABBITMQ Queues are awesome for scalability and

@lornajane THANKS! Feedback: Code: Managed RabbitMQ: Contact: [email protected] @lornajane https://joind.in/talk/4d555