Queues With RabbitMQ

Talk at the lovely PHP Minds meetup

Transcript

1. @lornajane QUEUES WITH RABBITMQ Lorna Mitchell, IBM Cloud Data Services September 2016 1
2. @lornajane WHY USE A QUEUE? loose coupling for scalability interaction between technology stacks asynchronous processing event-driven patterns 2
3. @lornajane WHY USE RABBITMQ? Homepage: Open source, uses AMQP standard. Scalable, robust and written in Erlang. Alternatives could be: Gearman Beanstalkd A lot of the theory applies regardless https://www.rabbitmq.com/ 3
4. @lornajane GETTING TO KNOW RABBITMQ 4
5. @lornajane RABBITMQ CHANNELS When we want to talk to RabbitMQ, we establish a channel Channels allow multiple connections between servers over a single TCP connection, but each is isolated from the others. 5
6. @lornajane DECLARING THE QUEUE We declare a queue before we write to it. if it doesn't exist, it is created if it does exist, cool to check if a queue exists but not create it, use passive mode Our example uses the default exchange, more on exchanges in a bit 6
7. @lornajane SIMPLE (BUT REAL) EXAMPLE Use queues to handle incoming webhooks (from GitHub) 7
8. @lornajane PHP AND AMQP Try the php-amqplib/php-amqplib package from Composer 8
9. @lornajane HANDLE WEBHOOKS <?php require "vendor/autoload.php"; $input = file_get_contents("php://input"); $rabbit = new PhpAmqpLib\Connection\AMQPStreamConnection( 'localhost', 5672, 'guest', 'guest'); $channel = $rabbit->channel(); $channel->queue_declare('pushes', false, true, false, false); $message = new PhpAmqpLib\Message\AMQPMessage( $input, ["delivery_mode" => 2]); $channel->basic_publish($message, '', 'pushes'); 9
10. @lornajane RABBITMQ-MANAGEMENT PLUGIN 10
11. @lornajane CONSUMING WITH PHP <?php require "vendor/autoload.php"; $rabbit = new PhpAmqpLib\Connection\AMQPStreamConnection( 'localhost', 5672, 'guest', 'guest'); $channel = $rabbit->channel(); $channel->queue_declare('pushes', false, true, false, false); echo "waiting for jobs ...\n"; 11
12. @lornajane CONSUMING WITH PHP $process = function ($message) { $data = json_decode($message->getBody(), true); // do the actual processing here ... $message->delivery_info['channel'] ->basic_ack($message->delivery_info['delivery_tag']); }; $channel->basic_consume('pushes', '', false, false, false, false, $process); while(count($channel->callbacks)) { $channel->wait(); } 12
13. @lornajane WRITING WORKERS Workers are long-running processes, beware scope issues and memory leaks. There can be many workers attached to one queue, each one independent. They need to be robust and handle failures. 13
14. @lornajane COMPLETED MESSAGES acknowledge when messages are processed successfully can acknowledge failure reject the message optionally: requeue 14
15. @lornajane HANDLING FAILURE: DEAD LETTER QUEUES reject without requeue exceed TTL queue length exceeded 15
16. @lornajane HANDLING FAILURE: RETRIES Implement your own logic to handle retries Create a new message with: all the existing message contents plus some metadata such as retry count or backoff time 16
17. @lornajane RABBITMQ EXCHANGES Exchanges are the routing logic of RabbitMQ Messages go to exchanges and the exchanges put them into the correct queues for storage (our first example used the empty-named default exchange) 17
18. @lornajane TYPES OF EXCHANGE Direct: a given routing key puts messages onto the matching queue(s) Topic: queues are bound by key, and messages are routed to as many queues as their routing key matches Fanout: messages go to all queues bound to this exchange 18
19. @lornajane REAL WORLD EXAMPLE An application needs images resized and sometimes modified 19
20. @lornajane IMAGE RESIZING APPLICATION 20
21. @lornajane DECLARING EXCHANGES AND BINDING QUEUES WITH PHP <?php require "vendor/autoload.php"; $rabbit = new PhpAmqpLib\Connection\AMQPStreamConnection( 'localhost', 5672, 'guest', 'guest'); $channel = $rabbit->channel(); $channel->exchange_declare('images', 'topic', false, true, false); $channel->queue_declare('triagev1', false, true, false, false); $channel->queue_bind('triagev1', 'images', 'image.v1.triage'); 21
22. @lornajane EXAMPLE MESSAGE STRUCTURE { "url": "https://www.gravatar.com/avatar/f6bb323eb6b2ad7f5c15f887", "badge": "pro", "sizes": [ [ 100, 100 ], [ 250, 250 ] ] } 22
23. @lornajane IMAGE RESIZING APPLICATION 23
24. @lornajane IMAGE RESIZING APPLICATION 24
25. @lornajane IMAGE RESIZING APPLICATION 25
26. @lornajane FEEDBACK MECHANISMS Rabbit is fire-and-forget; work is delegated Common pattern: return queue to put updates into for the original producer then to consume. 26
27. @lornajane QUEUES WITH RABBITMQ 27
28. @lornajane QUEUES WITH RABBITMQ Queues are awesome for scalability and robustness RabbitMQ is open source, lightweight and fast Queues help us meet the requirements for modern applications 28
29. @lornajane THANKS! Slides: Code: Managed RabbitMQ: Cool RabbitMQ design tool: Contact: lorna.mitchell@uk.ibm.com @lornajane http://lornajane.net/resources https://github.com/lornajane/queues-with-rabbitmq https://compose.com http://tryrabbitmq.com/ 29