Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Queues with RabbitMQ

D33d8bdd9096c80b8d1acca8d28410b5?s=47 Lorna Mitchell
September 30, 2016

Queues with RabbitMQ

Talk delivered at the PHPNW conference in Manchester

D33d8bdd9096c80b8d1acca8d28410b5?s=128

Lorna Mitchell

September 30, 2016
Tweet

Transcript

  1. @lornajane QUEUES WITH RABBITMQ Lorna Mitchell, IBM Cloud Data Services

    PHPNW, September 2016 (slides are available: ) http://lornajane.net/resources 1
  2. @lornajane WHY USE A QUEUE? loose coupling for scalability interaction

    between technology stacks asynchronous processing event-driven patterns 2
  3. @lornajane WHY USE RABBITMQ? Homepage: Open source, uses AMQP standard.

    Scalable, robust and written in Erlang. Alternatives could be: Gearman Beanstalkd A lot of the theory applies regardless https://www.rabbitmq.com/ 3
  4. @lornajane GETTING TO KNOW RABBITMQ 4

  5. @lornajane RABBITMQ CHANNELS When we want to talk to RabbitMQ,

    we establish a channel Channels allow multiple connections between servers over a single TCP connection, but each is isolated from the others. 5
  6. @lornajane DECLARING THE QUEUE We declare a queue before we

    write to it. if it doesn't exist, it is created if it does exist, cool to check if a queue exists but not create it, use passive mode Our example uses the default exchange, more on exchanges in a bit 6
  7. @lornajane SIMPLE (BUT REAL) EXAMPLE Use queues to handle incoming

    webhooks 7
  8. @lornajane PHP AND AMQP Try the php-amqplib/php-amqplib package from Composer

    8
  9. @lornajane HANDLE WEBHOOKS <?php require "vendor/autoload.php"; $input = file_get_contents("php://input"); $rabbit

    = new PhpAmqpLib\Connection\AMQPStreamConnection( 'localhost', 5672, 'guest', 'guest'); $channel = $rabbit->channel(); $channel->queue_declare('pushes', false, true, false, false); $message = new PhpAmqpLib\Message\AMQPMessage( $input, ["delivery_mode" => 2]); $channel->basic_publish($message, '', 'pushes'); 9
  10. @lornajane RABBITMQ-MANAGEMENT PLUGIN 10

  11. @lornajane CONSUMING WITH PHP <?php require "vendor/autoload.php"; $rabbit = new

    PhpAmqpLib\Connection\AMQPStreamConnection( 'localhost', 5672, 'guest', 'guest'); $channel = $rabbit->channel(); $channel->queue_declare('pushes', false, true, false, false); echo "waiting for jobs ...\n"; 11
  12. @lornajane CONSUMING WITH PHP $process = function ($message) { $data

    = json_decode($message->getBody(), true); // do the actual processing here ... $message->delivery_info['channel'] ->basic_ack($message->delivery_info['delivery_tag']); }; $channel->basic_consume('pushes', '', false, false, false, false, $process); while(count($channel->callbacks)) { $channel->wait(); } 12
  13. @lornajane WRITING WORKERS Workers are long-running processes, beware scope issues

    and memory leaks. There can be many workers attached to one queue, each one independent. They need to be robust and handle failures. 13
  14. @lornajane COMPLETED MESSAGES acknowledge when messages are processed successfully can

    acknowledge failure reject the message optionally: requeue 14
  15. @lornajane HANDLING FAILURE: DEAD LETTER QUEUES reject without requeue exceed

    TTL queue length exceeded 15
  16. @lornajane HANDLING FAILURE: RETRIES Implement your own logic to handle

    retries Create a new message with: all the existing message contents plus some metadata such as retry count or backoff time 16
  17. @lornajane RABBITMQ EXCHANGES Exchanges are the routing logic of RabbitMQ

    Messages go to exchanges and the exchanges put them into the correct queues for storage (our first example used the empty-named default exchange) 17
  18. @lornajane TYPES OF EXCHANGE Direct: a given routing key puts

    messages onto the matching queue(s) Topic: queues are bound by key, and messages are routed to as many queues as their routing key matches Fanout: messages go to all queues bound to this exchange 18
  19. @lornajane REAL WORLD EXAMPLE An application needs images resized and

    sometimes modified 19
  20. @lornajane IMAGE RESIZING APPLICATION 20

  21. @lornajane DECLARE EXCHANGE, BIND QUEUE IN PHP <?php require "vendor/autoload.php";

    $rabbit = new PhpAmqpLib\Connection\AMQPStreamConnection( 'localhost', 5672, 'guest', 'guest'); $channel = $rabbit->channel(); $channel->exchange_declare('images', 'topic', false, true, false); $channel->queue_declare('triagev1', false, true, false, false); $channel->queue_bind('triagev1', 'images', 'image.v1.triage'); 21
  22. @lornajane EXAMPLE MESSAGE STRUCTURE { "url": "https://www.gravatar.com/avatar/f6bb323eb6b2ad7f5c15f887", "badge": "pro", "sizes":

    [ [ 100, 100 ], [ 250, 250 ] ] } 22
  23. @lornajane IMAGE RESIZING APPLICATION 23

  24. @lornajane IMAGE RESIZING APPLICATION 24

  25. @lornajane IMAGE RESIZING APPLICATION 25

  26. @lornajane IMAGE RESIZING APPLICATION 26

  27. @lornajane FEEDBACK MECHANISMS Rabbit is fire-and-forget; work is delegated Common

    pattern: return queue to put updates into for the original producer then to consume. 27
  28. @lornajane QUEUES WITH RABBITMQ 28

  29. @lornajane QUEUES WITH RABBITMQ Queues are awesome for scalability and

    robustness RabbitMQ is open source, lightweight and fast Queues help us meet the requirements for modern applications 29
  30. @lornajane THANKS! Feedback: Code: Managed RabbitMQ: Contact: lorna.mitchell@uk.ibm.com @lornajane https://joind.in/talk/4d555

    https://github.com/lornajane/queues-with- rabbitmq https://compose.com 30