Queues with RabbitMQ

Talk delivered at the PHPNW conference in Manchester

Transcript

1. @lornajane
   QUEUES WITH RABBITMQ
   Lorna Mitchell, IBM Cloud Data Services
   PHPNW, September 2016
   (slides are available: )
   http://lornajane.net/resources
   1
   

   View Slide

2. @lornajane
   WHY USE A QUEUE?
   loose coupling for scalability
   interaction between technology stacks
   asynchronous processing
   event-driven patterns
   2
   

   View Slide

3. @lornajane
   WHY USE RABBITMQ?
   Homepage:
   Open source, uses AMQP standard. Scalable, robust
   and written in Erlang.
   Alternatives could be:
   Gearman
   Beanstalkd
   A lot of the theory applies regardless
   https://www.rabbitmq.com/
   3
   

   View Slide

4. @lornajane
   GETTING TO KNOW
   RABBITMQ
   4
   

   View Slide

5. @lornajane
   RABBITMQ CHANNELS
   When we want to talk to RabbitMQ, we establish a
   channel
   Channels allow multiple connections between
   servers over a single TCP connection, but each is
   isolated from the others.
   5
   

   View Slide

6. @lornajane
   DECLARING THE QUEUE
   We declare a queue before we write to it.
   if it doesn't exist, it is created
   if it does exist, cool
   to check if a queue exists but not create it, use
   passive mode
   Our example uses the default exchange, more on
   exchanges in a bit
   6
   

   View Slide

7. @lornajane
   SIMPLE (BUT REAL)
   EXAMPLE
   Use queues to handle incoming webhooks
   7
   

   View Slide

8. @lornajane
   PHP AND AMQP
   Try the php-amqplib/php-amqplib package from
   Composer
   8
   

   View Slide

9. @lornajane
   HANDLE WEBHOOKS
   require "vendor/autoload.php";
   $input = file_get_contents("php://input");
   $rabbit = new PhpAmqpLib\Connection\AMQPStreamConnection(
   'localhost', 5672, 'guest', 'guest');
   $channel = $rabbit->channel();
   $channel->queue_declare('pushes', false, true, false, false);
   $message = new PhpAmqpLib\Message\AMQPMessage(
   $input, ["delivery_mode" => 2]);
   $channel->basic_publish($message, '', 'pushes');
   9
   

   View Slide

10. @lornajane
    RABBITMQ-MANAGEMENT PLUGIN
    10
    

    View Slide

11. @lornajane
    CONSUMING WITH PHP
    require "vendor/autoload.php";
    $rabbit = new PhpAmqpLib\Connection\AMQPStreamConnection(
    'localhost', 5672, 'guest', 'guest');
    $channel = $rabbit->channel();
    $channel->queue_declare('pushes', false, true, false, false);
    echo "waiting for jobs ...\n";
    11
    

    View Slide

12. @lornajane
    CONSUMING WITH PHP
    $process = function ($message) {
    $data = json_decode($message->getBody(), true);
    // do the actual processing here ...
    $message->delivery_info['channel']
    ->basic_ack($message->delivery_info['delivery_tag']);
    };
    $channel->basic_consume('pushes', '',
    false, false, false, false, $process);
    while(count($channel->callbacks)) {
    $channel->wait();
    }
    12
    

    View Slide

13. @lornajane
    WRITING WORKERS
    Workers are long-running processes, beware scope
    issues and memory leaks.
    There can be many workers attached to one queue,
    each one independent.
    They need to be robust and handle failures.
    13
    

    View Slide

14. @lornajane
    COMPLETED MESSAGES
    acknowledge when messages are processed
    successfully
    can acknowledge failure
    reject the message
    optionally: requeue
    14
    

    View Slide

15. @lornajane
    HANDLING FAILURE: DEAD LETTER QUEUES
    reject without requeue
    exceed TTL
    queue length exceeded
    15
    

    View Slide

16. @lornajane
    HANDLING FAILURE: RETRIES
    Implement your own logic to handle retries
    Create a new message with:
    all the existing message contents
    plus some metadata such as retry count or
    backoff time
    16
    

    View Slide

17. @lornajane
    RABBITMQ EXCHANGES
    Exchanges are the routing logic of RabbitMQ
    Messages go to exchanges and the exchanges put
    them into the correct queues for storage
    (our first example used the empty-named default
    exchange)
    17
    

    View Slide

18. @lornajane
    TYPES OF EXCHANGE
    Direct: a given routing key puts messages onto the
    matching queue(s)
    Topic: queues are bound by key, and messages are
    routed to as many queues as their routing key
    matches
    Fanout: messages go to all queues bound to this
    exchange
    18
    

    View Slide

19. @lornajane
    REAL WORLD EXAMPLE
    An application needs images resized and
    sometimes modified
    19
    

    View Slide

20. @lornajane
    IMAGE RESIZING APPLICATION
    20
    

    View Slide

21. @lornajane
    DECLARE EXCHANGE, BIND QUEUE IN PHP
    require "vendor/autoload.php";
    $rabbit = new PhpAmqpLib\Connection\AMQPStreamConnection(
    'localhost', 5672, 'guest', 'guest');
    $channel = $rabbit->channel();
    $channel->exchange_declare('images', 'topic', false, true, false);
    $channel->queue_declare('triagev1', false, true, false, false);
    $channel->queue_bind('triagev1', 'images', 'image.v1.triage');
    21
    

    View Slide

22. @lornajane
    EXAMPLE MESSAGE STRUCTURE
    {
    "url": "https://www.gravatar.com/avatar/f6bb323eb6b2ad7f5c15f887",
    "badge": "pro",
    "sizes": [
    [ 100,
    100 ],
    [ 250,
    250 ]
    ]
    }
    22
    

    View Slide

23. @lornajane
    IMAGE RESIZING APPLICATION
    23
    

    View Slide

24. @lornajane
    IMAGE RESIZING APPLICATION
    24
    

    View Slide

25. @lornajane
    IMAGE RESIZING APPLICATION
    25
    

    View Slide

26. @lornajane
    IMAGE RESIZING APPLICATION
    26
    

    View Slide

27. @lornajane
    FEEDBACK MECHANISMS
    Rabbit is fire-and-forget; work is delegated
    Common pattern: return queue to put updates into
    for the original producer then to consume.
    27
    

    View Slide

28. @lornajane
    QUEUES WITH RABBITMQ
    28
    

    View Slide

29. @lornajane
    QUEUES WITH RABBITMQ
    Queues are awesome for scalability and robustness
    RabbitMQ is open source, lightweight and fast
    Queues help us meet the requirements for modern
    applications
    29
    

    View Slide

30. @lornajane
    THANKS!
    Feedback:
    Code:
    Managed RabbitMQ:
    Contact:
    lorna.mitchell@uk.ibm.com
    @lornajane
    https://joind.in/talk/4d555
    https://github.com/lornajane/queues-with-
    rabbitmq
    https://compose.com
    30
    

    View Slide