Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Queues with RabbitMQ

Lorna Mitchell
September 30, 2016

Queues with RabbitMQ

Talk delivered at the PHPNW conference in Manchester

Lorna Mitchell

September 30, 2016
Tweet

More Decks by Lorna Mitchell

Other Decks in Technology

Transcript

  1. @lornajane
    QUEUES WITH RABBITMQ
    Lorna Mitchell, IBM Cloud Data Services
    PHPNW, September 2016
    (slides are available: )
    http://lornajane.net/resources
    1

    View full-size slide

  2. @lornajane
    WHY USE A QUEUE?
    loose coupling for scalability
    interaction between technology stacks
    asynchronous processing
    event-driven patterns
    2

    View full-size slide

  3. @lornajane
    WHY USE RABBITMQ?
    Homepage:
    Open source, uses AMQP standard. Scalable, robust
    and written in Erlang.
    Alternatives could be:
    Gearman
    Beanstalkd
    A lot of the theory applies regardless
    https://www.rabbitmq.com/
    3

    View full-size slide

  4. @lornajane
    GETTING TO KNOW
    RABBITMQ
    4

    View full-size slide

  5. @lornajane
    RABBITMQ CHANNELS
    When we want to talk to RabbitMQ, we establish a
    channel
    Channels allow multiple connections between
    servers over a single TCP connection, but each is
    isolated from the others.
    5

    View full-size slide

  6. @lornajane
    DECLARING THE QUEUE
    We declare a queue before we write to it.
    if it doesn't exist, it is created
    if it does exist, cool
    to check if a queue exists but not create it, use
    passive mode
    Our example uses the default exchange, more on
    exchanges in a bit
    6

    View full-size slide

  7. @lornajane
    SIMPLE (BUT REAL)
    EXAMPLE
    Use queues to handle incoming webhooks
    7

    View full-size slide

  8. @lornajane
    PHP AND AMQP
    Try the php-amqplib/php-amqplib package from
    Composer
    8

    View full-size slide

  9. @lornajane
    HANDLE WEBHOOKS
    require "vendor/autoload.php";
    $input = file_get_contents("php://input");
    $rabbit = new PhpAmqpLib\Connection\AMQPStreamConnection(
    'localhost', 5672, 'guest', 'guest');
    $channel = $rabbit->channel();
    $channel->queue_declare('pushes', false, true, false, false);
    $message = new PhpAmqpLib\Message\AMQPMessage(
    $input, ["delivery_mode" => 2]);
    $channel->basic_publish($message, '', 'pushes');
    9

    View full-size slide

  10. @lornajane
    RABBITMQ-MANAGEMENT PLUGIN
    10

    View full-size slide

  11. @lornajane
    CONSUMING WITH PHP
    require "vendor/autoload.php";
    $rabbit = new PhpAmqpLib\Connection\AMQPStreamConnection(
    'localhost', 5672, 'guest', 'guest');
    $channel = $rabbit->channel();
    $channel->queue_declare('pushes', false, true, false, false);
    echo "waiting for jobs ...\n";
    11

    View full-size slide

  12. @lornajane
    CONSUMING WITH PHP
    $process = function ($message) {
    $data = json_decode($message->getBody(), true);
    // do the actual processing here ...
    $message->delivery_info['channel']
    ->basic_ack($message->delivery_info['delivery_tag']);
    };
    $channel->basic_consume('pushes', '',
    false, false, false, false, $process);
    while(count($channel->callbacks)) {
    $channel->wait();
    }
    12

    View full-size slide

  13. @lornajane
    WRITING WORKERS
    Workers are long-running processes, beware scope
    issues and memory leaks.
    There can be many workers attached to one queue,
    each one independent.
    They need to be robust and handle failures.
    13

    View full-size slide

  14. @lornajane
    COMPLETED MESSAGES
    acknowledge when messages are processed
    successfully
    can acknowledge failure
    reject the message
    optionally: requeue
    14

    View full-size slide

  15. @lornajane
    HANDLING FAILURE: DEAD LETTER QUEUES
    reject without requeue
    exceed TTL
    queue length exceeded
    15

    View full-size slide

  16. @lornajane
    HANDLING FAILURE: RETRIES
    Implement your own logic to handle retries
    Create a new message with:
    all the existing message contents
    plus some metadata such as retry count or
    backoff time
    16

    View full-size slide

  17. @lornajane
    RABBITMQ EXCHANGES
    Exchanges are the routing logic of RabbitMQ
    Messages go to exchanges and the exchanges put
    them into the correct queues for storage
    (our first example used the empty-named default
    exchange)
    17

    View full-size slide

  18. @lornajane
    TYPES OF EXCHANGE
    Direct: a given routing key puts messages onto the
    matching queue(s)
    Topic: queues are bound by key, and messages are
    routed to as many queues as their routing key
    matches
    Fanout: messages go to all queues bound to this
    exchange
    18

    View full-size slide

  19. @lornajane
    REAL WORLD EXAMPLE
    An application needs images resized and
    sometimes modified
    19

    View full-size slide

  20. @lornajane
    IMAGE RESIZING APPLICATION
    20

    View full-size slide

  21. @lornajane
    DECLARE EXCHANGE, BIND QUEUE IN PHP
    require "vendor/autoload.php";
    $rabbit = new PhpAmqpLib\Connection\AMQPStreamConnection(
    'localhost', 5672, 'guest', 'guest');
    $channel = $rabbit->channel();
    $channel->exchange_declare('images', 'topic', false, true, false);
    $channel->queue_declare('triagev1', false, true, false, false);
    $channel->queue_bind('triagev1', 'images', 'image.v1.triage');
    21

    View full-size slide

  22. @lornajane
    EXAMPLE MESSAGE STRUCTURE
    {
    "url": "https://www.gravatar.com/avatar/f6bb323eb6b2ad7f5c15f887",
    "badge": "pro",
    "sizes": [
    [ 100,
    100 ],
    [ 250,
    250 ]
    ]
    }
    22

    View full-size slide

  23. @lornajane
    IMAGE RESIZING APPLICATION
    23

    View full-size slide

  24. @lornajane
    IMAGE RESIZING APPLICATION
    24

    View full-size slide

  25. @lornajane
    IMAGE RESIZING APPLICATION
    25

    View full-size slide

  26. @lornajane
    IMAGE RESIZING APPLICATION
    26

    View full-size slide

  27. @lornajane
    FEEDBACK MECHANISMS
    Rabbit is fire-and-forget; work is delegated
    Common pattern: return queue to put updates into
    for the original producer then to consume.
    27

    View full-size slide

  28. @lornajane
    QUEUES WITH RABBITMQ
    28

    View full-size slide

  29. @lornajane
    QUEUES WITH RABBITMQ
    Queues are awesome for scalability and robustness
    RabbitMQ is open source, lightweight and fast
    Queues help us meet the requirements for modern
    applications
    29

    View full-size slide

  30. @lornajane
    THANKS!
    Feedback:
    Code:
    Managed RabbitMQ:
    Contact:
    [email protected]
    @lornajane
    https://joind.in/talk/4d555
    https://github.com/lornajane/queues-with-
    rabbitmq
    https://compose.com
    30

    View full-size slide