Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Queues with RabbitMQ

Queues with RabbitMQ

Overview of using RabbitMQ in your applications with PHP

Lorna Mitchell

October 24, 2017
Tweet

More Decks by Lorna Mitchell

Other Decks in Technology

Transcript

  1. Queues With
    RabbitMQ
    Lorna Mitchell, IBM

    View Slide

  2. Introducing Queues
    Use queues to:
    • asynchronously process tasks in your (existing) application
    • e.g. sending email, processing uploads
    • provide loose coupling points
    • e.g. delegate heavy tasks, split ownership/technology
    • enable parts of the system to scale appropriately
    • e.g. an event in one system causes a thousand other
    actions
    @lornajane

    View Slide

  3. Queue Tools
    A selection of queue-ish tools
    • RabbitMQ* http://www.rabbitmq.com/
    • Gearman http://gearman.org/
    • Beanstalkd http://kr.github.io/beanstalkd/
    • Kafka https://kafka.apache.org/
    • Redis https://redis.io/
    * RabbitMQ is used in today's examples
    @lornajane

    View Slide

  4. Getting To Know
    RabbitMQ
    @lornajane

    View Slide

  5. Management Plugin
    https://www.rabbitmq.com/management.html
    @lornajane

    View Slide

  6. Getting To Know RabbitMQ
    Some vocabulary
    • broker: the RabbitMQ instance
    • exchange: where to send the messages to
    • queue: where messages wait to be processed
    • binding key: rules for which messages go into this queue
    • message: the data to be processed
    • routing key: message route information
    • consumer: worker script to processes the messages
    @lornajane

    View Slide

  7. RabbitMQ Exchanges
    Exchanges are the routing logic of RabbitMQ
    Messages go to exchanges and the exchanges put them into
    the correct queues for storage
    @lornajane

    View Slide

  8. Types Of Exchange
    Direct: a given routing key puts messages onto the matching
    queue(s)
    Topic: queues are bound by key, and messages are routed to
    as many queues as their routing key matches
    Fanout: messages go to all queues bound to this exchange
    @lornajane

    View Slide

  9. Default Exchange
    There is a default exchange in RabbitMQ
    Its name is '' and routing is on queue name
    @lornajane

    View Slide

  10. RabbitMQ Queues
    Queues can:
    • have wildcards in binding keys
    • be durable (messages have their own durability)
    • have a maximum length
    • be configured with a "dead letter exchange"
    @lornajane

    View Slide

  11. RabbitMQ Messages
    Messages have:
    • a body consisting of a string of data (JSON is common)
    • additional data, including TTL (Time To Live) and message
    durability
    • may have priority information
    @lornajane

    View Slide

  12. RabbitMQ and PHP
    Sample code here:
    https://github.com/lornajane/queues-with-rabbitmq
    @lornajane

    View Slide

  13. RabbitMQ and PHP
    Best library: https://github.com/php-amqplib/php-amqplib
    composer require php-amqplib/php-amqplib
    • dependencies include bcmath and mbstring
    @lornajane

    View Slide

  14. Webhooks on GitHub
    @lornajane

    View Slide

  15. Ngrok for Local Webhooks
    https://ngrok.com/ - secure tunnel to your dev platform
    Use this tool to:
    • webhook into code running locally
    • inspect the request and response of the webhook
    • replay requests and see the responses
    @lornajane

    View Slide

  16. PHP Producer
    Receive webhooks from GitHub, add to queue
    1 require "vendor/autoload.php";
    2 $input = file_get_contents("php://input");
    3 $data = json_decode($input, true);
    4
    5 $rabbit = new PhpAmqpLib\Connection\AMQPStreamConnection(
    6 'localhost', 5672, 'guest', 'guest');
    7 $channel = $rabbit->channel();
    8
    9 $channel->queue_declare('pushes', false, true, false, false);
    10 $message = new PhpAmqpLib\Message\AMQPMessage(
    11 $input, ["delivery_mode" => 2]);
    12 $channel->basic_publish($message, '', 'pushes');
    @lornajane

    View Slide

  17. PHP Consumer
    Process a queue with PHP
    1 require "vendor/autoload.php";
    2 $rabbit = new PhpAmqpLib\Connection\AMQPStreamConnection(
    3 'localhost', 5672, 'guest', 'guest');
    4 $channel = $rabbit->channel();
    5
    6 $channel->queue_declare('pushes', false, true, false, false);
    @lornajane

    View Slide

  18. PHP Consumer
    Process a queue with PHP (continued)
    1 $process = function ($message) {
    2 $data = json_decode($message->getBody(), true);
    3 // do message processing here
    4 $message->delivery_info['channel']->
    5 basic_ack($message->delivery_info['delivery_tag']);
    6 };
    7
    8 $channel->basic_consume('pushes', '', false, false,
    9 false, false, $process);
    10
    11 while(count($channel->callbacks)) { $channel->wait(); }
    @lornajane

    View Slide

  19. Example App: Webhooks
    @lornajane

    View Slide

  20. Example App: Webhooks
    (code: https://github.com/ibm-watson-data-lab/guestbook )
    A simple guestbook application is extended to allow webhook
    notifications of new comments.
    @lornajane

    View Slide

  21. Example App: Webhooks
    @lornajane

    View Slide

  22. Example App: Webhooks
    @lornajane

    View Slide

  23. Example App: Webhooks
    @lornajane

    View Slide

  24. Example App: Webhooks
    @lornajane

    View Slide

  25. Processing Messages
    @lornajane

    View Slide

  26. Creating Workers
    Workers are disposable!
    • if things go wrong, exit
    • separate tool to monitor/restart as needed
    • beware long-running process hazards
    • everything processed "at least once" (but maybe more than
    once, and in any order...)
    @lornajane

    View Slide

  27. Independent Workers
    @lornajane

    View Slide

  28. Independent Workers
    For best results:
    • awesome, aggregated logging
    • monitoring: queue size, worker uptime, processing time
    • miniminum viable dependencies
    @lornajane

    View Slide

  29. Completed Messages
    • acknowledge when messages are processed successfully
    • can acknowledge failure
    • reject the message
    • optionally: requeue
    @lornajane

    View Slide

  30. Dead Letter Exchanges
    • reject without requeue
    • exceed TTL
    • queue length exceeded
    @lornajane

    View Slide

  31. Retries
    Implement your own logic to handle retries
    Create a new message with:
    • all the existing message contents
    • plus some metadata such as retry count or backoff time
    @lornajane

    View Slide

  32. Feedback Mechanisms
    Rabbit is fire-and-forget; work is delegated
    Common pattern: return queue to put updates into for the
    original producer then to consume.
    @lornajane

    View Slide

  33. Example App: Webhooks
    @lornajane

    View Slide

  34. Example App: Webhooks
    @lornajane

    View Slide

  35. Queues With RabbitMQ
    @lornajane

    View Slide

  36. Queues With RabbitMQ
    Queues are awesome for scalability and robustness
    RabbitMQ is open source, lightweight and fast
    Queues help us meet the requirements for modern
    applications
    @lornajane

    View Slide

  37. Thanks!
    Resources:
    • RabbitMQ: https://www.rabbitmq.com/
    • Try it: https://ibm.com/cloud
    • Blog: https://lornajane.net
    • Code: https://github.com/ibm-watson-data-lab/guestbook
    • @lornajane
    [email protected]
    @lornajane

    View Slide