Upgrade to Pro — share decks privately, control downloads, hide ads and more …

RabbitMQ Asynchronous RPC with

RabbitMQ Asynchronous RPC with

Dawid Mazurek

June 26, 2017
Tweet

More Decks by Dawid Mazurek

Other Decks in Programming

Transcript

  1. RabbitMQ
    Asynchronous RPC with

    View Slide

  2. World without queues

    View Slide

  3. Execution time
    Confirm
    payment
    Assign paid
    feature

    View Slide

  4. Execution time
    Confirm
    payment
    Add to stats
    Assign paid
    feature

    View Slide

  5. Execution time
    Create
    invoice
    Confirm
    payment
    Add to stats
    Assign paid
    feature
    Some users leave page

    View Slide

  6. Execution time
    Create
    invoice
    Confirm
    payment
    Refresh user
    wallet
    Add to stats
    Assign paid
    feature
    Some users leave page

    View Slide

  7. Execution time
    Send
    confirmation
    email
    Create
    invoice
    Confirm
    payment
    Refresh user
    wallet
    Add to stats
    Assign paid
    feature
    Some users leave page Many users leave page

    View Slide

  8. Execution time
    Send
    confirmation
    email
    Send invoice
    via email
    Send printed
    invoice
    Create
    invoice
    Confirm
    payment
    Refresh user
    wallet
    Add to stats
    Assign paid
    feature
    Some users leave page Many users leave page

    View Slide

  9. Execution time
    Send
    confirmation
    email
    Send invoice
    via email
    Send printed
    invoice
    Create
    invoice
    Confirm
    payment
    Refresh user
    wallet
    Add to stats
    Assign paid
    feature
    Some users leave page Many users leave page

    View Slide

  10. Problems
    Multiple page refresh
    Data consistency problems
    Increase system load
    Require support for broken/retried processes
    Whole atomic transaction is very heavy, and when part of
    it fail, everything need to be retried
    After outage, all data is gone.

    View Slide

  11. View Slide

  12. Execution time
    Send
    confirmation
    email
    Send invoice
    via email
    Send printed
    invoice
    Create
    invoice
    Confirm
    payment
    Refresh user
    wallet
    Add to stats
    Assign paid
    feature
    queue
    queue
    queue
    queue

    View Slide

  13. View Slide

  14. Producer
    Producing task
    Consumer
    MySql
    Database queue (MySql table)
    Producer write rows to mysql table with tasks
    definition
    Task Task Task
    id:1 id:2 id:3
    Task
    id:4

    View Slide

  15. Producer
    Consuming task
    Consumer
    MySql
    Database queue (MySql table)
    Task Task Task
    id:2 id:3 id:4
    Task
    id:1
    Consumer executes select query to retrieve
    tasks to process

    View Slide

  16. Database queue (MySql table)
    Consumer 2
    Consumer 1
    Task Task Task Task
    modulo 2
    id:1 id:2 id:3 id:4

    View Slide

  17. Database queue (MySql table)
    Consumer 2
    Consumer 1
    Task Task Task Task
    modulo 3
    id:1 id:2 id:3 id:4
    Scaling problem
    need to restart all
    workers
    Consumer 3

    View Slide

  18. View Slide

  19. Producer
    Consuming task
    Consumer
    redis
    Producer executes RPUSH to put tasks on
    end of queue under specified queue name
    Task Task Task
    id:1 id:2 id:3
    Task
    id:4
    Redis queue

    View Slide

  20. Producer
    Consuming task
    Consumer
    redis
    Database queue (MySql table)
    Task Task Task
    id:2 id:3 id:4
    Task
    id:1
    Consumer executes LRANGE to retrieve
    entries from the beginning of queue and
    LTRIM to remove them after processed

    View Slide

  21. Redis queue
    Consumer
    Task Task Task Task
    id:1 id:2 id:3 id:4
    queue.task

    View Slide

  22. Redis queue
    Consumer 2
    Consumer 1
    Task Task Task Task
    id:1 id:3 id:5 id:7
    Task Task Task Task
    id:2 id:4 id:6 id:8
    queue.task.1
    queue.task.2

    View Slide

  23. Redis queue
    Consumer 1
    Task Task Task Task
    id:1 id:2 id:3 id:8
    Task Task Task Task
    id:4 id:5 id:6 id:7
    queue.task
    processing.queue.task
    Consumer 2
    C1 C1
    C2 C2

    View Slide

  24. Redis queue
    Consumer 1
    Task Task Task Task
    id:1 id:2 id:3 id:8
    Task Task Task Task
    id:4 id:5 id:6 id:7
    queue.task
    processing.queue.task
    Consumer 2
    C1 C1
    C2 C2
    Lost task
    requeue

    View Slide

  25. View Slide

  26. RabbitMQ
    Consumer 1
    Task Task Task Task
    id:1 id:2 id:3 id:4
    queue.task

    View Slide

  27. RabbitMQ
    Consumer 1
    Task Task Task Task
    id:1 id:2 id:3 id:4
    queue.task
    Consumer 2

    View Slide

  28. RabbitMQ
    Consumer 1
    Task Task Task Task
    id:1 id:2 id:3 id:4
    queue.task
    Consumer 2
    Consumer 3
    Consumer 4
    Consumer 5
    Consumer 6
    Consumer 7
    Consumer 9
    Consumer 10
    Consumer 11
    Consumer 12

    View Slide

  29. Acknowledge

    View Slide

  30. Creates rabbitmq
    server instance with
    management plugin
    enabled
    http://localhost:15672
    Management panel
    http://localhost:5672
    AMQP API
    > docker run -d --hostname
    localhost --name
    rabbitmq-server
    -p 15672:15672 -p 5672:5672
    rabbitmq:3-management

    View Slide

  31. $messageQueue = $container->get(
    'messaging.rpc.producer'
    );
    $task = new SendAdExpiredMail();
    $task->setAdId($ad[AdFields::FIELD_ID]);
    $messageQueue->send($task);
    Producer
    Publishing new task
    RabbitMQ

    View Slide

  32. Producer
    Publishing new task
    RabbitMQ
    class SendAdExpiredMail extends AbstractMessage
    {
    public function setAdId(int $adId)
    {
    $this->payload['adId'] = $adId;
    }
    public function getAdId(): int
    {
    return $this->payload['adId'];
    }
    public function getQueueName(): string
    {
    return QueueName::MAILER_AD_EXPIRED;
    }
    }

    View Slide

  33. Producer
    Task json payload
    RabbitMQ
    {
    "payload": {
    "adId": 1
    },
    "metadata": {
    "className": "SendAdExpiredMail"
    }
    }

    View Slide

  34. $mailer = new MailHandler();
    $queueMapper = new QueueMapper();
    $queueMapper->addMapping(
    new QueueMapping('AdExpired',
    [$mailer, 'sendAdExpiredEmail']
    )
    );
    Producer
    Consuming task - listener
    Consumer
    RabbitMQ

    View Slide

  35. Producer
    Consuming task - listener
    Consumer
    RabbitMQ
    $messageExecutor = $container->get(
    'messaging.rpc.executor'
    );
    $messageExecutor->runOnVhostWithMap(
    'mailing',
    $queueMapper
    );

    View Slide

  36. Producer
    Consuming task - handler
    Consumer
    RabbitMQ
    class MailHandler
    {
    public function sendAdExpiredEmail
    (SendAdExpiredMail $task): bool
    {
    (...)
    }
    }

    View Slide

  37. View Slide

  38. class MessageQueue
    {
    private $writer;
    private $serializer;
    public function __construct(
    MessageQueueWriterInterface $writer,
    MessageSerializerInterface $serializer
    )
    {
    $this->writer = $writer;
    $this->serializer = $serializer;
    }
    public function send(MessageInterface $message)
    {
    $this->writer->write(
    $message->getQueueName(),
    $this->serializer->serialize($message)
    );
    }
    }
    Queuing class
    Write model
    High abstraction level

    View Slide

  39. class RabbitMqMessageQueueWriter implements
    MessageQueueWriterInterface
    {
    public function __construct(RabbitMqConnection $connection)
    {
    $this->connection = $connection;
    }
    public function write(string $queueName, string $payload)
    {
    $this->connection->setVHost($this->getVHostFromName($queueName));
    $this->connection->connect();
    $channel = $this->connection->openChannel();
    $channel->publish(
    new AMQPMessage($payload),
    $this->getExchangeFromName($queueName),
    $this->getRoutingKeyFromName($queueName)
    );
    $channel->close();
    $this->connection->disconnect();
    }
    }
    Write model
    implementation

    View Slide

  40. class RabbitMqChannel
    {
    public function publish(
    AMQPMessage $message,
    string $exchangeName,
    string $routingKey
    )
    {
    $this->channel->exchange_declare(
    $exchangeName,
    'topic',
    false, // passive
    true, // durable
    false // auto_delete
    );
    $this->channel->basic_publish(
    $message,
    $exchangeName,
    $routingKey
    );
    }
    }
    Queuing class
    Write model
    Low abstraction
    level

    View Slide

  41. class JsonMessageSerializer implements MessageSerializerInterface
    {
    const FIELD_PAYLOAD = 'payload';
    const FIELD_METADATA = 'metadata';
    public function serialize(MessageInterface $message): string
    {
    $unserialized = [
    self::FIELD_PAYLOAD => $message->getPayload(),
    self::FIELD_METADATA => $message->getMetadata()
    ];
    return json_encode($unserialized);
    }
    public function unserialize(string $serializedMessage): MessageInterface
    {
    $unserialized = json_decode($serializedMessage, true);
    $payload = $unserialized[self::FIELD_PAYLOAD];
    $metadata = $unserialized[self::FIELD_METADATA];
    $messageType = $metadata['className'];
    $message = new $messageType();
    $message->setPayload($payload);
    $message->setMetadata($metadata);
    return $message;
    }
    }
    Task serializer

    View Slide

  42. Task abstraction
    abstract class AbstractMessage implements MessageInterface{
    public function getQueueName(): string{
    return $this->queueName;
    }
    public function getPayload(): array {
    return $this->payload;
    }
    public function getMetadata(): array {
    return array_merge($this->metadata, ['className' => static::class]);
    }
    public function setMetadata(array $metadata) {
    $this->metadata = $metadata;
    }
    public function setPayload(array $payload)
    {
    $this->payload = $payload;
    }
    }

    View Slide

  43. class MessageQueueRPCExecutor
    {
    private $connection;
    public function __construct(RabbitMqConnection $connection)
    {
    $this->connection = $connection;
    }
    public function runOnVhostWithMap(
    string $vhost,
    QueueMapper $queueMapper)
    {
    $this->connection->setVHost($vhost);
    $messageExecutor = new MessageQueueExecutor(
    new RabbitMqMessageQueueExecutor(
    $this->connection
    ),
    new JsonMessageSerializer(),
    $queueMapper
    );
    $messageExecutor->run();
    }
    }
    Read model - executor

    View Slide

  44. class RabbitMqMessageQueueExecutor implements
    MessageQueueExecutorInterface
    {
    public function __construct(RabbitMqConnection $connection)
    {
    $this->connection = $connection;
    }
    public function run()
    {
    $channel = $this->connection->openChannel();
    foreach ($this->mapping as $queueName => $executor) {a
    $channel->bindQueueToExecutor($queueName, $executor);
    }
    $channel->listen();
    $channel->close();
    }
    public function mapQueueToExecutor(string $queueName, callable
    $executor)
    {
    $this->mapping[$queueName] = $this->getCallbackWrapper($executor);
    }
    Read model - executor

    View Slide

  45. private function getCallbackWrapper(callable $executor)
    {
    return function (AMQPMessage $message) use ($executor)
    {
    $success = $executor(new RabbitMqMessage($message));
    if ($success) {
    $message->delivery_info['channel']->basic_ack(
    $message->delivery_info['delivery_tag']
    );
    } else {
    $message->delivery_info['channel']->basic_nack(
    $message->delivery_info['delivery_tag'],
    false,
    true
    );
    }
    return $success;
    };
    }
    Low level
    processing

    View Slide

  46. class RabbitMqChannel
    {
    public function publish(
    AMQPMessage $message,
    string $exchangeName,
    string $routingKey
    )
    {
    $this->channel->exchange_declare($exchangeName, 'topic', false, true, false);
    $this->channel->basic_publish($message, $exchangeName, $routingKey);
    }
    public function bindQueueToExecutor(string $queueName, callable $executor)
    {
    $this->channel->queue_declare($queueName, false, true, false, false);
    $this->channel->basic_consume($queueName, '', false, false, false, false, $executor);
    }
    public function listen()
    {
    while(count($this->channel->callbacks)) {
    $this->channel->wait();
    }
    }
    }
    Low level read

    View Slide

  47. Producer
    Connect
    Open
    channel
    Declare
    exchange
    Basic publish
    Write flow

    View Slide

  48. Connect
    Open
    channel
    Declare queue
    Basic
    consume
    Consumer
    Read flow

    View Slide

  49. Consumer
    Producer
    Exchange Queue
    Application
    RabbitMQ
    Routing
    binding

    View Slide

  50. AdExpired
    AdModerated
    AdExpired
    AdModerated
    mailing_olxpl mailing_olxpt
    vhost

    View Slide

  51. Exchange types
    direct
    topic
    fanout
    header
    Exchange
    Queue A Queue B
    mailer.ad.expired mailer.ad.moderated
    mailer.ad.expired

    View Slide

  52. View Slide

  53. Exchange types
    direct
    topic
    fanout
    headers
    Exchange
    Queue A Queue B
    mailer.ad.expired
    mailer.ad.* mailer.user.*

    View Slide

  54. Exchange types
    direct
    topic
    fanout
    headers
    Exchange
    Mailer queue Stats queue
    mailer.ad.expired
    mailer.ad.* mailer.#

    View Slide

  55. Exchange types
    direct
    topic
    fanout
    headers
    Exchange
    Mailer queue Mail stats queue
    mailer.ad.expired

    View Slide

  56. Exchange types
    direct
    topic
    fanout
    headers
    Exchange
    Mailer queue Mail stats queue
    param1:1 param2:2

    View Slide

  57. Exchange
    Mailer queue Mail stats queue
    mailer.ad.expired
    Delayed delivery
    Exchange
    Exchange to
    exchange binding
    rabbitmq-plugins enable rabbitmq_delayed_message_exchange

    View Slide

  58. Delaying message
    delivery
    $channel->exchange_declare('delayed_exchange', 'x-delayed-message',
    false, true, false, false, false,
    new AMQPTable(array(
    "x-delayed-type" => "direct"
    )));
    $hdrs = new AMQPTable(array("x-delay" => 300));
    $msg = new AMQPMessage('5 minute delayed task',
    array('delivery_mode' => 2)
    );
    $msg->set('application_headers', $hdrs);
    $channel->basic_publish($msg, 'delayed_exchange', 'routing_key');

    View Slide

  59. Exchange
    Mailer queue Mail stats queue
    mailer.ad.expired
    Delayed delivery
    Exchange
    Exchange to
    exchange binding
    Delayed retries
    Producer
    Consumer
    Failed task execution

    View Slide

  60. Handling failed tasks
    Ignore repetition
    {
    "payload": {
    "adId": 1
    },
    "metadata": {
    "className": "SendAdExpiredMail",
    "retryStrategy": "ignore"
    }
    }

    View Slide

  61. Handling failed tasks
    Controlled repetition
    {
    "payload": {
    "adId": 1
    },
    "metadata": {
    "className": "SendAdExpiredMail",
    "retryStrategy": "repeat",
    "retryLimit": "5",
    "retryCount": "0",
    }
    }

    View Slide

  62. Handling failed tasks
    Controlled repetition
    with declared intervals
    {
    "payload": {
    "adId": 1
    },
    "metadata": {
    "className": "SendAdExpiredMail",
    "retryStrategy": "repeat",
    "retryLimit": "5",
    "retryCount": "0",
    "retryInterval": [15, 30, 60, 300, 86400],
    }
    }

    View Slide

  63. Fire and Forget

    View Slide

  64. Producer Consumer
    Asynchronous RPC call

    View Slide

  65. Producer Consumer
    Asynchronous RPC call
    Asynchronous RPC response

    View Slide

  66. Send task
    Wait for response
    Create response hash
    Open response queue
    Continue processing

    View Slide

  67. interface ResponseRPC {
    public function getResponseQueueName(): string;
    }
    class GetUserFriends extends AbstractMessage implements ResponseRPC
    {
    public function getResponseQueueName(): string
    {
    if (!isset($this->metadata['response_queue'])) {
    $this->metadata['response_queue'] = 'user-friends-' . $this->payload['user_id'] . '_' . uniqid('', true);
    }
    return $this->metadata['response_queue'];
    }

    View Slide

  68. $messageQueue = $this->container->get('messaging.rpc.producer.responding');
    $task = new GetUserFriends();
    $task->setId($user[UserFields::FIELD_ID]);
    $messageQueue->send($task);
    /* do some processing */
    $userFriends = $messageQueue->getAsyncResponse();

    View Slide

  69. class RespondingMessageQueue extends MessageQueue
    {
    public function __construct( MessageQueue $writer, MessageQueueExecutorInterface $executor)
    {
    $this->writer = $writer;
    $this->executor = $executor;
    }
    public function send(MessageInterface $message)
    {
    if ($message instanceof ResponseRPC){
    $this->openResponseQueue($message);
    }
    $this->writer->send($message);
    }
    public function getAsyncResponse(): MessageInterface
    {
    $this->executor->runSingle(10);
    return $this->response;
    }
    private function openResponseQueue(ResponseRPC $message)
    {
    $message->getResponseQueueName();
    $this->executor->mapQueueToExecutor( $message->getResponseQueueName(),[$this, 'catchResponse']);
    }
    private function catchResponse(MessageInterface $message)
    {
    $this->response = $message;
    }
    }

    View Slide

  70. Producer Consumer
    Asynchronous RPC calls
    Asynchronous RPC responses

    View Slide

  71. user(id:27192) {
    active_ads
    observed_ads
    recommendations
    messages
    }
    GraphQL
    Sample request

    View Slide

  72. getActiveAds
    getObservedAds
    getRecommendations
    getMessages
    Request Response

    View Slide

  73. getActiveAds
    getObservedAds
    getRecommendations
    getMessages
    Request Response

    View Slide

  74. $messageQueue = $this->container->get('messaging.rpc.producer.responding');
    $getActiveAds = new GetActiveAds();
    $getActiveAds->setUserId($user[UserFields::FIELD_ID]);
    $getObservedAds = new GetObservedAds();
    $getObservedAds->setUserId($user[UserFields::FIELD_ID]);
    $getRecommendations = new GetRecommendations();
    $getRecommendations->setUserId($user[UserFields::FIELD_ID]);
    $getMessages = new GetMessages();
    $getMessages->setUserId($user[UserFields::FIELD_ID]);
    $messageQueue->sendMultiple(
    [$getActiveAds, $getObservedAds, $getRecommendations, $getMessages]
    );
    /* do some processing */
    $userAggregateData = $messageQueue->getAsyncResponses();

    View Slide

  75. ● more consumers
    ● faster consumers
    ● higher prefetch count

    View Slide

  76. Useful links
    http://tryrabbitmq.com/
    https://github.com/php-amqplib/php-amqplib
    https://www.cloudamqp.com/

    View Slide

  77. RabbitMQ
    Asynchronous RPC with Dawid Mazurek
    PHP DEV at

    View Slide