RabbitMQ Asynchronous RPC with

Transcript

1. RabbitMQ
   Asynchronous RPC with
   

   View Slide

2. World without queues
   

   View Slide

3. Execution time
   Confirm
   payment
   Assign paid
   feature
   

   View Slide

4. Execution time
   Confirm
   payment
   Add to stats
   Assign paid
   feature
   

   View Slide

5. Execution time
   Create
   invoice
   Confirm
   payment
   Add to stats
   Assign paid
   feature
   Some users leave page
   

   View Slide

6. Execution time
   Create
   invoice
   Confirm
   payment
   Refresh user
   wallet
   Add to stats
   Assign paid
   feature
   Some users leave page
   

   View Slide

7. Execution time
   Send
   confirmation
   email
   Create
   invoice
   Confirm
   payment
   Refresh user
   wallet
   Add to stats
   Assign paid
   feature
   Some users leave page Many users leave page
   

   View Slide

8. Execution time
   Send
   confirmation
   email
   Send invoice
   via email
   Send printed
   invoice
   Create
   invoice
   Confirm
   payment
   Refresh user
   wallet
   Add to stats
   Assign paid
   feature
   Some users leave page Many users leave page
   

   View Slide

9. Execution time
   Send
   confirmation
   email
   Send invoice
   via email
   Send printed
   invoice
   Create
   invoice
   Confirm
   payment
   Refresh user
   wallet
   Add to stats
   Assign paid
   feature
   Some users leave page Many users leave page
   

   View Slide

10. Problems
    Multiple page refresh
    Data consistency problems
    Increase system load
    Require support for broken/retried processes
    Whole atomic transaction is very heavy, and when part of
    it fail, everything need to be retried
    After outage, all data is gone.
    

    View Slide

11. View Slide

12. Execution time
    Send
    confirmation
    email
    Send invoice
    via email
    Send printed
    invoice
    Create
    invoice
    Confirm
    payment
    Refresh user
    wallet
    Add to stats
    Assign paid
    feature
    queue
    queue
    queue
    queue
    

    View Slide

13. View Slide

14. Producer
    Producing task
    Consumer
    MySql
    Database queue (MySql table)
    Producer write rows to mysql table with tasks
    definition
    Task Task Task
    id:1 id:2 id:3
    Task
    id:4
    

    View Slide

15. Producer
    Consuming task
    Consumer
    MySql
    Database queue (MySql table)
    Task Task Task
    id:2 id:3 id:4
    Task
    id:1
    Consumer executes select query to retrieve
    tasks to process
    

    View Slide

16. Database queue (MySql table)
    Consumer 2
    Consumer 1
    Task Task Task Task
    modulo 2
    id:1 id:2 id:3 id:4
    

    View Slide

17. Database queue (MySql table)
    Consumer 2
    Consumer 1
    Task Task Task Task
    modulo 3
    id:1 id:2 id:3 id:4
    Scaling problem
    need to restart all
    workers
    Consumer 3
    

    View Slide

18. View Slide

19. Producer
    Consuming task
    Consumer
    redis
    Producer executes RPUSH to put tasks on
    end of queue under specified queue name
    Task Task Task
    id:1 id:2 id:3
    Task
    id:4
    Redis queue
    

    View Slide

20. Producer
    Consuming task
    Consumer
    redis
    Database queue (MySql table)
    Task Task Task
    id:2 id:3 id:4
    Task
    id:1
    Consumer executes LRANGE to retrieve
    entries from the beginning of queue and
    LTRIM to remove them after processed
    

    View Slide

21. Redis queue
    Consumer
    Task Task Task Task
    id:1 id:2 id:3 id:4
    queue.task
    

    View Slide

22. Redis queue
    Consumer 2
    Consumer 1
    Task Task Task Task
    id:1 id:3 id:5 id:7
    Task Task Task Task
    id:2 id:4 id:6 id:8
    queue.task.1
    queue.task.2
    

    View Slide

23. Redis queue
    Consumer 1
    Task Task Task Task
    id:1 id:2 id:3 id:8
    Task Task Task Task
    id:4 id:5 id:6 id:7
    queue.task
    processing.queue.task
    Consumer 2
    C1 C1
    C2 C2
    

    View Slide

24. Redis queue
    Consumer 1
    Task Task Task Task
    id:1 id:2 id:3 id:8
    Task Task Task Task
    id:4 id:5 id:6 id:7
    queue.task
    processing.queue.task
    Consumer 2
    C1 C1
    C2 C2
    Lost task
    requeue
    

    View Slide

25. View Slide

26. RabbitMQ
    Consumer 1
    Task Task Task Task
    id:1 id:2 id:3 id:4
    queue.task
    

    View Slide

27. RabbitMQ
    Consumer 1
    Task Task Task Task
    id:1 id:2 id:3 id:4
    queue.task
    Consumer 2
    

    View Slide

28. RabbitMQ
    Consumer 1
    Task Task Task Task
    id:1 id:2 id:3 id:4
    queue.task
    Consumer 2
    Consumer 3
    Consumer 4
    Consumer 5
    Consumer 6
    Consumer 7
    Consumer 9
    Consumer 10
    Consumer 11
    Consumer 12
    

    View Slide

29. Acknowledge
    

    View Slide

30. Creates rabbitmq
    server instance with
    management plugin
    enabled
    http://localhost:15672
    Management panel
    http://localhost:5672
    AMQP API
    > docker run -d --hostname
    localhost --name
    rabbitmq-server
    -p 15672:15672 -p 5672:5672
    rabbitmq:3-management
    

    View Slide

31. $messageQueue = $container->get(
    'messaging.rpc.producer'
    );
    $task = new SendAdExpiredMail();
    $task->setAdId($ad[AdFields::FIELD_ID]);
    $messageQueue->send($task);
    Producer
    Publishing new task
    RabbitMQ
    

    View Slide

32. Producer
    Publishing new task
    RabbitMQ
    class SendAdExpiredMail extends AbstractMessage
    {
    public function setAdId(int $adId)
    {
    $this->payload['adId'] = $adId;
    }
    public function getAdId(): int
    {
    return $this->payload['adId'];
    }
    public function getQueueName(): string
    {
    return QueueName::MAILER_AD_EXPIRED;
    }
    }
    

    View Slide

33. Producer
    Task json payload
    RabbitMQ
    {
    "payload": {
    "adId": 1
    },
    "metadata": {
    "className": "SendAdExpiredMail"
    }
    }
    

    View Slide

34. $mailer = new MailHandler();
    $queueMapper = new QueueMapper();
    $queueMapper->addMapping(
    new QueueMapping('AdExpired',
    [$mailer, 'sendAdExpiredEmail']
    )
    );
    Producer
    Consuming task - listener
    Consumer
    RabbitMQ
    

    View Slide

35. Producer
    Consuming task - listener
    Consumer
    RabbitMQ
    $messageExecutor = $container->get(
    'messaging.rpc.executor'
    );
    $messageExecutor->runOnVhostWithMap(
    'mailing',
    $queueMapper
    );
    

    View Slide

36. Producer
    Consuming task - handler
    Consumer
    RabbitMQ
    class MailHandler
    {
    public function sendAdExpiredEmail
    (SendAdExpiredMail $task): bool
    {
    (...)
    }
    }
    

    View Slide

37. View Slide

38. class MessageQueue
    {
    private $writer;
    private $serializer;
    public function __construct(
    MessageQueueWriterInterface $writer,
    MessageSerializerInterface $serializer
    )
    {
    $this->writer = $writer;
    $this->serializer = $serializer;
    }
    public function send(MessageInterface $message)
    {
    $this->writer->write(
    $message->getQueueName(),
    $this->serializer->serialize($message)
    );
    }
    }
    Queuing class
    Write model
    High abstraction level
    

    View Slide

39. class RabbitMqMessageQueueWriter implements
    MessageQueueWriterInterface
    {
    public function __construct(RabbitMqConnection $connection)
    {
    $this->connection = $connection;
    }
    public function write(string $queueName, string $payload)
    {
    $this->connection->setVHost($this->getVHostFromName($queueName));
    $this->connection->connect();
    $channel = $this->connection->openChannel();
    $channel->publish(
    new AMQPMessage($payload),
    $this->getExchangeFromName($queueName),
    $this->getRoutingKeyFromName($queueName)
    );
    $channel->close();
    $this->connection->disconnect();
    }
    }
    Write model
    implementation
    

    View Slide

40. class RabbitMqChannel
    {
    public function publish(
    AMQPMessage $message,
    string $exchangeName,
    string $routingKey
    )
    {
    $this->channel->exchange_declare(
    $exchangeName,
    'topic',
    false, // passive
    true, // durable
    false // auto_delete
    );
    $this->channel->basic_publish(
    $message,
    $exchangeName,
    $routingKey
    );
    }
    }
    Queuing class
    Write model
    Low abstraction
    level
    

    View Slide

41. class JsonMessageSerializer implements MessageSerializerInterface
    {
    const FIELD_PAYLOAD = 'payload';
    const FIELD_METADATA = 'metadata';
    public function serialize(MessageInterface $message): string
    {
    $unserialized = [
    self::FIELD_PAYLOAD => $message->getPayload(),
    self::FIELD_METADATA => $message->getMetadata()
    ];
    return json_encode($unserialized);
    }
    public function unserialize(string $serializedMessage): MessageInterface
    {
    $unserialized = json_decode($serializedMessage, true);
    $payload = $unserialized[self::FIELD_PAYLOAD];
    $metadata = $unserialized[self::FIELD_METADATA];
    $messageType = $metadata['className'];
    $message = new $messageType();
    $message->setPayload($payload);
    $message->setMetadata($metadata);
    return $message;
    }
    }
    Task serializer
    

    View Slide

42. Task abstraction
    abstract class AbstractMessage implements MessageInterface{
    public function getQueueName(): string{
    return $this->queueName;
    }
    public function getPayload(): array {
    return $this->payload;
    }
    public function getMetadata(): array {
    return array_merge($this->metadata, ['className' => static::class]);
    }
    public function setMetadata(array $metadata) {
    $this->metadata = $metadata;
    }
    public function setPayload(array $payload)
    {
    $this->payload = $payload;
    }
    }
    

    View Slide

43. class MessageQueueRPCExecutor
    {
    private $connection;
    public function __construct(RabbitMqConnection $connection)
    {
    $this->connection = $connection;
    }
    public function runOnVhostWithMap(
    string $vhost,
    QueueMapper $queueMapper)
    {
    $this->connection->setVHost($vhost);
    $messageExecutor = new MessageQueueExecutor(
    new RabbitMqMessageQueueExecutor(
    $this->connection
    ),
    new JsonMessageSerializer(),
    $queueMapper
    );
    $messageExecutor->run();
    }
    }
    Read model - executor
    

    View Slide

44. class RabbitMqMessageQueueExecutor implements
    MessageQueueExecutorInterface
    {
    public function __construct(RabbitMqConnection $connection)
    {
    $this->connection = $connection;
    }
    public function run()
    {
    $channel = $this->connection->openChannel();
    foreach ($this->mapping as $queueName => $executor) {a
    $channel->bindQueueToExecutor($queueName, $executor);
    }
    $channel->listen();
    $channel->close();
    }
    public function mapQueueToExecutor(string $queueName, callable
    $executor)
    {
    $this->mapping[$queueName] = $this->getCallbackWrapper($executor);
    }
    Read model - executor
    

    View Slide

45. private function getCallbackWrapper(callable $executor)
    {
    return function (AMQPMessage $message) use ($executor)
    {
    $success = $executor(new RabbitMqMessage($message));
    if ($success) {
    $message->delivery_info['channel']->basic_ack(
    $message->delivery_info['delivery_tag']
    );
    } else {
    $message->delivery_info['channel']->basic_nack(
    $message->delivery_info['delivery_tag'],
    false,
    true
    );
    }
    return $success;
    };
    }
    Low level
    processing
    

    View Slide

46. class RabbitMqChannel
    {
    public function publish(
    AMQPMessage $message,
    string $exchangeName,
    string $routingKey
    )
    {
    $this->channel->exchange_declare($exchangeName, 'topic', false, true, false);
    $this->channel->basic_publish($message, $exchangeName, $routingKey);
    }
    public function bindQueueToExecutor(string $queueName, callable $executor)
    {
    $this->channel->queue_declare($queueName, false, true, false, false);
    $this->channel->basic_consume($queueName, '', false, false, false, false, $executor);
    }
    public function listen()
    {
    while(count($this->channel->callbacks)) {
    $this->channel->wait();
    }
    }
    }
    Low level read
    

    View Slide

47. Producer
    Connect
    Open
    channel
    Declare
    exchange
    Basic publish
    Write flow
    

    View Slide

48. Connect
    Open
    channel
    Declare queue
    Basic
    consume
    Consumer
    Read flow
    

    View Slide

49. Consumer
    Producer
    Exchange Queue
    Application
    RabbitMQ
    Routing
    binding
    

    View Slide

50. AdExpired
    AdModerated
    AdExpired
    AdModerated
    mailing_olxpl mailing_olxpt
    vhost
    

    View Slide

51. Exchange types
    direct
    topic
    fanout
    header
    Exchange
    Queue A Queue B
    mailer.ad.expired mailer.ad.moderated
    mailer.ad.expired
    

    View Slide

52. View Slide

53. Exchange types
    direct
    topic
    fanout
    headers
    Exchange
    Queue A Queue B
    mailer.ad.expired
    mailer.ad.* mailer.user.*
    

    View Slide

54. Exchange types
    direct
    topic
    fanout
    headers
    Exchange
    Mailer queue Stats queue
    mailer.ad.expired
    mailer.ad.* mailer.#
    

    View Slide

55. Exchange types
    direct
    topic
    fanout
    headers
    Exchange
    Mailer queue Mail stats queue
    mailer.ad.expired
    

    View Slide

56. Exchange types
    direct
    topic
    fanout
    headers
    Exchange
    Mailer queue Mail stats queue
    param1:1 param2:2
    

    View Slide

57. Exchange
    Mailer queue Mail stats queue
    mailer.ad.expired
    Delayed delivery
    Exchange
    Exchange to
    exchange binding
    rabbitmq-plugins enable rabbitmq_delayed_message_exchange
    

    View Slide

58. Delaying message
    delivery
    $channel->exchange_declare('delayed_exchange', 'x-delayed-message',
    false, true, false, false, false,
    new AMQPTable(array(
    "x-delayed-type" => "direct"
    )));
    $hdrs = new AMQPTable(array("x-delay" => 300));
    $msg = new AMQPMessage('5 minute delayed task',
    array('delivery_mode' => 2)
    );
    $msg->set('application_headers', $hdrs);
    $channel->basic_publish($msg, 'delayed_exchange', 'routing_key');
    

    View Slide

59. Exchange
    Mailer queue Mail stats queue
    mailer.ad.expired
    Delayed delivery
    Exchange
    Exchange to
    exchange binding
    Delayed retries
    Producer
    Consumer
    Failed task execution
    

    View Slide

60. Handling failed tasks
    Ignore repetition
    {
    "payload": {
    "adId": 1
    },
    "metadata": {
    "className": "SendAdExpiredMail",
    "retryStrategy": "ignore"
    }
    }
    

    View Slide

61. Handling failed tasks
    Controlled repetition
    {
    "payload": {
    "adId": 1
    },
    "metadata": {
    "className": "SendAdExpiredMail",
    "retryStrategy": "repeat",
    "retryLimit": "5",
    "retryCount": "0",
    }
    }
    

    View Slide

62. Handling failed tasks
    Controlled repetition
    with declared intervals
    {
    "payload": {
    "adId": 1
    },
    "metadata": {
    "className": "SendAdExpiredMail",
    "retryStrategy": "repeat",
    "retryLimit": "5",
    "retryCount": "0",
    "retryInterval": [15, 30, 60, 300, 86400],
    }
    }
    

    View Slide

63. Fire and Forget
    

    View Slide

64. Producer Consumer
    Asynchronous RPC call
    

    View Slide

65. Producer Consumer
    Asynchronous RPC call
    Asynchronous RPC response
    

    View Slide

66. Send task
    Wait for response
    Create response hash
    Open response queue
    Continue processing
    

    View Slide

67. interface ResponseRPC {
    public function getResponseQueueName(): string;
    }
    class GetUserFriends extends AbstractMessage implements ResponseRPC
    {
    public function getResponseQueueName(): string
    {
    if (!isset($this->metadata['response_queue'])) {
    $this->metadata['response_queue'] = 'user-friends-' . $this->payload['user_id'] . '_' . uniqid('', true);
    }
    return $this->metadata['response_queue'];
    }
    

    View Slide

68. $messageQueue = $this->container->get('messaging.rpc.producer.responding');
    $task = new GetUserFriends();
    $task->setId($user[UserFields::FIELD_ID]);
    $messageQueue->send($task);
    /* do some processing */
    $userFriends = $messageQueue->getAsyncResponse();
    

    View Slide

69. class RespondingMessageQueue extends MessageQueue
    {
    public function __construct( MessageQueue $writer, MessageQueueExecutorInterface $executor)
    {
    $this->writer = $writer;
    $this->executor = $executor;
    }
    public function send(MessageInterface $message)
    {
    if ($message instanceof ResponseRPC){
    $this->openResponseQueue($message);
    }
    $this->writer->send($message);
    }
    public function getAsyncResponse(): MessageInterface
    {
    $this->executor->runSingle(10);
    return $this->response;
    }
    private function openResponseQueue(ResponseRPC $message)
    {
    $message->getResponseQueueName();
    $this->executor->mapQueueToExecutor( $message->getResponseQueueName(),[$this, 'catchResponse']);
    }
    private function catchResponse(MessageInterface $message)
    {
    $this->response = $message;
    }
    }
    

    View Slide

70. Producer Consumer
    Asynchronous RPC calls
    Asynchronous RPC responses
    

    View Slide

71. user(id:27192) {
    active_ads
    observed_ads
    recommendations
    messages
    }
    GraphQL
    Sample request
    

    View Slide

72. getActiveAds
    getObservedAds
    getRecommendations
    getMessages
    Request Response
    

    View Slide

73. getActiveAds
    getObservedAds
    getRecommendations
    getMessages
    Request Response
    

    View Slide

74. $messageQueue = $this->container->get('messaging.rpc.producer.responding');
    $getActiveAds = new GetActiveAds();
    $getActiveAds->setUserId($user[UserFields::FIELD_ID]);
    $getObservedAds = new GetObservedAds();
    $getObservedAds->setUserId($user[UserFields::FIELD_ID]);
    $getRecommendations = new GetRecommendations();
    $getRecommendations->setUserId($user[UserFields::FIELD_ID]);
    $getMessages = new GetMessages();
    $getMessages->setUserId($user[UserFields::FIELD_ID]);
    $messageQueue->sendMultiple(
    [$getActiveAds, $getObservedAds, $getRecommendations, $getMessages]
    );
    /* do some processing */
    $userAggregateData = $messageQueue->getAsyncResponses();
    

    View Slide

75. ● more consumers
    ● faster consumers
    ● higher prefetch count
    

    View Slide

76. Useful links
    http://tryrabbitmq.com/
    https://github.com/php-amqplib/php-amqplib
    https://www.cloudamqp.com/
    

    View Slide

77. RabbitMQ
    Asynchronous RPC with Dawid Mazurek
    PHP DEV at
    

    View Slide