Prime Time with Messenger: Queues, Workers & more Fun!

F5dfeeef276fcfd4751f4063487a5a3f?s=47 weaverryan
November 22, 2019

Prime Time with Messenger: Queues, Workers & more Fun!

In Symfony 4.4, Messenger will lose its experimental label and officially become stable! In this talk, we'll go through Messenger from the ground-up: learning about messages, message handlers, transports and how to consume messages asynchronously via worker commands. Everything you need to make your app faster by delaying work until later.

We'll also talk about the many new features that were added to Messenger since Symfony 4.3, like retries, the failure transport and support for using Doctrine and Redis as transports.

Let's get to work with Messenger!

F5dfeeef276fcfd4751f4063487a5a3f?s=128

weaverryan

November 22, 2019
Tweet

Transcript

  1. Prime Time with Messenger:
 Queues, Workers & more Fun! with

    your friend @weaverryan
  2. > Lead of the Symfony documentation team
 > Code story-teller

    for SymfonyCasts.com > Husband of the much more talented @leannapelham symfonycasts.com twitter.com/weaverryan Yo! I’m Ryan! > Father to my much more charming son, Beckett
  3. So, Messenger is…? @weaverryan [0]

  4. @weaverryan

  5. also… @weaverryan

  6. A Component for Sending Messages @weaverryan

  7. Messenger: • A message bus • A command bus •

    An event bus • A query bus • A school bus (beep beep) • A way to run code asynchronously @weaverryan
  8. Getting Started @weaverryan [1]

  9. Stroopwafel Ordering System @weaverryan our delicious app

  10. POST /stroopwafel/order { "topping": "plain", "size": "medium", "recipient": "Nicolas" }

  11. // src/Controller/StroopwafelOrderController.php /** * @Route("/stroopwafel/order") */ public function index(Request $request)

    { $data = json_decode($request->getContent(), true); $topping = $data['topping'] ?? 'plain'; $size = $data['size'] ?? 'large'; $recipient = $data['recipient'] ?? 'Leanna'; // ... } POST /stroopwafel/order { "topping": "plain", "size": "medium", "recipient": "Nicolas" }
  12. Our Proprietary, Secret Stroopwafel Baking Machine @weaverryan

  13. public function index(Request $request) { // ... } $this->logger->info(sprintf( 'Putting

    %s dough on the waffle iron', $size )); usleep(500000); $this->logger->info('Adding some caramel'); usleep(500000); if ($topping !== 'plain') { usleep(500000); $this->logger->info(sprintf( 'Adding topping: '.$topping )); } $this->logger->info('Delivering to '.$recipient); usleep(500000);
  14. public function index(Request $request) { // ... } $this->logger->info(sprintf( 'Putting

    %s dough on the waffle iron', $size )); usleep(500000); $this->logger->info('Adding some caramel'); usleep(500000); if ($topping !== 'plain') { usleep(500000); $this->logger->info(sprintf( 'Adding topping: '.$topping )); } $this->logger->info('Delivering to '.$recipient); usleep(500000);
  15. Delicious! Can we organize? @weaverryan Organize

  16. <?php // src/Message/StroopwafelOrder.php namespace App\Message; class StroopwafelOrder { private $topping;

    private $size; private $recipient; public function __construct(string $topping, string $size, string $recipient) { $this->topping = $topping; $this->size = $size; $this->recipient = $recipient; } // getters }
  17. /** * @Route("/stroopwafel/order") */ public function index(Request $request) { //

    ... // ... } $order = new StroopwafelOrder( $topping, $size, $recipient ); $this->logger->info(sprintf( 'Putting %s dough on the waffle iron', $order->getSize() )); usleep(500000);
  18. Hmm, maybe a service for the logic? @weaverryan Organize

  19. <?php // src/MessageHandler/StroopwafelOrderHandler.php class StroopwafelOrderHandler { private $logger; public function

    __construct(LoggerInterface $logger) { $this->logger = $logger; } public function handleOrder(StroopwafelOrder $order) { $this->logger->info(sprintf('%s dough on iron', $order->getSize())); usleep(500000); $this->logger->info('Adding some caramel'); usleep(500000); // ... } }
  20. /** * @Route("/stroopwafel/order") */ public function index(Request $request, StroopwafelOrderHandler $handler)

    { $data = json_decode($request->getContent(), true); // ... $order = new StroopwafelOrder(…) $topping, $size, $recipient ); $handler->handleOrder($order); return new Response(204); }
  21. Umm… so… Messenger? @weaverryan [2]

  22. @weaverryan Calling a Service Hey StroopwafelHandler! Please make me a

    plain, small Stroopwafel for Nicolas! $stroopHandler->handleOrder( 'plain', 'small', 'Nicolas' );
  23. @weaverryan Using a "message" Hey StroopwafelHandler! Please handle this StroopwafelOrder!

    $order = new StroopwafelOrder(…) $stroopHandler->handleOrder( $order );
  24. @weaverryan Message Bus Hey "message bus"! Call whoever handles this

    message $order = new StroopwafelOrder(…) $messageBus->dispatch( $order );
  25. Bus @weaverryan BrewCoffee CleanupBathroom Chef Barista Custodian BakeStroopwafel

  26. > composer require messenger

  27. How does Messenger know to call StroopwafelOrderHandler when we dispatch

    a StroopwafelOrder? @weaverryan Connecting messages & handlers
  28. <?php namespace App\MessageHandler; use Symfony\Component\Messenger\Handler\MessageHandlerInterface; class StroopwafelOrderHandler implements MessageHandlerInterface {

    // ... } autoconfigure: Tell Messenger this is a handler
  29. <?php namespace App\MessageHandler; use Symfony\Component\Messenger\Handler\MessageHandlerInterface; class StroopwafelOrderHandler implements MessageHandlerInterface {

    // ... public function __invoke(StroopwafelOrder $order) { // … } } Type-hint: Tell Messenger what object it handles
  30. public function index(MessageBusInterface $messageBus) { // ... $order = new

    StroopwafelOrder( $topping, $size, $recipient ); $stroopHandler->handleOrder($order); return new Response(204); } $messageBus->dispatch($order); Call the bus
  31. So… async? @weaverryan [3]

  32. @weaverryan Centralizing "messages" through some system (bus) has powerful consequences

  33. Centralization Possibilities • Log each time a "message" is dispatched

    • Wrap each handler inside a transaction • Store the message somewhere else… then run some command to read & handle it later
  34. Transports: where to send/read messages #.env ###> symfony/messenger ### #

    Choose one of the transports below # MESSENGER_TRANSPORT_DSN=amqp://guest@localhost:5672/%2f/messages # MESSENGER_TRANSPORT_DSN=doctrine://default # MESSENGER_TRANSPORT_DSN=redis://localhost:6379/messages ###< symfony/messenger ###
  35. > composer require doctrine

  36. Doctrine transport #.env ###> symfony/messenger ### # Choose one of

    the transports below # MESSENGER_TRANSPORT_DSN=amqp://guest@localhost:5672/%2f/messages MESSENGER_TRANSPORT_DSN=doctrine://default # MESSENGER_TRANSPORT_DSN=redis://localhost:6379/messages ###< symfony/messenger ### New in 4.3
  37. Doctrine transport # config/packages/messenger.yaml framework: messenger: transports: async: '%env(MESSENGER_TRANSPORT_DSN)%'

  38. @weaverryan Transport Ready! But we're not using it yet (StoopwafelOrder

    is still handled sync)
  39. Routing to a transport # config/packages/messenger.yaml framework: messenger: transports: async:

    '%env(MESSENGER_TRANSPORT_DSN)%' routing: 'App\Message\StroopwafelOrder': async
  40. POST /stroopwafel/order { "topping": "plain", "size": "medium", "recipient": "Nicolas" }

    … no Stroopwafel is baked!
  41. SELECT * FROM messenger_messages; * Table is created automatically

  42. How do we tell Messenger that we *are* ready to

    handle the message? (worker) @weaverryan Handling/Consuming
  43. @weaverryan > bin/console messenger:consume -vv

  44. @weaverryan > bin/console messenger:consume -vv

  45. @weaverryan Messenger, Done!

  46. THANK YOU!

  47. Working with Workers @weaverryan [4]

  48. @weaverryan > bin/console messenger:consume -vv Do we run this manually

    on production?
  49. @weaverryan Nope! Your worker WILL die at some point. You

    WANT it to die… on your terms
  50. @weaverryan > bin/console messenger:consume --help

  51. @weaverryan Running the "worker" • Supervisord (see Symfony docs) •

    Automatically via your PaaS (e.g. SymfonyCloud, Heroku) • Run as many workers as you need
  52. @weaverryan Nope! When you deploy code changes, will your running

    workers see the new code?
  53. @weaverryan > bin/console messenger:stop-workers (graceful exit) New in 4.3

  54. @weaverryan > symfony run -d \ symfony console messenger:consume \

    --watch=config,src,templates,vendor What about Locally?
  55. Envelopes & Stamps @weaverryan [5]

  56. Stamps and Envelopes @weaverryan Stamps

  57. Adding an Envelope public function orderStroop(MessageBusInterface $messageBus) { // ...

    $envelope = new Envelope($order); $messageBus->dispatch($envelope); // ... }
  58. Adding Some Stamps public function orderStroop(MessageBusInterface $messageBus) { // ...

    $envelope = new Envelope($order); $messageBus->dispatch($envelope, [ new DelayStamp(5000) ]); // ... }
  59. Middleware Add Stamps public function orderStroop(MessageBusInterface $messageBus) { // ...

    $envelope = new Envelope($order); $envelope = $messageBus->dispatch($envelope, [ new DelayStamp(5000) ]); dump($envelope); // ... }
  60. None
  61. Priority Transports @weaverryan [5] New in 4.3

  62. @weaverryan A queue of messages •StroopwafelOrder •CoffeeOrder •StroopwafelOrder •CleanBathroom •CoffeeOrder

    •WashDishes }In what order should these be handled?
  63. Add a 2nd Transport framework: messenger: transports: async: '%env(MESSENGER_TRANSPORT_DSN)%' async_high_priority:

    dsn: '%env(MESSENGER_TRANSPORT_DSN)%' options: queue_name: high_priority
  64. Route by priority framework: messenger: transports: async: # ... async_high_priority:

    #... routing: 'App\Message\StroopwafelOrder': async_high_priority 'App\Message\CoffeeOrder': async_high_priority 'App\Message\CleanBathroom': async 'App\Message\WashDishes': async
  65. None
  66. Failures & Retries @weaverryan [6] New in 4.3

  67. @weaverryan What should happen if an error occurs while handling?

  68. // ... class CoffeeOrderHandler implements MessageHandlerInterface { // ... public

    function __invoke(CoffeeOrder $order) { if (rand(0, 10) > 2) { throw new OutOfBeansException('Ahhh!'); } $this->logger->info('I made some coffee for you'); } }
  69. None
  70. @weaverryan And… if it keeps failing?

  71. async_high_priority: dsn: '...' retry_strategy: # RetryStrategyInterface # service: null max_retries:

    3 delay: 1000 multiplier: 2 max_delay: 0
  72. Failure Transport @weaverryan [7] New in 4.3

  73. framework: messenger: failure_transport: failed transports: async: # ... async_high_priority: #…

    failed: 'doctrine://default?queue_name=failed'
  74. None
  75. None
  76. None
  77. None
  78. None
  79. Fake Transports @weaverryan [8] New in 4.3

  80. Could I handle some/all messages sync while developing to make

    life easier
  81. # .env MESSENGER_TRANSPORT_DSN=sync://

  82. I don't want my messages to be sent to a

    queue while testing
  83. class StroopwafelControllerTest extends WebTestCase { public function testOrderStroopwafel() { $client

    = static::createClient(); $client->request( 'POST', '/stroopwafel/order', json_encode(['topping' => 'plain']) ); $this->assertResponseIsSuccessful(); } }
  84. # .env.test MESSENGER_TRANSPORT_DSN=in-memory://

  85. public function testOrderStroopwafel() { // ... $this->assertResponseIsSuccessful(); /** @var InMemoryTransport

    $transport */ $transport = self::$container ->get('messenger.transport.async'); } $this->assertCount(1, $transport->getSent()); $this->assertInstanceOf( StroopwafelOrder::class, $transport->getSent()[0]->getMessage() );
  86. Mailer Support @weaverryan [9] New in 4.3

  87. Could we send our emails async through Messenger?

  88. framework: messenger: # ... routing: # ... 'Symfony\Component\Mailer\Messenger': async

  89. What Else? @weaverryan [10]

  90. @weaverryan A queue of new features •Redis Transport (4.3) (delay

    support 4.4) •PhpSerializer (4.3) •Worker Events (4.3) •Auto Clear EntityManager (4.4) •from_transport (4.3) •API Platform integration • … and …
  91. It's STABLE

  92. Putting it all Together @weaverryan [fin]

  93. @weaverryan Messenger… Allows you to use a message bus

  94. @weaverryan Messenger… Allows you to save work for later

  95. @weaverryan Messenger… Allows you to retry and introspect errors

  96. @weaverryan Messenger… Is Ready for Production!

  97. So use it! @weaverryan

  98. Ryan Weaver @weaverryan THANK YOU! Messenger Tutorial https://symfonycasts.com/screencast/messenger