Prime Time with Messenger: Queues, Workers & more Fun!

In Symfony 4.4, Messenger will lose its experimental label and officially become stable! In this talk, we'll go through Messenger from the ground-up: learning about messages, message handlers, transports and how to consume messages asynchronously via worker commands. Everything you need to make your app faster by delaying work until later.

We'll also talk about the many new features that were added to Messenger since Symfony 4.3, like retries, the failure transport and support for using Doctrine and Redis as transports.

Let's get to work with Messenger!

Transcript

1. Prime Time with Messenger:
 Queues, Workers & more Fun! with your friend @weaverryan
2. > Lead of the Symfony documentation team
 > Code story-teller for SymfonyCasts.com > Husband of the much more talented @leannapelham symfonycasts.com twitter.com/weaverryan Yo! I’m Ryan! > Father to my much more charming son, Beckett
3. So, Messenger is…? @weaverryan [0]
4. @weaverryan
5. also… @weaverryan
6. A Component for Sending Messages @weaverryan
7. Messenger: • A message bus • A command bus • An event bus • A query bus • A school bus (beep beep) • A way to run code asynchronously @weaverryan
8. Getting Started @weaverryan [1]
9. Stroopwafel Ordering System @weaverryan our delicious app
10. POST /stroopwafel/order { "topping": "plain", "size": "medium", "recipient": "Nicolas" }
11. // src/Controller/StroopwafelOrderController.php /** * @Route("/stroopwafel/order") */ public function index(Request $request) { $data = json_decode($request->getContent(), true); $topping = $data['topping'] ?? 'plain'; $size = $data['size'] ?? 'large'; $recipient = $data['recipient'] ?? 'Leanna'; // ... } POST /stroopwafel/order { "topping": "plain", "size": "medium", "recipient": "Nicolas" }
12. Our Proprietary, Secret Stroopwafel Baking Machine @weaverryan
13. public function index(Request $request) { // ... } $this->logger->info(sprintf( 'Putting %s dough on the waffle iron', $size )); usleep(500000); $this->logger->info('Adding some caramel'); usleep(500000); if ($topping !== 'plain') { usleep(500000); $this->logger->info(sprintf( 'Adding topping: '.$topping )); } $this->logger->info('Delivering to '.$recipient); usleep(500000);
14. public function index(Request $request) { // ... } $this->logger->info(sprintf( 'Putting %s dough on the waffle iron', $size )); usleep(500000); $this->logger->info('Adding some caramel'); usleep(500000); if ($topping !== 'plain') { usleep(500000); $this->logger->info(sprintf( 'Adding topping: '.$topping )); } $this->logger->info('Delivering to '.$recipient); usleep(500000);
15. Delicious! Can we organize? @weaverryan Organize
16. <?php // src/Message/StroopwafelOrder.php namespace App\Message; class StroopwafelOrder { private $topping; private $size; private $recipient; public function __construct(string $topping, string $size, string $recipient) { $this->topping = $topping; $this->size = $size; $this->recipient = $recipient; } // getters }
17. /** * @Route("/stroopwafel/order") */ public function index(Request $request) { // ... // ... } $order = new StroopwafelOrder( $topping, $size, $recipient ); $this->logger->info(sprintf( 'Putting %s dough on the waffle iron', $order->getSize() )); usleep(500000);
18. Hmm, maybe a service for the logic? @weaverryan Organize
19. <?php // src/MessageHandler/StroopwafelOrderHandler.php class StroopwafelOrderHandler { private $logger; public function __construct(LoggerInterface $logger) { $this->logger = $logger; } public function handleOrder(StroopwafelOrder $order) { $this->logger->info(sprintf('%s dough on iron', $order->getSize())); usleep(500000); $this->logger->info('Adding some caramel'); usleep(500000); // ... } }
20. /** * @Route("/stroopwafel/order") */ public function index(Request $request, StroopwafelOrderHandler $handler) { $data = json_decode($request->getContent(), true); // ... $order = new StroopwafelOrder(…) $topping, $size, $recipient ); $handler->handleOrder($order); return new Response(204); }
21. Umm… so… Messenger? @weaverryan [2]
22. @weaverryan Calling a Service Hey StroopwafelHandler! Please make me a plain, small Stroopwafel for Nicolas! $stroopHandler->handleOrder( 'plain', 'small', 'Nicolas' );
23. @weaverryan Using a "message" Hey StroopwafelHandler! Please handle this StroopwafelOrder! $order = new StroopwafelOrder(…) $stroopHandler->handleOrder( $order );
24. @weaverryan Message Bus Hey "message bus"! Call whoever handles this message $order = new StroopwafelOrder(…) $messageBus->dispatch( $order );
25. Bus @weaverryan BrewCoffee CleanupBathroom Chef Barista Custodian BakeStroopwafel
26. > composer require messenger
27. How does Messenger know to call StroopwafelOrderHandler when we dispatch a StroopwafelOrder? @weaverryan Connecting messages & handlers
28. <?php namespace App\MessageHandler; use Symfony\Component\Messenger\Handler\MessageHandlerInterface; class StroopwafelOrderHandler implements MessageHandlerInterface { // ... } autoconfigure: Tell Messenger this is a handler
29. <?php namespace App\MessageHandler; use Symfony\Component\Messenger\Handler\MessageHandlerInterface; class StroopwafelOrderHandler implements MessageHandlerInterface { // ... public function __invoke(StroopwafelOrder $order) { // … } } Type-hint: Tell Messenger what object it handles
30. public function index(MessageBusInterface $messageBus) { // ... $order = new StroopwafelOrder( $topping, $size, $recipient ); $stroopHandler->handleOrder($order); return new Response(204); } $messageBus->dispatch($order); Call the bus
31. So… async? @weaverryan [3]
32. @weaverryan Centralizing "messages" through some system (bus) has powerful consequences
33. Centralization Possibilities • Log each time a "message" is dispatched • Wrap each handler inside a transaction • Store the message somewhere else… then run some command to read & handle it later
34. Transports: where to send/read messages #.env ###> symfony/messenger ### # Choose one of the transports below # MESSENGER_TRANSPORT_DSN=amqp://guest@localhost:5672/%2f/messages # MESSENGER_TRANSPORT_DSN=doctrine://default # MESSENGER_TRANSPORT_DSN=redis://localhost:6379/messages ###< symfony/messenger ###
35. > composer require doctrine
36. Doctrine transport #.env ###> symfony/messenger ### # Choose one of the transports below # MESSENGER_TRANSPORT_DSN=amqp://guest@localhost:5672/%2f/messages MESSENGER_TRANSPORT_DSN=doctrine://default # MESSENGER_TRANSPORT_DSN=redis://localhost:6379/messages ###< symfony/messenger ### New in 4.3
37. Doctrine transport # config/packages/messenger.yaml framework: messenger: transports: async: '%env(MESSENGER_TRANSPORT_DSN)%'
38. @weaverryan Transport Ready! But we're not using it yet (StoopwafelOrder is still handled sync)
39. Routing to a transport # config/packages/messenger.yaml framework: messenger: transports: async: '%env(MESSENGER_TRANSPORT_DSN)%' routing: 'App\Message\StroopwafelOrder': async
40. POST /stroopwafel/order { "topping": "plain", "size": "medium", "recipient": "Nicolas" } … no Stroopwafel is baked!
41. SELECT * FROM messenger_messages; * Table is created automatically
42. How do we tell Messenger that we *are* ready to handle the message? (worker) @weaverryan Handling/Consuming
43. @weaverryan > bin/console messenger:consume -vv
44. @weaverryan > bin/console messenger:consume -vv
45. @weaverryan Messenger, Done!
46. THANK YOU!
47. Working with Workers @weaverryan [4]
48. @weaverryan > bin/console messenger:consume -vv Do we run this manually on production?
49. @weaverryan Nope! Your worker WILL die at some point. You WANT it to die… on your terms
50. @weaverryan > bin/console messenger:consume --help
51. @weaverryan Running the "worker" • Supervisord (see Symfony docs) • Automatically via your PaaS (e.g. SymfonyCloud, Heroku) • Run as many workers as you need
52. @weaverryan Nope! When you deploy code changes, will your running workers see the new code?
53. @weaverryan > bin/console messenger:stop-workers (graceful exit) New in 4.3
54. @weaverryan > symfony run -d \ symfony console messenger:consume \ --watch=config,src,templates,vendor What about Locally?
55. Envelopes & Stamps @weaverryan [5]
56. Stamps and Envelopes @weaverryan Stamps
57. Adding an Envelope public function orderStroop(MessageBusInterface $messageBus) { // ... $envelope = new Envelope($order); $messageBus->dispatch($envelope); // ... }
58. Adding Some Stamps public function orderStroop(MessageBusInterface $messageBus) { // ... $envelope = new Envelope($order); $messageBus->dispatch($envelope, [ new DelayStamp(5000) ]); // ... }
59. Middleware Add Stamps public function orderStroop(MessageBusInterface $messageBus) { // ... $envelope = new Envelope($order); $envelope = $messageBus->dispatch($envelope, [ new DelayStamp(5000) ]); dump($envelope); // ... }
61. Priority Transports @weaverryan [5] New in 4.3
62. @weaverryan A queue of messages •StroopwafelOrder •CoffeeOrder •StroopwafelOrder •CleanBathroom •CoffeeOrder •WashDishes }In what order should these be handled?
63. Add a 2nd Transport framework: messenger: transports: async: '%env(MESSENGER_TRANSPORT_DSN)%' async_high_priority: dsn: '%env(MESSENGER_TRANSPORT_DSN)%' options: queue_name: high_priority
64. Route by priority framework: messenger: transports: async: # ... async_high_priority: #... routing: 'App\Message\StroopwafelOrder': async_high_priority 'App\Message\CoffeeOrder': async_high_priority 'App\Message\CleanBathroom': async 'App\Message\WashDishes': async
66. Failures & Retries @weaverryan [6] New in 4.3
67. @weaverryan What should happen if an error occurs while handling?
68. // ... class CoffeeOrderHandler implements MessageHandlerInterface { // ... public function __invoke(CoffeeOrder $order) { if (rand(0, 10) > 2) { throw new OutOfBeansException('Ahhh!'); } $this->logger->info('I made some coffee for you'); } }
70. @weaverryan And… if it keeps failing?
71. async_high_priority: dsn: '...' retry_strategy: # RetryStrategyInterface # service: null max_retries: 3 delay: 1000 multiplier: 2 max_delay: 0
72. Failure Transport @weaverryan [7] New in 4.3
73. framework: messenger: failure_transport: failed transports: async: # ... async_high_priority: #… failed: 'doctrine://default?queue_name=failed'
79. Fake Transports @weaverryan [8] New in 4.3
80. Could I handle some/all messages sync while developing to make life easier
81. # .env MESSENGER_TRANSPORT_DSN=sync://
82. I don't want my messages to be sent to a queue while testing
83. class StroopwafelControllerTest extends WebTestCase { public function testOrderStroopwafel() { $client = static::createClient(); $client->request( 'POST', '/stroopwafel/order', json_encode(['topping' => 'plain']) ); $this->assertResponseIsSuccessful(); } }
84. # .env.test MESSENGER_TRANSPORT_DSN=in-memory://
85. public function testOrderStroopwafel() { // ... $this->assertResponseIsSuccessful(); /** @var InMemoryTransport $transport */ $transport = self::$container ->get('messenger.transport.async'); } $this->assertCount(1, $transport->getSent()); $this->assertInstanceOf( StroopwafelOrder::class, $transport->getSent()[0]->getMessage() );
86. Mailer Support @weaverryan [9] New in 4.3
87. Could we send our emails async through Messenger?
88. framework: messenger: # ... routing: # ... 'Symfony\Component\Mailer\Messenger': async
89. What Else? @weaverryan [10]
90. @weaverryan A queue of new features •Redis Transport (4.3) (delay support 4.4) •PhpSerializer (4.3) •Worker Events (4.3) •Auto Clear EntityManager (4.4) •from_transport (4.3) •API Platform integration • … and …
91. It's STABLE
92. Putting it all Together @weaverryan [fin]
93. @weaverryan Messenger… Allows you to use a message bus
94. @weaverryan Messenger… Allows you to save work for later
95. @weaverryan Messenger… Allows you to retry and introspect errors
96. @weaverryan Messenger… Is Ready for Production!
97. So use it! @weaverryan
98. Ryan Weaver @weaverryan THANK YOU! Messenger Tutorial https://symfonycasts.com/screencast/messenger