Prime Time with Messenger: Queues, Workers & more Fun!

In Symfony 4.4, Messenger will lose its experimental label and officially become stable! In this talk, we'll go through Messenger from the ground-up: learning about messages, message handlers, transports and how to consume messages asynchronously via worker commands. Everything you need to make your app faster by delaying work until later.

We'll also talk about the many new features that were added to Messenger since Symfony 4.3, like retries, the failure transport and support for using Doctrine and Redis as transports.

Let's get to work with Messenger!

Transcript

1. Prime Time with Messenger:

   Queues, Workers & more Fun!
   with your friend @weaverryan
   

   View Slide

2. > Lead of the Symfony documentation team

   > Code story-teller
   for SymfonyCasts.com
   > Husband of the much more
   talented @leannapelham
   symfonycasts.com
   twitter.com/weaverryan
   Yo! I’m Ryan!
   > Father to my much more
   charming son, Beckett
   

   View Slide

3. So, Messenger is…?
   @weaverryan
   [0]
   

   View Slide

4. @weaverryan
   

   View Slide

5. also…
   @weaverryan
   

   View Slide

6. A Component for
   Sending Messages
   @weaverryan
   

   View Slide

7. Messenger:
   • A message bus
   • A command bus
   • An event bus
   • A query bus
   • A school bus (beep beep)
   • A way to run code asynchronously
   @weaverryan
   

   View Slide

8. Getting Started
   @weaverryan
   [1]
   

   View Slide

9. Stroopwafel
   Ordering System
   @weaverryan
   our delicious app
   

   View Slide

10. POST /stroopwafel/order
    {
    "topping": "plain",
    "size": "medium",
    "recipient": "Nicolas"
    }
    

    View Slide

11. // src/Controller/StroopwafelOrderController.php
    /**
    * @Route("/stroopwafel/order")
    */
    public function index(Request $request)
    {
    $data = json_decode($request->getContent(), true);
    $topping = $data['topping'] ?? 'plain';
    $size = $data['size'] ?? 'large';
    $recipient = $data['recipient'] ?? 'Leanna';
    // ...
    }
    POST /stroopwafel/order
    {
    "topping": "plain",
    "size": "medium",
    "recipient": "Nicolas"
    }
    

    View Slide

12. Our Proprietary,
    Secret Stroopwafel
    Baking Machine
    @weaverryan
    

    View Slide

13. public function index(Request $request)
    {
    // ...
    }
    $this->logger->info(sprintf(
    'Putting %s dough on the waffle iron',
    $size
    ));
    usleep(500000);
    $this->logger->info('Adding some caramel');
    usleep(500000);
    if ($topping !== 'plain') {
    usleep(500000);
    $this->logger->info(sprintf(
    'Adding topping: '.$topping
    ));
    }
    $this->logger->info('Delivering to '.$recipient);
    usleep(500000);
    

    View Slide

14. public function index(Request $request)
    {
    // ...
    }
    $this->logger->info(sprintf(
    'Putting %s dough on the waffle iron',
    $size
    ));
    usleep(500000);
    $this->logger->info('Adding some caramel');
    usleep(500000);
    if ($topping !== 'plain') {
    usleep(500000);
    $this->logger->info(sprintf(
    'Adding topping: '.$topping
    ));
    }
    $this->logger->info('Delivering to '.$recipient);
    usleep(500000);
    

    View Slide

15. Delicious!
    Can we organize?
    @weaverryan
    Organize
    

    View Slide

16. // src/Message/StroopwafelOrder.php
    namespace App\Message;
    class StroopwafelOrder
    {
    private $topping;
    private $size;
    private $recipient;
    public function __construct(string $topping, string $size, string $recipient)
    {
    $this->topping = $topping;
    $this->size = $size;
    $this->recipient = $recipient;
    }
    // getters
    }
    

    View Slide

17. /**
    * @Route("/stroopwafel/order")
    */
    public function index(Request $request)
    {
    // ...
    // ...
    }
    $order = new StroopwafelOrder(
    $topping,
    $size,
    $recipient
    );
    $this->logger->info(sprintf(
    'Putting %s dough on the waffle iron', $order->getSize()
    ));
    usleep(500000);
    

    View Slide

18. Hmm, maybe a
    service for the logic?
    @weaverryan
    Organize
    

    View Slide

19. // src/MessageHandler/StroopwafelOrderHandler.php
    class StroopwafelOrderHandler
    {
    private $logger;
    public function __construct(LoggerInterface $logger)
    {
    $this->logger = $logger;
    }
    public function handleOrder(StroopwafelOrder $order)
    {
    $this->logger->info(sprintf('%s dough on iron', $order->getSize()));
    usleep(500000);
    $this->logger->info('Adding some caramel');
    usleep(500000);
    // ...
    }
    }
    

    View Slide

20. /**
    * @Route("/stroopwafel/order")
    */
    public function index(Request $request, StroopwafelOrderHandler $handler)
    {
    $data = json_decode($request->getContent(), true);
    // ...
    $order = new StroopwafelOrder(…)
    $topping,
    $size,
    $recipient
    );
    $handler->handleOrder($order);
    return new Response(204);
    }
    

    View Slide

21. Umm… so…
    Messenger?
    @weaverryan
    [2]
    

    View Slide

22. @weaverryan
    Calling a Service
    Hey StroopwafelHandler!
    Please make me a plain, small
    Stroopwafel for Nicolas!
    $stroopHandler->handleOrder(
    'plain',
    'small',
    'Nicolas'
    );
    

    View Slide

23. @weaverryan
    Using a "message"
    Hey StroopwafelHandler!
    Please handle this
    StroopwafelOrder!
    $order = new StroopwafelOrder(…)
    $stroopHandler->handleOrder(
    $order
    );
    

    View Slide

24. @weaverryan
    Message Bus
    Hey "message bus"!
    Call whoever handles
    this message
    $order = new StroopwafelOrder(…)
    $messageBus->dispatch(
    $order
    );
    

    View Slide

25. Bus
    @weaverryan
    BrewCoffee
    CleanupBathroom Chef
    Barista
    Custodian
    BakeStroopwafel
    

    View Slide

26. > composer require messenger
    

    View Slide

27. How does Messenger know to call
    StroopwafelOrderHandler when
    we dispatch a StroopwafelOrder?
    @weaverryan
    Connecting messages & handlers
    

    View Slide

28. namespace App\MessageHandler;
    use Symfony\Component\Messenger\Handler\MessageHandlerInterface;
    class StroopwafelOrderHandler implements MessageHandlerInterface
    {
    // ...
    }
    autoconfigure:
    Tell Messenger this is a handler
    

    View Slide

29. namespace App\MessageHandler;
    use Symfony\Component\Messenger\Handler\MessageHandlerInterface;
    class StroopwafelOrderHandler implements MessageHandlerInterface
    {
    // ...
    public function __invoke(StroopwafelOrder $order)
    {
    // …
    }
    }
    Type-hint:
    Tell Messenger what object it handles
    

    View Slide

30. public function index(MessageBusInterface $messageBus)
    {
    // ...
    $order = new StroopwafelOrder(
    $topping,
    $size,
    $recipient
    );
    $stroopHandler->handleOrder($order);
    return new Response(204);
    }
    $messageBus->dispatch($order);
    Call the bus
    

    View Slide

31. So… async?
    @weaverryan
    [3]
    

    View Slide

32. @weaverryan
    Centralizing "messages" through
    some system (bus) has powerful
    consequences
    

    View Slide

33. Centralization Possibilities
    • Log each time a "message" is dispatched
    • Wrap each handler inside a transaction
    • Store the message somewhere else… then run
    some command to read & handle it later
    

    View Slide

34. Transports: where to send/read messages
    #.env
    ###> symfony/messenger ###
    # Choose one of the transports below
    # MESSENGER_TRANSPORT_DSN=amqp://guest@localhost:5672/%2f/messages
    # MESSENGER_TRANSPORT_DSN=doctrine://default
    # MESSENGER_TRANSPORT_DSN=redis://localhost:6379/messages
    ###< symfony/messenger ###
    

    View Slide

35. > composer require doctrine
    

    View Slide

36. Doctrine transport
    #.env
    ###> symfony/messenger ###
    # Choose one of the transports below
    # MESSENGER_TRANSPORT_DSN=amqp://guest@localhost:5672/%2f/messages
    MESSENGER_TRANSPORT_DSN=doctrine://default
    # MESSENGER_TRANSPORT_DSN=redis://localhost:6379/messages
    ###< symfony/messenger ###
    New in 4.3
    

    View Slide

37. Doctrine transport
    # config/packages/messenger.yaml
    framework:
    messenger:
    transports:
    async: '%env(MESSENGER_TRANSPORT_DSN)%'
    

    View Slide

38. @weaverryan
    Transport Ready!
    But we're not using it yet
    (StoopwafelOrder is still handled sync)
    

    View Slide

39. Routing to a transport
    # config/packages/messenger.yaml
    framework:
    messenger:
    transports:
    async: '%env(MESSENGER_TRANSPORT_DSN)%'
    routing:
    'App\Message\StroopwafelOrder': async
    

    View Slide

40. POST /stroopwafel/order
    {
    "topping": "plain",
    "size": "medium",
    "recipient": "Nicolas"
    }
    … no Stroopwafel is baked!
    

    View Slide

41. SELECT * FROM messenger_messages;
    * Table is created automatically
    

    View Slide

42. How do we tell Messenger that we
    *are* ready to handle the message?
    (worker)
    @weaverryan
    Handling/Consuming
    

    View Slide

43. @weaverryan
    > bin/console messenger:consume -vv
    

    View Slide

44. @weaverryan
    > bin/console messenger:consume -vv
    

    View Slide

45. @weaverryan
    Messenger, Done!
    

    View Slide

46. THANK YOU!
    

    View Slide

47. Working with Workers
    @weaverryan
    [4]
    

    View Slide

48. @weaverryan
    > bin/console messenger:consume -vv
    Do we run this manually on production?
    

    View Slide

49. @weaverryan
    Nope!
    Your worker WILL die at some point.
    You WANT it to die… on your terms
    

    View Slide

50. @weaverryan
    > bin/console messenger:consume --help
    

    View Slide

51. @weaverryan
    Running the "worker"
    • Supervisord (see Symfony docs)
    • Automatically via your PaaS (e.g. SymfonyCloud,
    Heroku)
    • Run as many workers as you need
    

    View Slide

52. @weaverryan
    Nope!
    When you deploy code changes, will
    your running workers see the new code?
    

    View Slide

53. @weaverryan
    > bin/console messenger:stop-workers
    (graceful exit)
    New in 4.3
    

    View Slide

54. @weaverryan
    > symfony run -d \
    symfony console messenger:consume \
    --watch=config,src,templates,vendor
    What about Locally?
    

    View Slide

55. Envelopes & Stamps
    @weaverryan
    [5]
    

    View Slide

56. Stamps and Envelopes
    @weaverryan
    Stamps
    

    View Slide

57. Adding an Envelope
    public function orderStroop(MessageBusInterface $messageBus)
    {
    // ...
    $envelope = new Envelope($order);
    $messageBus->dispatch($envelope);
    // ...
    }
    

    View Slide

58. Adding Some Stamps
    public function orderStroop(MessageBusInterface $messageBus)
    {
    // ...
    $envelope = new Envelope($order);
    $messageBus->dispatch($envelope, [
    new DelayStamp(5000)
    ]);
    // ...
    }
    

    View Slide

59. Middleware Add Stamps
    public function orderStroop(MessageBusInterface $messageBus)
    {
    // ...
    $envelope = new Envelope($order);
    $envelope = $messageBus->dispatch($envelope, [
    new DelayStamp(5000)
    ]);
    dump($envelope);
    // ...
    }
    

    View Slide

60. View Slide

61. Priority Transports
    @weaverryan
    [5]
    New in 4.3
    

    View Slide

62. @weaverryan
    A queue of messages
    •StroopwafelOrder
    •CoffeeOrder
    •StroopwafelOrder
    •CleanBathroom
    •CoffeeOrder
    •WashDishes
    }In what order should
    these be handled?
    

    View Slide

63. Add a 2nd Transport
    framework:
    messenger:
    transports:
    async: '%env(MESSENGER_TRANSPORT_DSN)%'
    async_high_priority:
    dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
    options:
    queue_name: high_priority
    

    View Slide

64. Route by priority
    framework:
    messenger:
    transports:
    async: # ...
    async_high_priority: #...
    routing:
    'App\Message\StroopwafelOrder': async_high_priority
    'App\Message\CoffeeOrder': async_high_priority
    'App\Message\CleanBathroom': async
    'App\Message\WashDishes': async
    

    View Slide

65. View Slide

66. Failures & Retries
    @weaverryan
    [6]
    New in 4.3
    

    View Slide

67. @weaverryan
    What should happen if an error
    occurs while handling?
    

    View Slide

68. // ...
    class CoffeeOrderHandler implements MessageHandlerInterface
    {
    // ...
    public function __invoke(CoffeeOrder $order)
    {
    if (rand(0, 10) > 2) {
    throw new OutOfBeansException('Ahhh!');
    }
    $this->logger->info('I made some coffee for you');
    }
    }
    

    View Slide

69. View Slide

70. @weaverryan
    And… if it keeps failing?
    

    View Slide

71. async_high_priority:
    dsn: '...'
    retry_strategy:
    # RetryStrategyInterface
    # service: null
    max_retries: 3
    delay: 1000
    multiplier: 2
    max_delay: 0
    

    View Slide

72. Failure Transport
    @weaverryan
    [7]
    New in 4.3
    

    View Slide

73. framework:
    messenger:
    failure_transport: failed
    transports:
    async: # ...
    async_high_priority: #…
    failed: 'doctrine://default?queue_name=failed'
    

    View Slide

74. View Slide

75. View Slide

76. View Slide

77. View Slide

78. View Slide

79. Fake Transports
    @weaverryan
    [8]
    New in 4.3
    

    View Slide

80. Could I handle some/all messages
    sync while developing to make life easier
    

    View Slide

81. # .env
    MESSENGER_TRANSPORT_DSN=sync://
    

    View Slide

82. I don't want my messages to be
    sent to a queue while testing
    

    View Slide

83. class StroopwafelControllerTest extends WebTestCase
    {
    public function testOrderStroopwafel()
    {
    $client = static::createClient();
    $client->request(
    'POST',
    '/stroopwafel/order',
    json_encode(['topping' => 'plain'])
    );
    $this->assertResponseIsSuccessful();
    }
    }
    

    View Slide

84. # .env.test
    MESSENGER_TRANSPORT_DSN=in-memory://
    

    View Slide

85. public function testOrderStroopwafel()
    {
    // ...
    $this->assertResponseIsSuccessful();
    /** @var InMemoryTransport $transport */
    $transport = self::$container
    ->get('messenger.transport.async');
    }
    $this->assertCount(1, $transport->getSent());
    $this->assertInstanceOf(
    StroopwafelOrder::class,
    $transport->getSent()[0]->getMessage()
    );
    

    View Slide

86. Mailer Support
    @weaverryan
    [9]
    New in 4.3
    

    View Slide

87. Could we send our emails async
    through Messenger?
    

    View Slide

88. framework:
    messenger:
    # ...
    routing:
    # ...
    'Symfony\Component\Mailer\Messenger': async
    

    View Slide

89. What Else?
    @weaverryan
    [10]
    

    View Slide

90. @weaverryan
    A queue of new features
    •Redis Transport (4.3) (delay support 4.4)
    •PhpSerializer (4.3)
    •Worker Events (4.3)
    •Auto Clear EntityManager (4.4)
    •from_transport (4.3)
    •API Platform integration
    • … and …
    

    View Slide

91. It's STABLE
    

    View Slide

92. Putting it all Together
    @weaverryan
    [fin]
    

    View Slide

93. @weaverryan
    Messenger…
    Allows you to use a message bus
    

    View Slide

94. @weaverryan
    Messenger…
    Allows you to save work for later
    

    View Slide

95. @weaverryan
    Messenger…
    Allows you to retry
    and introspect errors
    

    View Slide

96. @weaverryan
    Messenger…
    Is Ready for Production!
    

    View Slide

97. So use it!
    @weaverryan
    

    View Slide

98. Ryan Weaver
    @weaverryan
    THANK YOU!
    Messenger Tutorial
    https://symfonycasts.com/screencast/messenger
    

    View Slide