$30 off During Our Annual Pro Sale. View Details »

Prime Time with Messenger: Queues, Workers & more Fun!

weaverryan
November 22, 2019

Prime Time with Messenger: Queues, Workers & more Fun!

In Symfony 4.4, Messenger will lose its experimental label and officially become stable! In this talk, we'll go through Messenger from the ground-up: learning about messages, message handlers, transports and how to consume messages asynchronously via worker commands. Everything you need to make your app faster by delaying work until later.

We'll also talk about the many new features that were added to Messenger since Symfony 4.3, like retries, the failure transport and support for using Doctrine and Redis as transports.

Let's get to work with Messenger!

weaverryan

November 22, 2019
Tweet

More Decks by weaverryan

Other Decks in Technology

Transcript

  1. Prime Time with Messenger:

    Queues, Workers & more Fun!
    with your friend @weaverryan

    View Slide

  2. > Lead of the Symfony documentation team

    > Code story-teller
    for SymfonyCasts.com
    > Husband of the much more
    talented @leannapelham
    symfonycasts.com
    twitter.com/weaverryan
    Yo! I’m Ryan!
    > Father to my much more
    charming son, Beckett

    View Slide

  3. So, Messenger is…?
    @weaverryan
    [0]

    View Slide

  4. @weaverryan

    View Slide

  5. also…
    @weaverryan

    View Slide

  6. A Component for
    Sending Messages
    @weaverryan

    View Slide

  7. Messenger:
    • A message bus
    • A command bus
    • An event bus
    • A query bus
    • A school bus (beep beep)
    • A way to run code asynchronously
    @weaverryan

    View Slide

  8. Getting Started
    @weaverryan
    [1]

    View Slide

  9. Stroopwafel
    Ordering System
    @weaverryan
    our delicious app

    View Slide

  10. POST /stroopwafel/order
    {
    "topping": "plain",
    "size": "medium",
    "recipient": "Nicolas"
    }

    View Slide

  11. // src/Controller/StroopwafelOrderController.php
    /**
    * @Route("/stroopwafel/order")
    */
    public function index(Request $request)
    {
    $data = json_decode($request->getContent(), true);
    $topping = $data['topping'] ?? 'plain';
    $size = $data['size'] ?? 'large';
    $recipient = $data['recipient'] ?? 'Leanna';
    // ...
    }
    POST /stroopwafel/order
    {
    "topping": "plain",
    "size": "medium",
    "recipient": "Nicolas"
    }

    View Slide

  12. Our Proprietary,
    Secret Stroopwafel
    Baking Machine
    @weaverryan

    View Slide

  13. public function index(Request $request)
    {
    // ...
    }
    $this->logger->info(sprintf(
    'Putting %s dough on the waffle iron',
    $size
    ));
    usleep(500000);
    $this->logger->info('Adding some caramel');
    usleep(500000);
    if ($topping !== 'plain') {
    usleep(500000);
    $this->logger->info(sprintf(
    'Adding topping: '.$topping
    ));
    }
    $this->logger->info('Delivering to '.$recipient);
    usleep(500000);

    View Slide

  14. public function index(Request $request)
    {
    // ...
    }
    $this->logger->info(sprintf(
    'Putting %s dough on the waffle iron',
    $size
    ));
    usleep(500000);
    $this->logger->info('Adding some caramel');
    usleep(500000);
    if ($topping !== 'plain') {
    usleep(500000);
    $this->logger->info(sprintf(
    'Adding topping: '.$topping
    ));
    }
    $this->logger->info('Delivering to '.$recipient);
    usleep(500000);

    View Slide

  15. Delicious!
    Can we organize?
    @weaverryan
    Organize

    View Slide

  16. // src/Message/StroopwafelOrder.php
    namespace App\Message;
    class StroopwafelOrder
    {
    private $topping;
    private $size;
    private $recipient;
    public function __construct(string $topping, string $size, string $recipient)
    {
    $this->topping = $topping;
    $this->size = $size;
    $this->recipient = $recipient;
    }
    // getters
    }

    View Slide

  17. /**
    * @Route("/stroopwafel/order")
    */
    public function index(Request $request)
    {
    // ...
    // ...
    }
    $order = new StroopwafelOrder(
    $topping,
    $size,
    $recipient
    );
    $this->logger->info(sprintf(
    'Putting %s dough on the waffle iron', $order->getSize()
    ));
    usleep(500000);

    View Slide

  18. Hmm, maybe a
    service for the logic?
    @weaverryan
    Organize

    View Slide

  19. // src/MessageHandler/StroopwafelOrderHandler.php
    class StroopwafelOrderHandler
    {
    private $logger;
    public function __construct(LoggerInterface $logger)
    {
    $this->logger = $logger;
    }
    public function handleOrder(StroopwafelOrder $order)
    {
    $this->logger->info(sprintf('%s dough on iron', $order->getSize()));
    usleep(500000);
    $this->logger->info('Adding some caramel');
    usleep(500000);
    // ...
    }
    }

    View Slide

  20. /**
    * @Route("/stroopwafel/order")
    */
    public function index(Request $request, StroopwafelOrderHandler $handler)
    {
    $data = json_decode($request->getContent(), true);
    // ...
    $order = new StroopwafelOrder(…)
    $topping,
    $size,
    $recipient
    );
    $handler->handleOrder($order);
    return new Response(204);
    }

    View Slide

  21. Umm… so…
    Messenger?
    @weaverryan
    [2]

    View Slide

  22. @weaverryan
    Calling a Service
    Hey StroopwafelHandler!
    Please make me a plain, small
    Stroopwafel for Nicolas!
    $stroopHandler->handleOrder(
    'plain',
    'small',
    'Nicolas'
    );

    View Slide

  23. @weaverryan
    Using a "message"
    Hey StroopwafelHandler!
    Please handle this
    StroopwafelOrder!
    $order = new StroopwafelOrder(…)
    $stroopHandler->handleOrder(
    $order
    );

    View Slide

  24. @weaverryan
    Message Bus
    Hey "message bus"!
    Call whoever handles
    this message
    $order = new StroopwafelOrder(…)
    $messageBus->dispatch(
    $order
    );

    View Slide

  25. Bus
    @weaverryan
    BrewCoffee
    CleanupBathroom Chef
    Barista
    Custodian
    BakeStroopwafel

    View Slide

  26. > composer require messenger

    View Slide

  27. How does Messenger know to call
    StroopwafelOrderHandler when
    we dispatch a StroopwafelOrder?
    @weaverryan
    Connecting messages & handlers

    View Slide

  28. namespace App\MessageHandler;
    use Symfony\Component\Messenger\Handler\MessageHandlerInterface;
    class StroopwafelOrderHandler implements MessageHandlerInterface
    {
    // ...
    }
    autoconfigure:
    Tell Messenger this is a handler

    View Slide

  29. namespace App\MessageHandler;
    use Symfony\Component\Messenger\Handler\MessageHandlerInterface;
    class StroopwafelOrderHandler implements MessageHandlerInterface
    {
    // ...
    public function __invoke(StroopwafelOrder $order)
    {
    // …
    }
    }
    Type-hint:
    Tell Messenger what object it handles

    View Slide

  30. public function index(MessageBusInterface $messageBus)
    {
    // ...
    $order = new StroopwafelOrder(
    $topping,
    $size,
    $recipient
    );
    $stroopHandler->handleOrder($order);
    return new Response(204);
    }
    $messageBus->dispatch($order);
    Call the bus

    View Slide

  31. So… async?
    @weaverryan
    [3]

    View Slide

  32. @weaverryan
    Centralizing "messages" through
    some system (bus) has powerful
    consequences

    View Slide

  33. Centralization Possibilities
    • Log each time a "message" is dispatched
    • Wrap each handler inside a transaction
    • Store the message somewhere else… then run
    some command to read & handle it later

    View Slide

  34. Transports: where to send/read messages
    #.env
    ###> symfony/messenger ###
    # Choose one of the transports below
    # MESSENGER_TRANSPORT_DSN=amqp://guest@localhost:5672/%2f/messages
    # MESSENGER_TRANSPORT_DSN=doctrine://default
    # MESSENGER_TRANSPORT_DSN=redis://localhost:6379/messages
    ###< symfony/messenger ###

    View Slide

  35. > composer require doctrine

    View Slide

  36. Doctrine transport
    #.env
    ###> symfony/messenger ###
    # Choose one of the transports below
    # MESSENGER_TRANSPORT_DSN=amqp://guest@localhost:5672/%2f/messages
    MESSENGER_TRANSPORT_DSN=doctrine://default
    # MESSENGER_TRANSPORT_DSN=redis://localhost:6379/messages
    ###< symfony/messenger ###
    New in 4.3

    View Slide

  37. Doctrine transport
    # config/packages/messenger.yaml
    framework:
    messenger:
    transports:
    async: '%env(MESSENGER_TRANSPORT_DSN)%'

    View Slide

  38. @weaverryan
    Transport Ready!
    But we're not using it yet
    (StoopwafelOrder is still handled sync)

    View Slide

  39. Routing to a transport
    # config/packages/messenger.yaml
    framework:
    messenger:
    transports:
    async: '%env(MESSENGER_TRANSPORT_DSN)%'
    routing:
    'App\Message\StroopwafelOrder': async

    View Slide

  40. POST /stroopwafel/order
    {
    "topping": "plain",
    "size": "medium",
    "recipient": "Nicolas"
    }
    … no Stroopwafel is baked!

    View Slide

  41. SELECT * FROM messenger_messages;
    * Table is created automatically

    View Slide

  42. How do we tell Messenger that we
    *are* ready to handle the message?
    (worker)
    @weaverryan
    Handling/Consuming

    View Slide

  43. @weaverryan
    > bin/console messenger:consume -vv

    View Slide

  44. @weaverryan
    > bin/console messenger:consume -vv

    View Slide

  45. @weaverryan
    Messenger, Done!

    View Slide

  46. THANK YOU!

    View Slide

  47. Working with Workers
    @weaverryan
    [4]

    View Slide

  48. @weaverryan
    > bin/console messenger:consume -vv
    Do we run this manually on production?

    View Slide

  49. @weaverryan
    Nope!
    Your worker WILL die at some point.
    You WANT it to die… on your terms

    View Slide

  50. @weaverryan
    > bin/console messenger:consume --help

    View Slide

  51. @weaverryan
    Running the "worker"
    • Supervisord (see Symfony docs)
    • Automatically via your PaaS (e.g. SymfonyCloud,
    Heroku)
    • Run as many workers as you need

    View Slide

  52. @weaverryan
    Nope!
    When you deploy code changes, will
    your running workers see the new code?

    View Slide

  53. @weaverryan
    > bin/console messenger:stop-workers
    (graceful exit)
    New in 4.3

    View Slide

  54. @weaverryan
    > symfony run -d \
    symfony console messenger:consume \
    --watch=config,src,templates,vendor
    What about Locally?

    View Slide

  55. Envelopes & Stamps
    @weaverryan
    [5]

    View Slide

  56. Stamps and Envelopes
    @weaverryan
    Stamps

    View Slide

  57. Adding an Envelope
    public function orderStroop(MessageBusInterface $messageBus)
    {
    // ...
    $envelope = new Envelope($order);
    $messageBus->dispatch($envelope);
    // ...
    }

    View Slide

  58. Adding Some Stamps
    public function orderStroop(MessageBusInterface $messageBus)
    {
    // ...
    $envelope = new Envelope($order);
    $messageBus->dispatch($envelope, [
    new DelayStamp(5000)
    ]);
    // ...
    }

    View Slide

  59. Middleware Add Stamps
    public function orderStroop(MessageBusInterface $messageBus)
    {
    // ...
    $envelope = new Envelope($order);
    $envelope = $messageBus->dispatch($envelope, [
    new DelayStamp(5000)
    ]);
    dump($envelope);
    // ...
    }

    View Slide

  60. View Slide

  61. Priority Transports
    @weaverryan
    [5]
    New in 4.3

    View Slide

  62. @weaverryan
    A queue of messages
    •StroopwafelOrder
    •CoffeeOrder
    •StroopwafelOrder
    •CleanBathroom
    •CoffeeOrder
    •WashDishes
    }In what order should
    these be handled?

    View Slide

  63. Add a 2nd Transport
    framework:
    messenger:
    transports:
    async: '%env(MESSENGER_TRANSPORT_DSN)%'
    async_high_priority:
    dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
    options:
    queue_name: high_priority

    View Slide

  64. Route by priority
    framework:
    messenger:
    transports:
    async: # ...
    async_high_priority: #...
    routing:
    'App\Message\StroopwafelOrder': async_high_priority
    'App\Message\CoffeeOrder': async_high_priority
    'App\Message\CleanBathroom': async
    'App\Message\WashDishes': async

    View Slide

  65. View Slide

  66. Failures & Retries
    @weaverryan
    [6]
    New in 4.3

    View Slide

  67. @weaverryan
    What should happen if an error
    occurs while handling?

    View Slide

  68. // ...
    class CoffeeOrderHandler implements MessageHandlerInterface
    {
    // ...
    public function __invoke(CoffeeOrder $order)
    {
    if (rand(0, 10) > 2) {
    throw new OutOfBeansException('Ahhh!');
    }
    $this->logger->info('I made some coffee for you');
    }
    }

    View Slide

  69. View Slide

  70. @weaverryan
    And… if it keeps failing?

    View Slide

  71. async_high_priority:
    dsn: '...'
    retry_strategy:
    # RetryStrategyInterface
    # service: null
    max_retries: 3
    delay: 1000
    multiplier: 2
    max_delay: 0

    View Slide

  72. Failure Transport
    @weaverryan
    [7]
    New in 4.3

    View Slide

  73. framework:
    messenger:
    failure_transport: failed
    transports:
    async: # ...
    async_high_priority: #…
    failed: 'doctrine://default?queue_name=failed'

    View Slide

  74. View Slide

  75. View Slide

  76. View Slide

  77. View Slide

  78. View Slide

  79. Fake Transports
    @weaverryan
    [8]
    New in 4.3

    View Slide

  80. Could I handle some/all messages
    sync while developing to make life easier

    View Slide

  81. # .env
    MESSENGER_TRANSPORT_DSN=sync://

    View Slide

  82. I don't want my messages to be
    sent to a queue while testing

    View Slide

  83. class StroopwafelControllerTest extends WebTestCase
    {
    public function testOrderStroopwafel()
    {
    $client = static::createClient();
    $client->request(
    'POST',
    '/stroopwafel/order',
    json_encode(['topping' => 'plain'])
    );
    $this->assertResponseIsSuccessful();
    }
    }

    View Slide

  84. # .env.test
    MESSENGER_TRANSPORT_DSN=in-memory://

    View Slide

  85. public function testOrderStroopwafel()
    {
    // ...
    $this->assertResponseIsSuccessful();
    /** @var InMemoryTransport $transport */
    $transport = self::$container
    ->get('messenger.transport.async');
    }
    $this->assertCount(1, $transport->getSent());
    $this->assertInstanceOf(
    StroopwafelOrder::class,
    $transport->getSent()[0]->getMessage()
    );

    View Slide

  86. Mailer Support
    @weaverryan
    [9]
    New in 4.3

    View Slide

  87. Could we send our emails async
    through Messenger?

    View Slide

  88. framework:
    messenger:
    # ...
    routing:
    # ...
    'Symfony\Component\Mailer\Messenger': async

    View Slide

  89. What Else?
    @weaverryan
    [10]

    View Slide

  90. @weaverryan
    A queue of new features
    •Redis Transport (4.3) (delay support 4.4)
    •PhpSerializer (4.3)
    •Worker Events (4.3)
    •Auto Clear EntityManager (4.4)
    •from_transport (4.3)
    •API Platform integration
    • … and …

    View Slide

  91. It's STABLE

    View Slide

  92. Putting it all Together
    @weaverryan
    [fin]

    View Slide

  93. @weaverryan
    Messenger…
    Allows you to use a message bus

    View Slide

  94. @weaverryan
    Messenger…
    Allows you to save work for later

    View Slide

  95. @weaverryan
    Messenger…
    Allows you to retry
    and introspect errors

    View Slide

  96. @weaverryan
    Messenger…
    Is Ready for Production!

    View Slide

  97. So use it!
    @weaverryan

    View Slide

  98. Ryan Weaver
    @weaverryan
    THANK YOU!
    Messenger Tutorial
    https://symfonycasts.com/screencast/messenger

    View Slide