Slide 1
Slide 1 text
Execute, Rinse and Repeat
The shampoo algorithm
Fabien Potencier
Slide 2
Slide 2 text
Out-of-band activities
Like sending an email
When doing something that should not be
in the HTTP request/response hot-path
Slide 3
Slide 3 text
Handle the message out-of-band
$bus->dispatch(new SendEmailMessage($email));
Moving activities out-of-band
$ symfony console messenger:consume async
A message identifies the task
to be done and contains
the task's context
Message handling can happen asynchronously
Slide 4
Slide 4 text
class SomeMessage
{
public function __construct(public string $message)
{
}
}
#[AsMessageHandler]
class SomeHandler
{
public function __invoke(SomeMessage $message)
{
error_log($message->message);
}
}
Handling messages
The handling logic
is wrapped in
another service
A serializable
data object
Slide 5
Slide 5 text
Handling messages
This pattern is also useful
as an abstraction layer
even without async processing
$bus->dispatch(new SignupMessage($user));
The business logic is
abstracted in the handler
Slide 6
Slide 6 text
Async but handled almost immediately
$ symfony console messenger:consume async
For quick activities, messages are handled
almost simultaneously as the "main" flow
Messages are handled as they come in
Slide 7
Slide 7 text
Delaying handling is an option
$msg = new EndOfTrialMessage($userId);
End of trial email shouldn't be sent now...
Envelopes add metadata to the message
$endOfTrial = (new \DateTimeImmutable())->modify('+30 days');
$msg = Envelope::wrap($msg)->with(DelayStamp::delayUntil($endOfTrial));
... but scheduled in 30 days
Slide 8
Slide 8 text
Message as data objects
class EndOfTrialMessage
{
public function __construct(public int $userId)
{
}
}
#[AsMessageHandler]
class EndOfTrialMessageHandler
{
public function __invoke(EndOfTrialMessage $message)
{
// Get the User from DB and send them an email
error_log('Processing '.$message->userId);
}
}
Slide 9
Slide 9 text
Detour: Quickly testing message handling
$transport = new InMemoryTransport();
$bus = new MessageBus([
new SendMessageMiddleware(
new SendersLocator([
EndOfTrialMessage::class => ['mem'],
],
new class(['mem' => fn () => $transport]) implements ContainerInterface {
use ServiceLocatorTrait;
},
,
),
new HandleMessageMiddleware(new HandlersLocator([
EndOfTrialMessage::class => [new EndOfTrialMessageHandler()],
])),
]);
A non-persistent
transport
Message/transport
wiring
Message/handler
wiring
Slide 10
Slide 10 text
Detour of the detour
$transport = new InMemoryTransport();
DelayStamp support as of 6.3
Implemented just for this keynote
https://github.com/symfony/symfony/pull/49725
Slide 11
Slide 11 text
Detour: The simplest possible Container
$transport = new InMemoryTransport();
$bus = new MessageBus([
new SendMessageMiddleware(
new SendersLocator([
EndOfTrialMessage::class => ['mem'],
],
new class(['mem' => fn () => $transport]) implements ContainerInterface {
use ServiceLocatorTrait;
},
,
),
new HandleMessageMiddleware(new HandlersLocator([
EndOfTrialMessage::class => [new EndOfTrialMessageHandler()],
])),
]);
PSR Container
Implements the interface
as an array of callables
Slide 12
Slide 12 text
Detour: Quickly testing message handling
$bus->dispatch($msg);
error_log('Nothing happens before running the workers');
$receivers = ['mem' => $transport];
$worker = new Worker($receivers, $bus);
$worker->run();
Dispatch messages
Process incoming messages forever
Check it actually works
Slide 13
Slide 13 text
Clean-up ended trials?
$cleanupDate = (new \DateTimeImmutable())->modify('+30 days');
$msg = Envelope::wrap($msg)->with(DelayStamp::delayUntil($cleanupDate));
Is it the best way?
1. Should discard converted trials
2. No need to act precisely at +30 days
3. More e
ff i
cient as batch
Slide 14
Slide 14 text
Recurring messages
A message that should be handled
over and over again, on a pre-defined schedule
Slide 15
Slide 15 text
Scheduler
Will land in 6.3
as an experimental component
Slide 16
Slide 16 text
Recurring messages
RecurringMessage::every('1 week', new EndOfTrialMessage())
$schedule = (new Schedule())->add($msg);
A trigger that defines
the frequency
A message to handle
periodically
A schedule describes the config
of one or more recurring messages
Slide 17
Slide 17 text
Running the scheduler
$scheduler = new Scheduler(
[EndOfTrialMessage::class => new EndOfTrialMessageHandler()],
[$schedule],
);
$scheduler->run();
Runs forever and will handle
messages as per the trigger definition
Only useful when using
the component standalone
Slide 18
Slide 18 text
Reusing Messenger infrastructure?
1. Learn it fast as it uses the same Messenger concepts
Messages, handlers, envelopes, stamps, ...
2. The scheduler can be integrated with Messenger via a special
transport
Triggers and message generators
3. Scheduler uses the same Messenger worker: messenger:consume
Time limit, memory management, signal handling, ...
The beauty of the new component ❤
Slide 19
Slide 19 text
Detour: Beautiful contributions
Credits to Sergey Rabochiy ❤
Slide 20
Slide 20 text
The Scheduler component
in the context of a
Symfony full-stack app
Slide 21
Slide 21 text
Register a Schedule provider
#[AsSchedule('default')]
class DefaultScheduleProvider implements ScheduleProviderInterface
{
public function getSchedule(): Schedule
{
return (new Schedule())
->add(RecurringMessage::every('1 week', new EndOfTrialMessage()))
;
}
}
This automatically wires
the Scheduler with Messenger
Like before
1
Slide 22
Slide 22 text
Run the Messenger consumer
#[AsSchedule('default')]
symfony console messenger:consume -v scheduler_default
The name is used for the receiver name
Equivalent to registering scheduler://default
2
Slide 23
Slide 23 text
Enjoy!
3
Slide 24
Slide 24 text
Triggers: "every()"
Frequency as text
RecurringMessage::every('10 seconds', $msg)
RecurringMessage::every('2 weeks', $msg)
RecurringMessage::every('1 day', $msg)
Slide 25
Slide 25 text
Triggers: "every()"
RecurringMessage::every('next tuesday', $msg)
RecurringMessage::every('last day of next month', $msg)
RecurringMessage::every('first monday of next month', $msg)
Relative date format
https://php.net/datetime.formats.relative
Slide 26
Slide 26 text
Triggers: Quickly testing
$msg = RecurringMessage::every('next tuesday', new stdClass());
$i = 0;
$next = new \DateTimeImmutable();
while ($i++ < 20) {
$next = $msg->getTrigger()->getNextRunDate($next);
if (!$next) {
break;
}
echo $next->format('Y-m-d H:i:s')."\n";
}
That's what makes the Scheduler tick!
Slide 27
Slide 27 text
Triggers: "every()"
Time of the day
RecurringMessage::every('1 day', $msg, from: '13:47')
2023-03-20 13:47:00
2023-03-21 13:47:00
2023-03-22 13:47:00
2023-03-23 13:47:00
2023-03-24 13:47:00
2023-03-25 13:47:00
Slide 28
Slide 28 text
Triggers: "every()"
Timezone
RecurringMessage::every('1 day', $msg, from: '13:47+0400')
2023-03-20 13:47:00 +04:00
2023-03-21 13:47:00 +04:00
2023-03-22 13:47:00 +04:00
2023-03-23 13:47:00 +04:00
2023-03-24 13:47:00 +04:00
2023-03-25 13:47:00 +04:00
Slide 29
Slide 29 text
Triggers: "every()"
Can be a DateTimeImmutable
$from = new \DateTimeImmutable('13:47', new \DateTimeZone('Europe/Paris'));
$msg = RecurringMessage::every('1 day', new stdClass(), from: $from);
2023-03-20 13:47:00 Europe/Paris
2023-03-21 13:47:00 Europe/Paris
2023-03-22 13:47:00 Europe/Paris
2023-03-23 13:47:00 Europe/Paris
2023-03-24 13:47:00 Europe/Paris
2023-03-25 13:47:00 Europe/Paris
Slide 30
Slide 30 text
Triggers: "every()"
End date
RecurringMessage::every('1 day', new stdClass(), until: '2023-03-23');
2023-03-20 13:45:00
2023-03-21 13:45:00
2023-03-22 13:45:00
---
Slide 31
Slide 31 text
Triggers: CronExpression
RecurringMessage::cron('* * * * *', $msg)
⚠ Max frequency is 1 minute
Slide 32
Slide 32 text
Triggers: Your choice
RecurringMessage::trigger(TriggerInterface, $msg)
interface TriggerInterface
{
public function getNextRunDate(\DateTimeImmutable $run): ?\DateTimeImmutable;
}
Slide 33
Slide 33 text
Triggers: Be different
😭
Midnight
On top of the hour
At any "round number"
Add some randomness to your times
🤩
Use 12:13 or 11:54
not 12:00
Prefer :17/:23/:47
not :15/:25/:45
Slide 34
Slide 34 text
Each Schedule is a Messenger transport
No stateful "transport"
Messages are handled immediately when generated (like the sync transport)
Messages are never "dispatched"
They are generated by the transport directly
Messages are handled one by one
For all recurring messages configured on a schedule
Scale by using di
ff
erent workers for di
ff
erence schedules
Or use more than one worker for a schedule
Slide 35
Slide 35 text
Multiple schedules
No guarantee that a message is executed right on time or at the
required frequency
It's up to you to configure schedules properly
Create more than one schedule to con
fi
gure them di
ff
erently
You can then scale each worker and schedule independently
Slide 36
Slide 36 text
Messages are handled synchronously one by one
EndOfTrialCleanupMessage
2023-03-24 13:10
EndOfTrialCleanupMessage
EndOfTrialCleanupMessage
2023-03-24 13:00
handled
Next message will be handled
just after the current one
A message that takes
a lot of time
to handle
10 minutes 10 minutes
Recurring message
EndOfTrialCleanupMessage
2023-03-24 13:20
EndOfTrialCleanupMessage
handled
The backlog keeps growing
until the worker is restarted
Slide 37
Slide 37 text
Rule of thumb
Con
fi
gure a frequency that is greater
than the average time
it takes to handle a message
Slide 38
Slide 38 text
messenger:consume downtime?
SignupEmailMessage
for User 1337
queued
Messenger worker down
SignupEmailMessage
for User 1337
handled
SignupEmailMessage
for User 42
queued
SignupEmailMessage
for User 42
handled
"Regular" message Another message
Slide 39
Slide 39 text
messenger:consume downtime?
Messenger worker down
EndOfTrialCleanupMessage
generated
EndOfTrialCleanupMessage
handled
not generated
EndOfTrialCleanupMessage
EndOfTrialCleanupMessage
EndOfTrialCleanupMessage
handled
generated
10 minutes 10 minutes
Recurring message
2023-03-24 13:00 2023-03-25 13:10 2023-03-26 13:20
Slide 40
Slide 40 text
Stateful schedules
#[AsSchedule('default')]
class DefaultScheduleProvider implements ScheduleProviderInterface
{
public function __construct(
private CacheInterface $cache,
) {
}
public function getSchedule(): Schedule
{
$msg = new EndOfTrialMessage();
return (new Schedule())
->add(RecurringMessage::every('2 seconds', $msg))
->stateful($this->cache)
;
}
}
When missing messages
is not an option,
Stateful schedules will catch up.
Slide 41
Slide 41 text
messenger:consume downtime?
Messenger worker down
EndOfTrialCleanupMessage
generated
EndOfTrialCleanupMessage
handled
EndOfTrialCleanupMessage
EndOfTrialCleanupMessage
handled
generated
Recurring message
generated
EndOfTrialCleanupMessage
handled
Stateful schedules
catch up
10 minutes 10 minutes
2023-03-24 13:00 2023-03-25 13:10
EndOfTrialCleanupMessage
2023-03-26 13: 20
Slide 42
Slide 42 text
Many messenger:consume in //?
EndOfTrialCleanupMessage
EndOfTrialCleanupMessage
EndOfTrialCleanupMessage
2023-03-24 13:10
Scaling by running
2 workers running at the same
Worker 1
Worker 2
handled
EndOfTrialCleanupMessage
handled
Oops, the message
is generated
and handled
twice!
2023-03-24 13:10
Slide 43
Slide 43 text
Locking schedules
#[AsSchedule('default')]
class DefaultScheduleProvider implements ScheduleProviderInterface
{
public function __construct(
private CacheInterface $cache,
private LockFactory $lockFactory,
) {
}
public function getSchedule(): Schedule
{
$msg = new EndOfTrialMessage();
return (new Schedule())
->add(RecurringMessage::every('2 seconds', $msg))
->stateful($this->cache)
->lock($this->lockFactory->createLock('default-scheduler'))
;
}
}
Don't use a lock without a cache
Slide 44
Slide 44 text
Many messenger:consume in //?
EndOfTrialCleanupMessage
EndOfTrialCleanupMessage
EndOfTrialCleanupMessage
Scaling by running
2 workers running at the same
Worker 1
Worker 2
handled
not generated
The lock prevents
the message to be generated
multiple times
2023-03-24 13:10
2023-03-24 13:10
Slide 45
Slide 45 text
Frequent but slow message handling?
EndOfTrialCleanupMessage
EndOfTrialCleanupMessage
2023-03-24 13:00
handled
A message that takes
a lot of time
to execute
EndOfTrialCleanupMessage
2023-03-24 13:10
not generated
The lock prevents
the next message to be generated
as another is still handled
Worker 1
Worker 2
Slide 46
Slide 46 text
Send to another Messenger transport
$msg = new RedispatchMessage(new UpdateDocMessage(), 'async');
Message will be handled asynchronously
⚠ Lock is only partially supported
The same message is probably not generated twice
But several messages can be handled in //
Slide 47
Slide 47 text
#[AsMessageHandler()]
class UpdateDocMessageHandler
{
private LockInterface $lock;
public function __construct(
LockFactory $lockFactory,
) {
$this->lock = $lockFactory->createLock('update-doc');
}
public function __invoke(UpdateDocMessage $message)
{
if (!$this->lock->acquire(false)) {
echo "Doc is already being generated, skip...\n";
return;
}
try {
// do something
} finally {
$this->lock->release();
}
}
}
Locking schedules
Handling the lock in the handler
works great!
Slide 48
Slide 48 text
Cron vs Scheduler?
1. Cron can only run "executables" think Symfony Console commands
2. Cron support on Windows?
3. Cron is yet another moving part to monitor
4. Cron max frequency limited to one minute
5. Cron needs tooling for reporting (errors, ...)
6. Cron does not support multiple "workers", locks, states, ...
Cron is still probably best
for any recurring command
not defined in Symfony
Slide 49
Slide 49 text
🙋 I need your help
‣ Test before 6.3
fi
nal and provide feedback!
‣ ProcessMessage and ProcessMessageHandler
‣ RecurringRule support for triggers (I have a branch for that)
‣ debug:scheduler command
‣ ...
Next Steps?
The main concepts
and low-level infrastructure
are well-defined
or don't complain :)
Slide 50
Slide 50 text
https://symfony.com/sponsor
Sponsor Symfony
Thank you!
Slide 51
Slide 51 text
Recap
For a schedule
No cache
No lock
Cache
Cache
Lock
On a single
worker
Catch up missed
messages when
downtime
Not needed
On several
workers
Not recommended Not recommended
Avoid messages to
be executed
several times