Exploring the Queue

Learn about queues by building up abstractions in PHP

Transcript

1. Exploring the Queue
2. The Q?
3. Queue (Data Structure) • Collection in a definite order ◦ Last In First Out (LIFO) ◦ First In First Out (FIFO) ◦ Some persistent backends don’t guarantee order! • Two operations ◦ Enqueue ◦ Dequeue /** FIFO */ $fifoQueue->enqueue('Message 1'); $fifoQueue->enqueue('Message 2'); $fifoQueue->enqueue('Message 3'); var_dump($fifoQueue->dequeue()); // Message 1 var_dump($fifoQueue->dequeue()); // Message 2 var_dump($fifoQueue->dequeue()); // Message 3 /** LIFO */ $lifoQueue->enqueue('Message 1'); $lifoQueue->enqueue('Message 2'); $lifoQueue->enqueue('Message 3'); var_dump($lifoQueue->dequeue()); // Message 3 var_dump($lifoQueue->dequeue()); // Message 2 var_dump($lifoQueue->dequeue()); // Message 1
4. Without a Queue • You tell an object to do something and it does (hopefully) // email is sent right now! $mailer->send(new Email( 'chris@pmg.com', 'Hello', 'World' ));
5. With a Queue • Pass a Message to the queue • Pick the Message up later and process it $queue->enqueue('chris@pmg.com'); // pretend there’s a ton of stuff here while (!$queue->isEmpty()) { $email = $queue->dequeue(); $mailer->send(new Email( $email, 'Hello', 'World' )); }
6. Two Roles in the Queue • Producer • Consumer /** Producer */ $queue->enqueue('chris@pmg.com'); /** Consumer */ while (!$queue->isEmpty()) { $email = $queue->dequeue(); $mailer->send(new Email( $email, 'Hello', 'World' )); }
7. Producer • Handles the enqueue side of things • Like all storage abstractions, avoid indicating the underlying storage backend interface Producer { public function send($message); } class InMemoryProducer implements Producer { private $queue; public function __construct(SplQueue $queue) { $this->queue = $queue; } public function send($message) { $this->queue->enqueue($message); } }
8. Consumer • Where the work happens • Often has methods that... ◦ Run continuously ◦ Consume a single message • Maybe a method to stop as well • Again, avoid queue terminology • Clicky for Implementation interface Consumer { /** * Consume a single message */ public function once(); /** * Run continuously */ public function run(); /** * Tell the consumer to stop itself */ public function stop(); }
9. Message? Don’t You Mean Job?
10. Nope. Message. • Better reflects real code (queues are not special) • We pass messages to objects all the time (methods)
11. If everything is a message, what goes in the queue? • Command Query Responsibility Segregation (CQRS) • Commands change something about the state of the system (or world) • Queries return information about the state of the system (or world) • Any command is a good candidate to queue
12. Our consumer only does one thing!
13. We Could Send Arrays? • Don’t ever do this • Not pictured: isset everywhere /** @var Producer $producer */ $producer->send([ 'type' => 'password', 'email' => 'chris@pmg.com', ]); $producer->send([ 'type' => 'hello', 'email' => 'chris@pmg.com', ]); // in Consumer::once $message = $this->queue->dequeue(); $addy = $message['email']; switch ($message['type']) { case 'password': $email = new Email($addy, 'Password'); break; case 'hello': $email = new Email($addy, 'Hello'); break; default: throw new \InvalidArgumentException(); } $this->mailer->send($email);
14. Pass Message Objects • Much cleaner • No need to isset check everything /** @var Producer $producer */ $producer->send(new ResetPassword( 'chris@pmg.com' )); $producer->send(new SayHello( 'chris@pmg.com' )); // in Consumer::once $msg = $this->queue->dequeue(); switch (get_class($msg)) { case ResetPassword::class: $email = new Email( $msg->getEmailAddy(), 'Password'); break; case SayHello::class: $email = new Email( $msg->getEmailAddy(), 'Hello'); break; default: throw new \InvalidArgumentException(); } $this->mailer->send($email);
15. $msg = $this->queue->dequeue(); switch (get_class($msg)) { // ... }
16. ಠ_ಠ
17. MessageHandler • Pull out the message handling code • Hide it behind an interface interface MessageHandler { public function handle($message); } final class SwitchMessageHandler implements MessageHandler { public function handle($msg) { switch (get_class($msg)) { case ResetPassword::class: $email = new Email( $msg->getEmailAddy(), 'Password'); break; case SayHello::class: $email = new Email( $msg->getEmailAddy(), 'Hello'); break; default: throw new InvalidArgumentException(); } $this->mailer->send($email); } }
18. How could MessageHandler be better?
19. final class SayHelloHandler implements MessageHandler { public function handle($message) { $this->mailer->send(new Email( $message->getEmailAddy(), 'Hello')); } } final class ResetPasswordHandler implements MessageHandler { public function handle($message) { $this->mailer->send(new Email( $message->getEmailAddy(), 'Password')); } } Delegate to Child Message Handlers final class DelegatingHandler implements MessageHandler { private $handlers; public function __construct(array $handlers) { $this->handlers = $handlers; } public function handle($message) { $cls = get_class($message); if (!isset($this->handlers[$cls])) { throw new InvalidArgumentException(); } $this->handlers[$cls]->handle($message); } }
20. final class SayHelloHandler { public function __invoke(SayHello $message) { $this->mailer->send(new Email( $message->getEmailAddy(), 'Hello')); } } final class ResetPasswordHandler { public function __invoke(ResetPassword $message) { $this->mailer->send(new Email( $message->getEmailAddy(), 'Password')); } } Delegate to Callables final class DelegatingHandler implements MessageHandler { private $handlers; public function __construct(array $handlers) { $this->handlers = $handlers; } public function handle($message) { $cls = get_class($message); if (!isset($this->handlers[$cls])) { throw new InvalidArgumentException(); } call_user_func($this->handlers[$cls], $message); } }
21. It’s all in memory! That’s not useful!
22. Typical Queue Architecture Webserver Cron Scripts Producers Persistent Storage Backend Consumer Consumer Consumer Consumer Consumer Messages
23. SplQueue is Not Going to Cut It
24. Driver • Rather than talking to SplQueue directly, let’s hide the details behind a Driver interface • Could also call this interface Queue interface Driver { public function enqueue($message); public function dequeue($message); } final class InMemoryDriver implements Driver { private $queue; public function __construct(SplQueue $queue=null) { $this->queue = $queue ?: new SplQueue(); } public function enqueue($message) { $this->queue->enqueue($message); } public function dequeue() { return $this->queue->dequeue(); } }
25. class DriverProducer implements Producer { private $driver; public function __construct(Driver $driver) { $this->driver = $driver; } public function send($message) { $this->driver->enqueue($message); } } New Consumer & Producer class DriverConsumer implements Consumer { private $driver; private $handler; // ... public function __construct( SplQueue $queue, MessageHandler $handler ) { $this->queue = $queue; $this->handler = $handler; } public function once() { $message = $this->driver->dequeue(); $this->handler->handle($message); } }
26. PredisDriver • A driver backed by redis • Uses the predis library • Don’t blindly unserialize like this! use Predis\ClientInterface; final class PredisDriver implements Driver { const LISTNAME = 'chrisguitarguy:queue'; private $redis; public function __construct(ClientInterface $redis) { $this->redis = $redis; } public function enqueue($message) { $this->redis->rpush( self::LISTNAME, serialize($message)); } public function dequeue() { $message = $this->redis->blpop( [self::LISTNAME], 10); return null === $message ? null : unserialize($message[1]); } }
27. Pretty much every queue framework looks like this
28. We Skipped a Lot of Details • Error handling in the consumer • Logging • Safe serialization/deserialization of messages • Acknowledgement & failure of message in drivers and consumers • How to set up and run consumers on servers
29. Looking for a queue framework? https://github.com/AgencyPMG/Queue
30. Thanks!