Using redis streams to build more resilient services

Redis, das Schweizer Taschenmesser unter den Datenbanken, hat in der Version 5.0 einen neuen Datentyp dazu bekommen, der es wert ist, etwas genauer unter die Lupe genommen zu werden. Konkret geht es um Redis Streams und darum wie sie uns helfen, fehlertolerante und skalierbare Services zu bauen.

Transcript

1. Using redis streams to build more resilient services
2. @chirimoya thomas@sulu.io https://github.com/chirimoya Hey, I’m Thomas – Co-founder & CEO of Sulu GmbH – More than 15 years of experience in web technologies & development – PHP, Symfony, React, SQL, Elasticsearch, … – Open source enthusiast – Loves cooking and mountains
4. Tell, don’t ask. Generate an optimised read model for your service by consuming domain events from a central stream.
5. tail -f -n 1 Probably not the most robust implementation for a real-life project.
7. “ In computer science, a stream is a sequence of data elements made available over time.
8. Stream
9. https://brandur.org/redis-streams
10. Redis Streams A new data type introduced with Redis 5.0.0.
11. Connect to Redis Assuming you have installed Redis server and client. And keep in mind, we need to use the latest version of Redis 5.0. PHP $redis = new \Redis(); $redis->connect('127.0.0.1', '6379'); CLI > redis-cli -h 127.0.0.1 -p 6379
12. Add message to stream The stream elements can be one or more key-value pairs. Let's add elements to the stream: PHP $redis->xAdd('mystream', '*', [
 'KEY' => 'VALUE'
 ]); // MESSAGE ID (e.g. 1526919030474-55) $redis->xLen('mystream'); // (integer) 1 CLI > XADD mystream * KEY VALUE # MESSAGE ID (e.g. 1526919030474-55) > XLEN mystream # (integer) 1
13. Read message from stream Read message from one or multiple streams, only returning entries with an ID greater than the last received ID reported by the caller. PHP $redis->xRead(['mystream' => 0], 2); // MESSAGES // [
 // 'STREAMNAME' => [
 // 'MESSAGEID' => [
 // 'KEY' => 'VALUE'
 // ],
 // 'MESSAGEID‘ => [
 // 'KEY' => 'VALUE'
 CLI > XREAD COUNT 2 STREAMS mystream 0 # MESSAGES # STREAMNAME [
 # MESSAGEID [
 # KEY
 # VALUE
 # ]
 # MESSAGEID [
 # KEY
14. Block for data In order to avoid polling at a fixed or adaptive interval, the command is able to block until new entries get pushed into the requested stream. PHP $redis->xRead(['mystream' => '$'], 1, 0); // MESSAGES // [
 // 'STREAMNAME' => [
 // 'MESSAGEID' => [
 // 'KEY' => 'VALUE'
 // ]
 // ],
 // ] CLI > XREAD BLOCK 0 STREAMS mystream $ # MESSAGES # STREAMNAME [
 # MESSAGEID [
 # KEY
 # VALUE
 # ]
 # ]
15. All clients are served with all the entries arriving in a stream. How can I scale my reads?
16. Consumer Groups
17. Create a Consumer Group Redis stream provides the concept of consumer groups, allowing multiple consumers to process the same stream to implement load balancing. PHP $redis->xGroup(
 'CREATE', 
 'mystream',
 'mygroup'
 ); // 1/0 CLI > XGROUP CREATE mystream mygroup $ # 1/0
18. Read data via Consumer Group Redis stream provides the concept of consumer groups, allowing multiple consumers to process the same stream to implement load balancing. PHP $redis->xReadGroup(
 'mygroup', 'myconsumer',
 ['mystream' => 0]
 ); // MESSAGES $redis->xReadGroup(
 'mygroup', 'myconsumer',
 ['mystream' => '>']
 ); CLI > XREADGROUP
 GROUP mygroup mycomsumer
 STREAMS mystream 0 # MESSAGES > XREADGROUP
 GROUP mygroup mycomsumer
 STREAMS mystream >
19. Data processed confirmation Consumer groups require explicit acknowledgement of the messages successfully processed by the consumer. PHP $redis->xAck(
 'mystream', 
 'mygroup', 
 ['message-id']
 ); // 1/0 CLI > XACK mystream mygroup message-id # 1/0
20. Failure Processing – Identify the message data that has been delivered but not confirmed – Change the owner of these data and redeliver PHP $redis->xPending('mystream', 'mygroup'); // MESSAGE IDs $redis->xClaim(
 'mystream', 
 'mygroup',
 'myconsumer',
 3600,
 ['message-id']
 ); CLI > XPENDING mystream mygroup # MESSAGE IDs > XCLAIM mystream mygroup myconsumer 3600 message-id
21. Worked pretty good out of the box, but the API is quite low-level. Let's search for a high-level abstraction.
22. Symfony Messenger Version 4.2
23. Create your own transport?
24. <?php declare(strict_types=1); namespace HandcraftedInTheAlps\Bundle\RedisTransportBundle\Transport; use Symfony\Component\Messenger\Transport\TransportFactoryInterface; use Symfony\Component\Messenger\Transport\TransportInterface; class RedisStreamTransportFactory implements TransportFactoryInterface { public function createTransport(string $dsn, array $options): TransportInterface { $parsedUrl = parse_url($dsn); ... return new RedisStreamTransport($parsedUrl['host'], $parsedUrl['port'], ...); } public function supports(string $dsn, array $options): bool { return 0 === mb_strpos($dsn, 'redis-stream://'); } }
30. The Redis transport was introduced in Symfony 4.3 Open source … FTW!
32. # config/packages/messenger.yaml framework: messenger: buses: query_bus: ~ event_bus: ~ message_bus: middleware: - Avodi\Bundle\EventBundle\Common\DomainEvent\DomainEventMiddleware - App\Common\Messenger\SymfonyEventMiddleware - doctrine_transaction routing: 'Avodi\Bundle\EventBundle\Common\DomainEvent\DomainEventMessage': senders: ['domain_events'] transports: domain_events: 'redis://passwor@localhost:6379/mystream/mygroup/myconsumer?auto_setup=true'
33. <?php class DomainEventCollector { /** * @var DomainEventMessage[] */ private $domainEvents = []; public function push(DomainEventMessage $event): void { $this->domainEvents[] = $event; } /** * @return DomainEventMessage[] */ public function release(): array { $domainEvents = $this->domainEvents; $this->domainEvents = []; return $domainEvents; } }
34. public function cancelTicket(Ticket $ticket): void { $this->workflowRegistry->get($ticket)->apply($ticket, Ticket::TRANSITION_CANCEL); $newTicketId = Uuid::uuid4()->toString(); $this->createTicket($newTicketId, $ticket); $this->domainEventCollector->push( new DomainEventMessage( Ticket::class, 'canceled', $ticket->getId(), [ 'newTicketId' => $newTicketId, ] ) ); }
35. <?php class TicketCanceledDomainEventMessageHandler extends AbstractTicketDomainEventMessageHandler { public function __invoke(DomainEventMessage $message): void { if ('canceled' !== $message->getType() || Ticket::class !== $message->getName()) { return; } $ticketIds = [$message->getId()]; $newTicketId = $message->getPayload()['newTicketId'] ?? ''; if ($newTicketId) { $ticketIds[] = $newTicketId; } else { $this->logger->error('Could not find newTicketId in canceled for access system.', [ 'canceledTicketId' => $message->getId(), ]); } $this->syncTickets($ticketIds); } }
36. # supervicord.conf [program:consumer] directory=/var/www/html command=bin/console messenger:consume --bus event_bus domain_events --time-limit=3600 autostart=true autorestart=true
37. Trim Data from the stream When ~ argument is used, the trimming is performed only when Redis is able to remove a whole macro node. PHP $redis->xTrim('mystream', 1000, true); CLI > XTRIM mystream MAXLEN ~ 1000
38. <?php class DomainEventHandler implements MessageHandlerInterface { /** * @var ElasticsearchStorage */ private $storage; public function __construct(ElasticsearchStorage $storage) { $this->storage = $storage; } public function __invoke(DomainEventMessage $message): void { $this->storage->store($message); } }