Using redis streams to build more resilient services

@chirimoya [email protected] https://github.com/chirimoya Hey, I’m Thomas – Co-founder & CEO of Sulu GmbH – More than 15 years of experience in web technologies & development – PHP, Symfony, React, SQL, Elasticsearch, … – Open source enthusiast – Loves cooking and mountains

No content

Tell, don’t ask. Generate an optimised read model for your service by consuming domain events from a central stream.

tail -f -n 1 Probably not the most robust implementation for a real-life project.

No content

“ In computer science, a stream is a sequence of data elements made available over time.

Stream

https://brandur.org/redis-streams

Redis Streams A new data type introduced with Redis 5.0.0.

Connect to Redis Assuming you have installed Redis server and client. And keep in mind, we need to use the latest version of Redis 5.0. PHP $redis = new \Redis(); $redis->connect('127.0.0.1', '6379'); CLI > redis-cli -h 127.0.0.1 -p 6379

Add message to stream The stream elements can be one or more key-value pairs. Let's add elements to the stream: PHP $redis->xAdd('mystream', '*', [
 'KEY' => 'VALUE'
 ]); // MESSAGE ID (e.g. 1526919030474-55) $redis->xLen('mystream'); // (integer) 1 CLI > XADD mystream * KEY VALUE # MESSAGE ID (e.g. 1526919030474-55) > XLEN mystream # (integer) 1

Read message from stream Read message from one or multiple streams, only returning entries with an ID greater than the last received ID reported by the caller. PHP $redis->xRead(['mystream' => 0], 2); // MESSAGES // [
 // 'STREAMNAME' => [
 // 'MESSAGEID' => [
 // 'KEY' => 'VALUE'
 // ],
 // 'MESSAGEID‘ => [
 // 'KEY' => 'VALUE'
 CLI > XREAD COUNT 2 STREAMS mystream 0 # MESSAGES # STREAMNAME [
 # MESSAGEID [
 # KEY
 # VALUE
 # ]
 # MESSAGEID [
 # KEY

Block for data In order to avoid polling at a fixed or adaptive interval, the command is able to block until new entries get pushed into the requested stream. PHP $redis->xRead(['mystream' => '$'], 1, 0); // MESSAGES // [
 // 'STREAMNAME' => [
 // 'MESSAGEID' => [
 // 'KEY' => 'VALUE'
 // ]
 // ],
 // ] CLI > XREAD BLOCK 0 STREAMS mystream $ # MESSAGES # STREAMNAME [
 # MESSAGEID [
 # KEY
 # VALUE
 # ]
 # ]

All clients are served with all the entries arriving in a stream. How can I scale my reads?

Consumer Groups

Create a Consumer Group Redis stream provides the concept of consumer groups, allowing multiple consumers to process the same stream to implement load balancing. PHP $redis->xGroup(
 'CREATE', 
 'mystream',
 'mygroup'
 ); // 1/0 CLI > XGROUP CREATE mystream mygroup $ # 1/0

Read data via Consumer Group Redis stream provides the concept of consumer groups, allowing multiple consumers to process the same stream to implement load balancing. PHP $redis->xReadGroup(
 'mygroup', 'myconsumer',
 ['mystream' => 0]
 ); // MESSAGES $redis->xReadGroup(
 'mygroup', 'myconsumer',
 ['mystream' => '>']
 ); CLI > XREADGROUP
 GROUP mygroup mycomsumer
 STREAMS mystream 0 # MESSAGES > XREADGROUP
 GROUP mygroup mycomsumer
 STREAMS mystream >

Data processed confirmation Consumer groups require explicit acknowledgement of the messages successfully processed by the consumer. PHP $redis->xAck(
 'mystream', 
 'mygroup', 
 ['message-id']
 ); // 1/0 CLI > XACK mystream mygroup message-id # 1/0

Failure Processing – Identify the message data that has been delivered but not confirmed – Change the owner of these data and redeliver PHP $redis->xPending('mystream', 'mygroup'); // MESSAGE IDs $redis->xClaim(
 'mystream', 
 'mygroup',
 'myconsumer',
 3600,
 ['message-id']
 ); CLI > XPENDING mystream mygroup # MESSAGE IDs > XCLAIM mystream mygroup myconsumer 3600 message-id

Worked pretty good out of the box, but the API is quite low-level. Let's search for a high-level abstraction.

Symfony Messenger Version 4.2

Create your own transport?

No content

No content

No content

No content

No content

The Redis transport was introduced in Symfony 4.3 Open source … FTW!

No content

# config/packages/messenger.yaml framework: messenger: buses: query_bus: ~ event_bus: ~ message_bus: middleware: - Avodi\Bundle\EventBundle\Common\DomainEvent\DomainEventMiddleware - App\Common\Messenger\SymfonyEventMiddleware - doctrine_transaction routing: 'Avodi\Bundle\EventBundle\Common\DomainEvent\DomainEventMessage': senders: ['domain_events'] transports: domain_events: 'redis://passwor@localhost:6379/mystream/mygroup/myconsumer?auto_setup=true'

domainEvents[] = $event; } /** * @return DomainEventMessage[] */ public function release(): array { $domainEvents = $this->domainEvents; $this->domainEvents = []; return $domainEvents; } }

public function cancelTicket(Ticket $ticket): void { $this->workflowRegistry->get($ticket)->apply($ticket, Ticket::TRANSITION_CANCEL); $newTicketId = Uuid::uuid4()->toString(); $this->createTicket($newTicketId, $ticket); $this->domainEventCollector->push( new DomainEventMessage( Ticket::class, 'canceled', $ticket->getId(), [ 'newTicketId' => $newTicketId, ] ) ); }

getType() || Ticket::class !== $message->getName()) { return; } $ticketIds = [$message->getId()]; $newTicketId = $message->getPayload()['newTicketId'] ?? ''; if ($newTicketId) { $ticketIds[] = $newTicketId; } else { $this->logger->error('Could not find newTicketId in canceled for access system.', [ 'canceledTicketId' => $message->getId(), ]); } $this->syncTickets($ticketIds); } }

# supervicord.conf [program:consumer] directory=/var/www/html command=bin/console messenger:consume --bus event_bus domain_events --time-limit=3600 autostart=true autorestart=true

Trim Data from the stream When ~ argument is used, the trimming is performed only when Redis is able to remove a whole macro node. PHP $redis->xTrim('mystream', 1000, true); CLI > XTRIM mystream MAXLEN ~ 1000

storage = $storage; } public function __invoke(DomainEventMessage $message): void { $this->storage->store($message); } }

No content