Using redis streams to build more resilient services

Redis, das Schweizer Taschenmesser unter den Datenbanken, hat in der Version 5.0 einen neuen Datentyp dazu bekommen, der es wert ist, etwas genauer unter die Lupe genommen zu werden. Konkret geht es um Redis Streams und darum wie sie uns helfen, fehlertolerante und skalierbare Services zu bauen.

Transcript

1. Using redis streams to
   build more resilient
   services
   

   View Slide

2. @chirimoya
   thomas@sulu.io
   https://github.com/chirimoya
   Hey, I’m Thomas
   – Co-founder & CEO of Sulu GmbH
   – More than 15 years of experience in
   web technologies & development
   – PHP, Symfony, React, SQL,
   Elasticsearch, …
   – Open source enthusiast
   – Loves cooking and mountains
   

   View Slide

3. View Slide

4. Tell, don’t ask.
   Generate an optimised read model for your service by
   consuming domain events from a central stream.
   

   View Slide

5. tail -f -n 1
   Probably not the most robust implementation
   for a real-life project.
   

   View Slide

6. View Slide

7. “
   In computer science, a stream is a
   sequence of data elements made
   available over time.
   

   View Slide

8. Stream
   

   View Slide

9. https://brandur.org/redis-streams
   

   View Slide

10. Redis Streams
    A new data type introduced with Redis 5.0.0.
    

    View Slide

11. Connect to Redis
    Assuming you have installed Redis server and client.
    And keep in mind, we need to use the latest version of Redis 5.0.
    PHP
    $redis = new \Redis();
    $redis->connect('127.0.0.1', '6379');
    CLI
    > redis-cli -h 127.0.0.1 -p 6379
    

    View Slide

12. Add message to stream
    The stream elements can be one or more key-value pairs.
    Let's add elements to the stream:
    PHP
    $redis->xAdd('mystream', '*', [

    'KEY' => 'VALUE'

    ]);
    // MESSAGE ID (e.g. 1526919030474-55)
    $redis->xLen('mystream');
    // (integer) 1
    CLI
    > XADD mystream * KEY VALUE
    # MESSAGE ID (e.g. 1526919030474-55)
    > XLEN mystream
    # (integer) 1
    

    View Slide

13. Read message from stream
    Read message from one or multiple streams, only returning entries with an ID greater
    than the last received ID reported by the caller.
    PHP
    $redis->xRead(['mystream' => 0], 2);
    // MESSAGES
    // [

    // 'STREAMNAME' => [

    // 'MESSAGEID' => [

    // 'KEY' => 'VALUE'

    // ],

    // 'MESSAGEID‘ => [

    // 'KEY' => 'VALUE'

    CLI
    > XREAD COUNT 2 STREAMS mystream 0
    # MESSAGES
    # STREAMNAME [

    # MESSAGEID [

    # KEY

    # VALUE

    # ]

    # MESSAGEID [

    # KEY
    

    View Slide

14. Block for data
    In order to avoid polling at a fixed or adaptive interval, the command is able to block
    until new entries get pushed into the requested stream.
    PHP
    $redis->xRead(['mystream' => '$'], 1, 0);
    // MESSAGES
    // [

    // 'STREAMNAME' => [

    // 'MESSAGEID' => [

    // 'KEY' => 'VALUE'

    // ]

    // ],

    // ]
    CLI
    > XREAD BLOCK 0 STREAMS mystream $
    # MESSAGES
    # STREAMNAME [

    # MESSAGEID [

    # KEY

    # VALUE

    # ]

    # ]
    

    View Slide

15. All clients are served with all the
    entries arriving in a stream.
    How can I scale my reads?
    

    View Slide

16. Consumer Groups
    

    View Slide

17. Create a Consumer Group
    Redis stream provides the concept of consumer groups, allowing multiple consumers to
    process the same stream to implement load balancing.
    PHP
    $redis->xGroup(

    'CREATE', 

    'mystream',

    'mygroup'

    );
    // 1/0
    CLI
    > XGROUP CREATE mystream mygroup $
    # 1/0
    

    View Slide

18. Read data via Consumer Group
    Redis stream provides the concept of consumer groups, allowing multiple consumers to
    process the same stream to implement load balancing.
    PHP
    $redis->xReadGroup(

    'mygroup', 'myconsumer',

    ['mystream' => 0]

    );
    // MESSAGES
    $redis->xReadGroup(

    'mygroup', 'myconsumer',

    ['mystream' => '>']

    );
    CLI
    > XREADGROUP

    GROUP mygroup mycomsumer

    STREAMS mystream 0
    # MESSAGES
    > XREADGROUP

    GROUP mygroup mycomsumer

    STREAMS mystream >
    

    View Slide

19. Data processed confirmation
    Consumer groups require explicit acknowledgement of the messages successfully processed
    by the consumer.
    PHP
    $redis->xAck(

    'mystream', 

    'mygroup', 

    ['message-id']

    );
    // 1/0
    CLI
    > XACK mystream mygroup message-id
    # 1/0
    

    View Slide

20. Failure Processing
    – Identify the message data that has been delivered but not confirmed
    – Change the owner of these data and redeliver
    PHP
    $redis->xPending('mystream', 'mygroup');
    // MESSAGE IDs
    $redis->xClaim(

    'mystream', 

    'mygroup',

    'myconsumer',

    3600,

    ['message-id']

    );
    CLI
    > XPENDING mystream mygroup
    # MESSAGE IDs
    > XCLAIM mystream mygroup myconsumer 3600
    message-id
    

    View Slide

21. Worked pretty good out of the box,
    but the API is quite low-level.
    Let's search for a high-level abstraction.
    

    View Slide

22. Symfony Messenger
    Version 4.2
    

    View Slide

23. Create your own transport?
    

    View Slide

24. declare(strict_types=1);
    namespace HandcraftedInTheAlps\Bundle\RedisTransportBundle\Transport;
    use Symfony\Component\Messenger\Transport\TransportFactoryInterface;
    use Symfony\Component\Messenger\Transport\TransportInterface;
    class RedisStreamTransportFactory implements TransportFactoryInterface
    {
    public function createTransport(string $dsn, array $options): TransportInterface
    {
    $parsedUrl = parse_url($dsn);
    ...
    return new RedisStreamTransport($parsedUrl['host'], $parsedUrl['port'], ...);
    }
    public function supports(string $dsn, array $options): bool
    {
    return 0 === mb_strpos($dsn, 'redis-stream://');
    }
    }
    

    View Slide

25. View Slide

26. View Slide

27. View Slide

28. View Slide

29. View Slide

30. The Redis transport was
    introduced in Symfony 4.3
    Open source … FTW!
    

    View Slide

31. View Slide

32. # config/packages/messenger.yaml
    framework:
    messenger:
    buses:
    query_bus: ~
    event_bus: ~
    message_bus:
    middleware:
    - Avodi\Bundle\EventBundle\Common\DomainEvent\DomainEventMiddleware
    - App\Common\Messenger\SymfonyEventMiddleware
    - doctrine_transaction
    routing:
    'Avodi\Bundle\EventBundle\Common\DomainEvent\DomainEventMessage':
    senders: ['domain_events']
    transports:
    domain_events: 'redis://passwor@localhost:6379/mystream/mygroup/myconsumer?auto_setup=true'
    

    View Slide

33. class DomainEventCollector
    {
    /**
    * @var DomainEventMessage[]
    */
    private $domainEvents = [];
    public function push(DomainEventMessage $event): void
    {
    $this->domainEvents[] = $event;
    }
    /**
    * @return DomainEventMessage[]
    */
    public function release(): array
    {
    $domainEvents = $this->domainEvents;
    $this->domainEvents = [];
    return $domainEvents;
    }
    }
    

    View Slide

34. public function cancelTicket(Ticket $ticket): void
    {
    $this->workflowRegistry->get($ticket)->apply($ticket, Ticket::TRANSITION_CANCEL);
    $newTicketId = Uuid::uuid4()->toString();
    $this->createTicket($newTicketId, $ticket);
    $this->domainEventCollector->push(
    new DomainEventMessage(
    Ticket::class,
    'canceled',
    $ticket->getId(),
    [
    'newTicketId' => $newTicketId,
    ]
    )
    );
    }
    

    View Slide

35. class TicketCanceledDomainEventMessageHandler extends AbstractTicketDomainEventMessageHandler
    {
    public function __invoke(DomainEventMessage $message): void
    {
    if ('canceled' !== $message->getType() || Ticket::class !== $message->getName()) {
    return;
    }
    $ticketIds = [$message->getId()];
    $newTicketId = $message->getPayload()['newTicketId'] ?? '';
    if ($newTicketId) {
    $ticketIds[] = $newTicketId;
    } else {
    $this->logger->error('Could not find newTicketId in canceled for access system.', [
    'canceledTicketId' => $message->getId(),
    ]);
    }
    $this->syncTickets($ticketIds);
    }
    }
    

    View Slide

36. # supervicord.conf
    [program:consumer]
    directory=/var/www/html
    command=bin/console messenger:consume --bus event_bus domain_events --time-limit=3600
    autostart=true
    autorestart=true
    

    View Slide

37. Trim Data from the stream
    When ~ argument is used, the trimming is performed only when Redis is able to remove a
    whole macro node.
    PHP
    $redis->xTrim('mystream', 1000, true);
    CLI
    > XTRIM mystream MAXLEN ~ 1000
    

    View Slide

38. class DomainEventHandler implements MessageHandlerInterface
    {
    /**
    * @var ElasticsearchStorage
    */
    private $storage;
    public function __construct(ElasticsearchStorage $storage)
    {
    $this->storage = $storage;
    }
    public function __invoke(DomainEventMessage $message): void
    {
    $this->storage->store($message);
    }
    }
    

    View Slide

39. View Slide