Using Redis Streams to Build More Resilient Services

Redis, the swiss army knife of databases, has recieved a new data type with version 5.0 which is worth taking a closer look. Specifically, Redis Streams. In this session Thomas explains how he used Redis Streams to build a more fault-tolerant and scalable service with Symfony and the Symfony Messenger Component.

Transcript

1. PRESENTED BY
   

   View Slide

2. PRESENTED BY
   • Co-founder & CEO of Sulu GmbH
   • More than 15 years of experience
   in web technologies & development
   • PHP, Symfony, React, SQL, Redis,
   Elasticsearch, …
   • Open source enthusiast
   • Loves cooking and mountains
   Hi, I’m Thomas Schedler
   chirimoya thomas@sulu.io
   

   View Slide

3. PRESENTED BY
   1 The story how I came across Redis Streams
   2 Introduction to Redis Stream commands
   3 Real-world application using Redis Streams
   Agenda:
   

   View Slide

4. PRESENTED BY
   https://github.com/matthiasnoback/building-autonomous-services-workshop
   

   View Slide

5. PRESENTED BY
   Tell, don’t ask.
   Generate an optimised read model for your service by 

   consuming domain events from a central stream.
   

   View Slide

6. PRESENTED BY
   tail -f -n 1
   Probably not the most robust implementation

   for a real-world application.
   

   View Slide

7. PRESENTED BY
   

   View Slide

8. PRESENTED BY
   In computer science, a stream is a
   sequence of data elements made
   available over time.
   “
   

   View Slide

9. PRESENTED BY
   Stream
   00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 18
   17
   consumer 1
   position
   consumer 2
   position
   new
   record
   truncatable records
   producer emitting records
   

   View Slide

10. PRESENTED BY
    https://brandur.org/redis-streams
    

    View Slide

11. PRESENTED BY
    Redis Streams
    A new data type introduced with Redis 5.0.0.
    

    View Slide

12. PRESENTED BY
    Assuming you have installed Redis server and client.

    And keep in mind, we need to use the latest version of Redis 5.0.
    Connect to Redis
    PHP
    $redis = new \Redis();
    $redis->connect('127.0.0.1', '6379');
    CLI
    > redis-cli -h 127.0.0.1 -p 6379
    

    View Slide

13. PRESENTED BY
    The stream elements can be one or more key-value pairs.

    Let's add elements to the stream:
    Add message to stream
    PHP
    $redis->xAdd('mystream', '*', [

    'KEY' => 'VALUE'

    ]);
    // MESSAGE ID (e.g. 1526919030474-55)
    $redis->xLen('mystream');
    // (integer) 1
    CLI
    > XADD mystream * KEY VALUE
    # MESSAGE ID (e.g. 1526919030474-55)
    > XLEN mystream
    # (integer) 1
    

    View Slide

14. PRESENTED BY
    Read message from one or multiple streams, only returning entries with an ID
    greater than the last received ID reported by the caller.
    Read message from stream
    PHP
    $redis->xRead(['mystream' => 0], 2);
    // MESSAGES
    // [

    // 'STREAMNAME' => [

    // 'MESSAGEID' => [

    // 'KEY' => 'VALUE'

    // ],

    // 'MESSAGEID' => [

    // 'KEY' => 'VALUE'
    CLI
    > XREAD COUNT 2 STREAMS mystream 0
    # MESSAGES
    # STREAMNAME [

    # MESSAGEID [

    # KEY

    # VALUE

    # ]
    

    View Slide

15. PRESENTED BY
    In order to avoid polling at a fixed or adaptive interval, the command is able
    to block until new entries get pushed into the requested stream.
    Block for data
    PHP
    $redis->xRead(['mystream' => '$'], 1, 0);
    // MESSAGES
    // [

    // 'STREAMNAME' => [

    // 'MESSAGEID' => [

    // 'KEY' => 'VALUE'

    // ]

    // ]

    // ]
    CLI
    > XREAD BLOCK 0 STREAMS mystream $
    # MESSAGES
    # STREAMNAME [

    # MESSAGEID [

    # KEY

    # VALUE

    # ]
    

    View Slide

16. PRESENTED BY
    All clients are served with all the
    entries arriving in a stream.
    How can we scale reads?
    

    View Slide

17. PRESENTED BY
    Consumer Groups
    00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 18
    17
    group a
    consumer 1
    position
    group b
    consumer 2
    position
    new
    record
    truncatable records
    group a
    consumer 2
    position
    group b
    consumer 1
    position
    

    View Slide

18. PRESENTED BY
    Redis stream provides the concept of consumer groups, allowing multiple
    consumers to process the same stream to implement load balancing.
    Create a Consumer Group
    PHP
    $redis->xGroup(

    'CREATE', 

    'mystream',

    'mygroup'

    );
    // 1/0
    CLI
    > XGROUP CREATE mystream mygroup $
    # 1/0
    

    View Slide

19. PRESENTED BY
    Redis stream provides the concept of consumer groups, allowing multiple
    consumers to process the same stream to implement load balancing.
    Read data via Consumer Group
    PHP
    $redis->xReadGroup(

    'mygroup', 'myconsumer',

    ['mystream' => 0]

    );
    // MESSAGES
    $redis->xReadGroup(

    'mygroup', 'myconsumer',

    ['mystream' => '>']

    );
    CLI
    > XREADGROUP

    GROUP mygroup mycomsumer

    STREAMS mystream 0
    # MESSAGES
    > XREADGROUP

    GROUP mygroup mycomsumer

    STREAMS mystream >
    

    View Slide

20. PRESENTED BY
    Consumer groups require explicit acknowledgement of the messages
    successfully processed by the consumer.
    Data processed confirmation
    PHP
    $redis->xAck(

    'mystream', 

    'mygroup', 

    ['message-id']

    );
    // 1/0
    CLI
    > XACK mystream mygroup message-id
    # 1/0
    

    View Slide

21. PRESENTED BY
    • Identify the message data that has been delivered but not confirmed
    • Change the owner of these data and redeliver
    Failure Processing
    PHP
    $redis->xPending('mystream', 'mygroup');
    // MESSAGE IDs
    $redis->xClaim(

    'mystream', 

    'mygroup',

    'myconsumer',

    3600,

    ['message-id']

    );
    CLI
    > XPENDING mystream mygroup
    # MESSAGE IDs
    > XCLAIM mystream mygroup myconsumer
    3600 message-id
    

    View Slide

22. PRESENTED BY
    Worked pretty good out of the box,
    but the API is quite low-level.
    Let's search for a high-level abstraction.
    

    View Slide

23. PRESENTED BY
    

    View Slide

24. PRESENTED BY
    • The Messenger component helps
    applications send and receive
    messages — message bus
    • By default, messages are handled
    as soon as they are dispatched
    • If you want to handle a message
    asynchronously, you can configure
    a transport
    Symfony Messenger Component
    

    View Slide

25. PRESENTED BY
    Create our own transport?
    No official Redis transport — Symfony Messenger 4.2
    

    View Slide

26. declare(strict_types=1);
    namespace HandcraftedInTheAlps\Bundle\RedisTransportBundle\Transport;
    use Symfony\Component\Messenger\Transport\TransportFactoryInterface;
    use Symfony\Component\Messenger\Transport\TransportInterface;
    class RedisStreamTransportFactory implements TransportFactoryInterface
    {
    public function createTransport(string $dsn, array $options): TransportInterface
    {
    $parsedUrl = parse_url($dsn);
    ...
    return new RedisStreamTransport($parsedUrl['host'], $parsedUrl['port'], ...);
    }
    public function supports(string $dsn, array $options): bool
    {
    return 0 === mb_strpos($dsn, 'redis-stream://');
    }
    }
    

    View Slide

27. PRESENTED BY
    

    View Slide

28. PRESENTED BY
    

    View Slide

29. PRESENTED BY
    

    View Slide

30. PRESENTED BY
    

    View Slide

31. PRESENTED BY
    

    View Slide

32. PRESENTED BY
    The Redis Streams transport was
    introduced in Symfony 4.3
    Open source … FTW!
    

    View Slide

33. PRESENTED BY
    Real-world showcase — Ticketing Application
    Ticketing Application
    Ticket Access
    Order
    Event
    ...
    Access System
    REST API
    

    View Slide

34. PRESENTED BY
    Real-world showcase — Ticketing Application
    Ticketing Application
    Ticket Access
    Order
    Event
    ...
    Access System
    REST API
    Message Bus
    Contingent
    

    View Slide

35. PRESENTED BY
    Tell, don’t ask.
    Domain Events are used to capture things that can trigger 

    a change to the state of your application.
    

    View Slide

36. class DomainEventMessage
    {
    ...
    public function __construct(
    string $name,
    string $type,
    string $id,
    array $payload = [],
    ?\DateTimeImmutable $created = null
    ) {
    $this->name = $name;
    $this->type = $type;
    $this->id = $id;
    $this->payload = $payload;
    $this->created = $created ?? new \DateTimeImmutable();
    }
    ...
    }
    

    View Slide

37. public function cancelTicket(Ticket $ticket): void
    {
    $this->workflowRegistry->get($ticket)->apply($ticket, Ticket::TRANSITION_CANCEL);
    $newTicketId = Uuid::uuid4()->toString();
    $this->createTicket($newTicketId, $ticket);
    $this->domainEventCollector->push(
    new DomainEventMessage(
    Ticket::class,
    'canceled',
    $ticket->getId(),
    [
    'newTicketId' => $newTicketId,
    ]
    )
    );
    }
    

    View Slide

38. class TicketCanceledDomainEventMessageHandler extends AbstractTicketDomainEventMessageHandler
    {
    public function __invoke(DomainEventMessage $message): void
    {
    if ('canceled' !== $message->getType() || Ticket::class !== $message->getName()) {
    return;
    }
    $ticketIds = [$message->getId()];
    $newTicketId = $message->getPayload()['newTicketId'] ?? '';
    if ($newTicketId) {
    $ticketIds[] = $newTicketId;
    } else {
    $this->logger->error('Could not find newTicketId in canceled for access system.', [
    'canceledTicketId' => $message->getId(),
    ]);
    }
    $this->syncTickets($ticketIds);
    }
    }
    

    View Slide

39. PRESENTED BY
    When ~ argument is used, the trimming is performed only when Redis is able
    to remove a whole macro node.
    Trim Data from the stream
    PHP
    $redis->xTrim('mystream', 1000, true);
    CLI
    > XTRIM mystream MAXLEN ~ 1000
    

    View Slide

40. class DomainEventHandler implements MessageHandlerInterface
    {
    /**
    * @var ElasticsearchStorage
    */
    private $storage;
    public function __construct(ElasticsearchStorage $storage)
    {
    $this->storage = $storage;
    }
    public function __invoke(DomainEventMessage $message): void
    {
    $this->storage->store($message);
    }
    }
    

    View Slide

41. Thank you!
    Thank you!
    

    View Slide