Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using redis streams to build more resilient services

Using redis streams to build more resilient services

Redis, das Schweizer Taschenmesser unter den Datenbanken, hat in der Version 5.0 einen neuen Datentyp dazu bekommen, der es wert ist, etwas genauer unter die Lupe genommen zu werden. Konkret geht es um Redis Streams und darum wie sie uns helfen, fehlertolerante und skalierbare Services zu bauen.

Thomas Schedler

September 27, 2019
Tweet

More Decks by Thomas Schedler

Other Decks in Programming

Transcript

  1. Using redis streams to
    build more resilient
    services

    View full-size slide

  2. @chirimoya
    [email protected]
    https://github.com/chirimoya
    Hey, I’m Thomas
    – Co-founder & CEO of Sulu GmbH
    – More than 15 years of experience in
    web technologies & development
    – PHP, Symfony, React, SQL,
    Elasticsearch, …
    – Open source enthusiast
    – Loves cooking and mountains

    View full-size slide

  3. Tell, don’t ask.
    Generate an optimised read model for your service by
    consuming domain events from a central stream.

    View full-size slide

  4. tail -f -n 1
    Probably not the most robust implementation
    for a real-life project.

    View full-size slide


  5. In computer science, a stream is a
    sequence of data elements made
    available over time.

    View full-size slide

  6. https://brandur.org/redis-streams

    View full-size slide

  7. Redis Streams
    A new data type introduced with Redis 5.0.0.

    View full-size slide

  8. Connect to Redis
    Assuming you have installed Redis server and client.
    And keep in mind, we need to use the latest version of Redis 5.0.
    PHP
    $redis = new \Redis();
    $redis->connect('127.0.0.1', '6379');
    CLI
    > redis-cli -h 127.0.0.1 -p 6379

    View full-size slide

  9. Add message to stream
    The stream elements can be one or more key-value pairs.
    Let's add elements to the stream:
    PHP
    $redis->xAdd('mystream', '*', [

    'KEY' => 'VALUE'

    ]);
    // MESSAGE ID (e.g. 1526919030474-55)
    $redis->xLen('mystream');
    // (integer) 1
    CLI
    > XADD mystream * KEY VALUE
    # MESSAGE ID (e.g. 1526919030474-55)
    > XLEN mystream
    # (integer) 1

    View full-size slide

  10. Read message from stream
    Read message from one or multiple streams, only returning entries with an ID greater
    than the last received ID reported by the caller.
    PHP
    $redis->xRead(['mystream' => 0], 2);
    // MESSAGES
    // [

    // 'STREAMNAME' => [

    // 'MESSAGEID' => [

    // 'KEY' => 'VALUE'

    // ],

    // 'MESSAGEID‘ => [

    // 'KEY' => 'VALUE'

    CLI
    > XREAD COUNT 2 STREAMS mystream 0
    # MESSAGES
    # STREAMNAME [

    # MESSAGEID [

    # KEY

    # VALUE

    # ]

    # MESSAGEID [

    # KEY

    View full-size slide

  11. Block for data
    In order to avoid polling at a fixed or adaptive interval, the command is able to block
    until new entries get pushed into the requested stream.
    PHP
    $redis->xRead(['mystream' => '$'], 1, 0);
    // MESSAGES
    // [

    // 'STREAMNAME' => [

    // 'MESSAGEID' => [

    // 'KEY' => 'VALUE'

    // ]

    // ],

    // ]
    CLI
    > XREAD BLOCK 0 STREAMS mystream $
    # MESSAGES
    # STREAMNAME [

    # MESSAGEID [

    # KEY

    # VALUE

    # ]

    # ]

    View full-size slide

  12. All clients are served with all the
    entries arriving in a stream.
    How can I scale my reads?

    View full-size slide

  13. Consumer Groups

    View full-size slide

  14. Create a Consumer Group
    Redis stream provides the concept of consumer groups, allowing multiple consumers to
    process the same stream to implement load balancing.
    PHP
    $redis->xGroup(

    'CREATE', 

    'mystream',

    'mygroup'

    );
    // 1/0
    CLI
    > XGROUP CREATE mystream mygroup $
    # 1/0

    View full-size slide

  15. Read data via Consumer Group
    Redis stream provides the concept of consumer groups, allowing multiple consumers to
    process the same stream to implement load balancing.
    PHP
    $redis->xReadGroup(

    'mygroup', 'myconsumer',

    ['mystream' => 0]

    );
    // MESSAGES
    $redis->xReadGroup(

    'mygroup', 'myconsumer',

    ['mystream' => '>']

    );
    CLI
    > XREADGROUP

    GROUP mygroup mycomsumer

    STREAMS mystream 0
    # MESSAGES
    > XREADGROUP

    GROUP mygroup mycomsumer

    STREAMS mystream >

    View full-size slide

  16. Data processed confirmation
    Consumer groups require explicit acknowledgement of the messages successfully processed
    by the consumer.
    PHP
    $redis->xAck(

    'mystream', 

    'mygroup', 

    ['message-id']

    );
    // 1/0
    CLI
    > XACK mystream mygroup message-id
    # 1/0

    View full-size slide

  17. Failure Processing
    – Identify the message data that has been delivered but not confirmed
    – Change the owner of these data and redeliver
    PHP
    $redis->xPending('mystream', 'mygroup');
    // MESSAGE IDs
    $redis->xClaim(

    'mystream', 

    'mygroup',

    'myconsumer',

    3600,

    ['message-id']

    );
    CLI
    > XPENDING mystream mygroup
    # MESSAGE IDs
    > XCLAIM mystream mygroup myconsumer 3600
    message-id

    View full-size slide

  18. Worked pretty good out of the box,
    but the API is quite low-level.
    Let's search for a high-level abstraction.

    View full-size slide

  19. Symfony Messenger
    Version 4.2

    View full-size slide

  20. Create your own transport?

    View full-size slide

  21. declare(strict_types=1);
    namespace HandcraftedInTheAlps\Bundle\RedisTransportBundle\Transport;
    use Symfony\Component\Messenger\Transport\TransportFactoryInterface;
    use Symfony\Component\Messenger\Transport\TransportInterface;
    class RedisStreamTransportFactory implements TransportFactoryInterface
    {
    public function createTransport(string $dsn, array $options): TransportInterface
    {
    $parsedUrl = parse_url($dsn);
    ...
    return new RedisStreamTransport($parsedUrl['host'], $parsedUrl['port'], ...);
    }
    public function supports(string $dsn, array $options): bool
    {
    return 0 === mb_strpos($dsn, 'redis-stream://');
    }
    }

    View full-size slide

  22. The Redis transport was
    introduced in Symfony 4.3
    Open source … FTW!

    View full-size slide

  23. # config/packages/messenger.yaml
    framework:
    messenger:
    buses:
    query_bus: ~
    event_bus: ~
    message_bus:
    middleware:
    - Avodi\Bundle\EventBundle\Common\DomainEvent\DomainEventMiddleware
    - App\Common\Messenger\SymfonyEventMiddleware
    - doctrine_transaction
    routing:
    'Avodi\Bundle\EventBundle\Common\DomainEvent\DomainEventMessage':
    senders: ['domain_events']
    transports:
    domain_events: 'redis://passwor@localhost:6379/mystream/mygroup/myconsumer?auto_setup=true'

    View full-size slide

  24. class DomainEventCollector
    {
    /**
    * @var DomainEventMessage[]
    */
    private $domainEvents = [];
    public function push(DomainEventMessage $event): void
    {
    $this->domainEvents[] = $event;
    }
    /**
    * @return DomainEventMessage[]
    */
    public function release(): array
    {
    $domainEvents = $this->domainEvents;
    $this->domainEvents = [];
    return $domainEvents;
    }
    }

    View full-size slide

  25. public function cancelTicket(Ticket $ticket): void
    {
    $this->workflowRegistry->get($ticket)->apply($ticket, Ticket::TRANSITION_CANCEL);
    $newTicketId = Uuid::uuid4()->toString();
    $this->createTicket($newTicketId, $ticket);
    $this->domainEventCollector->push(
    new DomainEventMessage(
    Ticket::class,
    'canceled',
    $ticket->getId(),
    [
    'newTicketId' => $newTicketId,
    ]
    )
    );
    }

    View full-size slide

  26. class TicketCanceledDomainEventMessageHandler extends AbstractTicketDomainEventMessageHandler
    {
    public function __invoke(DomainEventMessage $message): void
    {
    if ('canceled' !== $message->getType() || Ticket::class !== $message->getName()) {
    return;
    }
    $ticketIds = [$message->getId()];
    $newTicketId = $message->getPayload()['newTicketId'] ?? '';
    if ($newTicketId) {
    $ticketIds[] = $newTicketId;
    } else {
    $this->logger->error('Could not find newTicketId in canceled for access system.', [
    'canceledTicketId' => $message->getId(),
    ]);
    }
    $this->syncTickets($ticketIds);
    }
    }

    View full-size slide

  27. # supervicord.conf
    [program:consumer]
    directory=/var/www/html
    command=bin/console messenger:consume --bus event_bus domain_events --time-limit=3600
    autostart=true
    autorestart=true

    View full-size slide

  28. Trim Data from the stream
    When ~ argument is used, the trimming is performed only when Redis is able to remove a
    whole macro node.
    PHP
    $redis->xTrim('mystream', 1000, true);
    CLI
    > XTRIM mystream MAXLEN ~ 1000

    View full-size slide

  29. class DomainEventHandler implements MessageHandlerInterface
    {
    /**
    * @var ElasticsearchStorage
    */
    private $storage;
    public function __construct(ElasticsearchStorage $storage)
    {
    $this->storage = $storage;
    }
    public function __invoke(DomainEventMessage $message): void
    {
    $this->storage->store($message);
    }
    }

    View full-size slide