Slide 1

Slide 1 text

PRESENTED BY

Slide 2

Slide 2 text

PRESENTED BY • Co-founder & CEO of Sulu GmbH • More than 15 years of experience in web technologies & development • PHP, Symfony, React, SQL, Redis, Elasticsearch, … • Open source enthusiast • Loves cooking and mountains Hi, I’m Thomas Schedler chirimoya [email protected]

Slide 3

Slide 3 text

PRESENTED BY 1 The story how I came across Redis Streams 2 Introduction to Redis Stream commands 3 Real-world application using Redis Streams Agenda:

Slide 4

Slide 4 text

PRESENTED BY https://github.com/matthiasnoback/building-autonomous-services-workshop

Slide 5

Slide 5 text

PRESENTED BY Tell, don’t ask. Generate an optimised read model for your service by 
 consuming domain events from a central stream.

Slide 6

Slide 6 text

PRESENTED BY tail -f -n 1 Probably not the most robust implementation
 for a real-world application.

Slide 7

Slide 7 text

PRESENTED BY

Slide 8

Slide 8 text

PRESENTED BY In computer science, a stream is a sequence of data elements made available over time. “

Slide 9

Slide 9 text

PRESENTED BY Stream 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 18 17 consumer 1 position consumer 2 position new record truncatable records producer emitting records

Slide 10

Slide 10 text

PRESENTED BY https://brandur.org/redis-streams

Slide 11

Slide 11 text

PRESENTED BY Redis Streams A new data type introduced with Redis 5.0.0.

Slide 12

Slide 12 text

PRESENTED BY Assuming you have installed Redis server and client.
 And keep in mind, we need to use the latest version of Redis 5.0. Connect to Redis PHP $redis = new \Redis(); $redis->connect('127.0.0.1', '6379'); CLI > redis-cli -h 127.0.0.1 -p 6379

Slide 13

Slide 13 text

PRESENTED BY The stream elements can be one or more key-value pairs.
 Let's add elements to the stream: Add message to stream PHP $redis->xAdd('mystream', '*', [
 'KEY' => 'VALUE'
 ]); // MESSAGE ID (e.g. 1526919030474-55) $redis->xLen('mystream'); // (integer) 1 CLI > XADD mystream * KEY VALUE # MESSAGE ID (e.g. 1526919030474-55) > XLEN mystream # (integer) 1

Slide 14

Slide 14 text

PRESENTED BY Read message from one or multiple streams, only returning entries with an ID greater than the last received ID reported by the caller. Read message from stream PHP $redis->xRead(['mystream' => 0], 2); // MESSAGES // [
 // 'STREAMNAME' => [
 // 'MESSAGEID' => [
 // 'KEY' => 'VALUE'
 // ],
 // 'MESSAGEID' => [
 // 'KEY' => 'VALUE' CLI > XREAD COUNT 2 STREAMS mystream 0 # MESSAGES # STREAMNAME [
 # MESSAGEID [
 # KEY
 # VALUE
 # ]

Slide 15

Slide 15 text

PRESENTED BY In order to avoid polling at a fixed or adaptive interval, the command is able to block until new entries get pushed into the requested stream. Block for data PHP $redis->xRead(['mystream' => '$'], 1, 0); // MESSAGES // [
 // 'STREAMNAME' => [
 // 'MESSAGEID' => [
 // 'KEY' => 'VALUE'
 // ]
 // ]
 // ] CLI > XREAD BLOCK 0 STREAMS mystream $ # MESSAGES # STREAMNAME [
 # MESSAGEID [
 # KEY
 # VALUE
 # ]

Slide 16

Slide 16 text

PRESENTED BY All clients are served with all the entries arriving in a stream. How can we scale reads?

Slide 17

Slide 17 text

PRESENTED BY Consumer Groups 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 18 17 group a consumer 1 position group b consumer 2 position new record truncatable records group a consumer 2 position group b consumer 1 position

Slide 18

Slide 18 text

PRESENTED BY Redis stream provides the concept of consumer groups, allowing multiple consumers to process the same stream to implement load balancing. Create a Consumer Group PHP $redis->xGroup(
 'CREATE', 
 'mystream',
 'mygroup'
 ); // 1/0 CLI > XGROUP CREATE mystream mygroup $ # 1/0

Slide 19

Slide 19 text

PRESENTED BY Redis stream provides the concept of consumer groups, allowing multiple consumers to process the same stream to implement load balancing. Read data via Consumer Group PHP $redis->xReadGroup(
 'mygroup', 'myconsumer',
 ['mystream' => 0]
 ); // MESSAGES $redis->xReadGroup(
 'mygroup', 'myconsumer',
 ['mystream' => '>']
 ); CLI > XREADGROUP
 GROUP mygroup mycomsumer
 STREAMS mystream 0 # MESSAGES > XREADGROUP
 GROUP mygroup mycomsumer
 STREAMS mystream >

Slide 20

Slide 20 text

PRESENTED BY Consumer groups require explicit acknowledgement of the messages successfully processed by the consumer. Data processed confirmation PHP $redis->xAck(
 'mystream', 
 'mygroup', 
 ['message-id']
 ); // 1/0 CLI > XACK mystream mygroup message-id # 1/0

Slide 21

Slide 21 text

PRESENTED BY • Identify the message data that has been delivered but not confirmed • Change the owner of these data and redeliver Failure Processing PHP $redis->xPending('mystream', 'mygroup'); // MESSAGE IDs $redis->xClaim(
 'mystream', 
 'mygroup',
 'myconsumer',
 3600,
 ['message-id']
 ); CLI > XPENDING mystream mygroup # MESSAGE IDs > XCLAIM mystream mygroup myconsumer 3600 message-id

Slide 22

Slide 22 text

PRESENTED BY Worked pretty good out of the box, but the API is quite low-level. Let's search for a high-level abstraction.

Slide 23

Slide 23 text

PRESENTED BY

Slide 24

Slide 24 text

PRESENTED BY • The Messenger component helps applications send and receive messages — message bus • By default, messages are handled as soon as they are dispatched • If you want to handle a message asynchronously, you can configure a transport Symfony Messenger Component

Slide 25

Slide 25 text

PRESENTED BY Create our own transport? No official Redis transport — Symfony Messenger 4.2

Slide 26

Slide 26 text

declare(strict_types=1); namespace HandcraftedInTheAlps\Bundle\RedisTransportBundle\Transport; use Symfony\Component\Messenger\Transport\TransportFactoryInterface; use Symfony\Component\Messenger\Transport\TransportInterface; class RedisStreamTransportFactory implements TransportFactoryInterface { public function createTransport(string $dsn, array $options): TransportInterface { $parsedUrl = parse_url($dsn); ... return new RedisStreamTransport($parsedUrl['host'], $parsedUrl['port'], ...); } public function supports(string $dsn, array $options): bool { return 0 === mb_strpos($dsn, 'redis-stream://'); } }

Slide 27

Slide 27 text

PRESENTED BY

Slide 28

Slide 28 text

PRESENTED BY

Slide 29

Slide 29 text

PRESENTED BY

Slide 30

Slide 30 text

PRESENTED BY

Slide 31

Slide 31 text

PRESENTED BY

Slide 32

Slide 32 text

PRESENTED BY The Redis Streams transport was introduced in Symfony 4.3 Open source … FTW!

Slide 33

Slide 33 text

PRESENTED BY Real-world showcase — Ticketing Application Ticketing Application Ticket Access Order Event ... Access System REST API

Slide 34

Slide 34 text

PRESENTED BY Real-world showcase — Ticketing Application Ticketing Application Ticket Access Order Event ... Access System REST API Message Bus Contingent

Slide 35

Slide 35 text

PRESENTED BY Tell, don’t ask. Domain Events are used to capture things that can trigger 
 a change to the state of your application.

Slide 36

Slide 36 text

class DomainEventMessage { ... public function __construct( string $name, string $type, string $id, array $payload = [], ?\DateTimeImmutable $created = null ) { $this->name = $name; $this->type = $type; $this->id = $id; $this->payload = $payload; $this->created = $created ?? new \DateTimeImmutable(); } ... }

Slide 37

Slide 37 text

public function cancelTicket(Ticket $ticket): void { $this->workflowRegistry->get($ticket)->apply($ticket, Ticket::TRANSITION_CANCEL); $newTicketId = Uuid::uuid4()->toString(); $this->createTicket($newTicketId, $ticket); $this->domainEventCollector->push( new DomainEventMessage( Ticket::class, 'canceled', $ticket->getId(), [ 'newTicketId' => $newTicketId, ] ) ); }

Slide 38

Slide 38 text

class TicketCanceledDomainEventMessageHandler extends AbstractTicketDomainEventMessageHandler { public function __invoke(DomainEventMessage $message): void { if ('canceled' !== $message->getType() || Ticket::class !== $message->getName()) { return; } $ticketIds = [$message->getId()]; $newTicketId = $message->getPayload()['newTicketId'] ?? ''; if ($newTicketId) { $ticketIds[] = $newTicketId; } else { $this->logger->error('Could not find newTicketId in canceled for access system.', [ 'canceledTicketId' => $message->getId(), ]); } $this->syncTickets($ticketIds); } }

Slide 39

Slide 39 text

PRESENTED BY When ~ argument is used, the trimming is performed only when Redis is able to remove a whole macro node. Trim Data from the stream PHP $redis->xTrim('mystream', 1000, true); CLI > XTRIM mystream MAXLEN ~ 1000

Slide 40

Slide 40 text

class DomainEventHandler implements MessageHandlerInterface { /** * @var ElasticsearchStorage */ private $storage; public function __construct(ElasticsearchStorage $storage) { $this->storage = $storage; } public function __invoke(DomainEventMessage $message): void { $this->storage->store($message); } }

Slide 41

Slide 41 text

Thank you! Thank you!