Slide 1

Slide 1 text

Confidential │ ©2021 VMware, Inc. RabbitMQ Stream Overview July 13th, 2021 Arnaud Cogoluègnes - RabbitMQ Team

Slide 2

Slide 2 text

Confidential │ ©2021 VMware, Inc. Streams: a new type of data structure in RabbitMQ Models an append-only log Persistent and replicated Non-destructive consumer semantics AMQP 0.9.1 and new protocol

Slide 3

Slide 3 text

Confidential │ ©2021 VMware, Inc. Streams Strong Points Large fan-outs Replay / Time travelling High Throughput Large logs

Slide 4

Slide 4 text

Confidential │ ©2021 VMware, Inc. Why Streams in RabbitMQ? “Traditional” queues have limitations Streams complement queues Streams unlock new use cases

Slide 5

Slide 5 text

Confidential │ ©2021 VMware, Inc. Consuming with a traditional queue M1 M2 C

Slide 6

Slide 6 text

Confidential │ ©2021 VMware, Inc. Consuming with a traditional queue M1 M2 C M1

Slide 7

Slide 7 text

Confidential │ ©2021 VMware, Inc. Consuming with a traditional queue M2 C Ack M1

Slide 8

Slide 8 text

Confidential │ ©2021 VMware, Inc. Consuming with a traditional queue is destructive M1 M2 C M1 M2 C M1 M2 C Ack M1

Slide 9

Slide 9 text

Confidential │ ©2021 VMware, Inc. Fan-out with traditional queues M M C Exchange

Slide 10

Slide 10 text

Confidential │ ©2021 VMware, Inc. Fan-out with traditional queues M M M C C Exchange

Slide 11

Slide 11 text

Confidential │ ©2021 VMware, Inc. Fan-out with traditional queues can require many queues M M M M C C C Exchange

Slide 12

Slide 12 text

Confidential │ ©2021 VMware, Inc. The Log Abstraction ● FIFO data structure ● No destructive reading A stream models an append-only log M 0 M 1 M 2 M 3 M 4 M 5 M 6 Oldest message Offset Last message Next message will be inserted here

Slide 13

Slide 13 text

Confidential │ ©2021 VMware, Inc. Consuming from the beginning of a stream 0 1 C 2 3 4 5 6

Slide 14

Slide 14 text

Confidential │ ©2021 VMware, Inc. Consuming from the end of a stream 0 1 C 2 3 4 5 6

Slide 15

Slide 15 text

Confidential │ ©2021 VMware, Inc. Consuming from a point in the stream 0 1 C 2 3 4 5 6

Slide 16

Slide 16 text

Confidential │ ©2021 VMware, Inc. Any number of consumers can consume from a stream 0 1 C 2 3 4 5 6 C C

Slide 17

Slide 17 text

Confidential │ ©2021 VMware, Inc. A consumer can read the stream several times 0 1 C 2 3 4 5 6 C

Slide 18

Slide 18 text

Confidential │ ©2021 VMware, Inc. A stream is persistent and replicated RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica

Slide 19

Slide 19 text

Confidential │ ©2021 VMware, Inc. Well-behaved publishers and consumers RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica C C P Consume from replica Publish to leader Consume from replica

Slide 20

Slide 20 text

Confidential │ ©2021 VMware, Inc. Streams spread across the cluster RabbitMQ node Stream 1 Leader RabbitMQ node Stream 1 Replica RabbitMQ node Stream 1 Replica Stream 2 Replica Stream 2 Leader Stream 2 Replica Stream 3 Replica Stream 3 Replica Stream 3 Leader

Slide 21

Slide 21 text

Confidential │ ©2021 VMware, Inc. Stream Creation with any AMQP 0.9.1 Client channel.queueDeclare( "my-stream", true, // durable false, false, // not exclusive, not auto-delete Collections.singletonMap("x-queue-type", "stream") );

Slide 22

Slide 22 text

Confidential │ ©2021 VMware, Inc. Consuming from a Stream with any AMQP 0.9.1 Client channel.basicConsume( "my-stream", false, // not auto-ack Collections.singletonMap("x-stream-offset", 0), (s, delivery) -> { }, // delivery callback s -> { } // cancel callback );

Slide 23

Slide 23 text

Confidential │ ©2021 VMware, Inc. A stream is exposed as a regular AMQP 0.9.1 queue M M Classic Queue M Quorum Queue M Stream Exchange

Slide 24

Slide 24 text

Confidential │ ©2021 VMware, Inc. 3-node cluster (c2-standard-16 instances) Streams on AMQP Publishing Rate (messages / second) 39,000 64,600 Quorum Queue Stream on AMQP

Slide 25

Slide 25 text

Confidential │ ©2021 VMware, Inc. 3-node cluster (c2-standard-16 instances) Stream Protocol Publishing Rate (messages / second) 39,000 1,141,754 Quorum Queue Stream with stream protocol 64,600 Stream on AMQP

Slide 26

Slide 26 text

Confidential │ ©2021 VMware, Inc. Streams are also accessible through a new protocol Fast Plugin in core distribution Management integration

Slide 27

Slide 27 text

Confidential │ ©2021 VMware, Inc. Tooling Java client Go client (WIP) Performance tool

Slide 28

Slide 28 text

Confidential │ ©2021 VMware, Inc. Direct interaction between client and streams No exchange involved Stream Stream Client Stream Protocol

Slide 29

Slide 29 text

Confidential │ ©2021 VMware, Inc. Stream Interoperability A stream is accessible from AMQP 0.9.1, MQTT, STOMP, and the stream protocol Stream Stream Client Stream Protocol AMQP 0.9.1 Client AMQP 0.9.1

Slide 30

Slide 30 text

Confidential │ ©2021 VMware, Inc. Traditional Routing with AMQP AMQP 0.9.1 Queue Publisher Publisher Publisher Queue Queue << AMER >> << EMEA >> << APAC >> AMER processing EMEA processing APAC processing AMQP 0.9.1 AMQP 0.9.1 (Possibly) multi-protocol publishers

Slide 31

Slide 31 text

Confidential │ ©2021 VMware, Inc. Add Stream for Analytics AMQP 0.9.1 Queue Publisher Publisher Publisher Queue Queue << AMER >> << EMEA >> << APAC >> << * >> Stream AMER processing EMEA processing APAC processing World-wide analytics AMQP 0.9.1 AMQP 0.9.1 Stream Protocol (Possibly) multi-protocol publishers

Slide 32

Slide 32 text

Confidential │ ©2021 VMware, Inc. 3-node cluster (c2-standard-16 instances) Performance Rate 1.14 M msg/s 1 stream 33.3 MB/s 4.93 M msg/s 143.8 MB/s 5 streams 1 stream 5 streams

Slide 33

Slide 33 text

Confidential │ ©2021 VMware, Inc. 3-node cluster (c2-standard-16 instances) Performance (TLS) Rate 1.09 M msg/s 1 stream (TLS) 1.14 M msg/s 1 stream (plain)

Slide 34

Slide 34 text

Confidential │ ©2021 VMware, Inc. 3-node cluster (c2-standard-16 instances) Performance (sub-entry batching) Rate 4.14 M msg/s 1 stream 93.2 MB/s 16.97 M msg/s 382.3 MB/s 5 streams 1 stream 5 streams

Slide 35

Slide 35 text

Confidential │ ©2021 VMware, Inc.

Slide 36

Slide 36 text

Confidential │ ©2021 VMware, Inc.

Slide 37

Slide 37 text

Confidential │ ©2021 VMware, Inc. Guarantees At-least-once (no message loss) Message de-duplication (publishing) Flow control

Slide 38

Slide 38 text

Confidential │ ©2021 VMware, Inc. Other features Integer- and time-based offset index Retention policy (age or size) Server-side offset tracking Client/server TLS

Slide 39

Slide 39 text

Confidential │ ©2021 VMware, Inc. Stream Java client

Slide 40

Slide 40 text

Confidential │ ©2021 VMware, Inc. Java Client: Stream Management Environment environment = Environment.builder() .build(); environment.streamCreator().stream("my-stream") .create(); environment.deleteStream("my-stream");

Slide 41

Slide 41 text

Confidential │ ©2021 VMware, Inc. Client Environment Java Client: Resource Allocation RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica

Slide 42

Slide 42 text

Confidential │ ©2021 VMware, Inc. Java Client: Producer declaration Producer producer = environment.producerBuilder() .stream("my-stream") .build();

Slide 43

Slide 43 text

Confidential │ ©2021 VMware, Inc. Java Client: Sending a message Message message = producer.messageBuilder() .addData(messagePayload).build(); producer.send( message, confirmationStatus -> { // confirmation callback });

Slide 44

Slide 44 text

Confidential │ ©2021 VMware, Inc. Java Client: Publisher dispatched to stream leader RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica P Connection

Slide 45

Slide 45 text

Confidential │ ©2021 VMware, Inc. Java Client: Sending a message Consumer consumer = environment.consumerBuilder() .stream("my-stream") .offset(OffsetSpecification.first()) .messageHandler((offset, message) -> { // message processing callback }) .build();

Slide 46

Slide 46 text

Confidential │ ©2021 VMware, Inc. Java Client: Consumers dispatched to stream replicas RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica C Connection

Slide 47

Slide 47 text

Confidential │ ©2021 VMware, Inc. Java Client: Configuration-based Producer producer = environment.producerBuilder() .stream("my-stream") .batchSize(1000) // batching configuration .build();

Slide 48

Slide 48 text

Confidential │ ©2021 VMware, Inc. Java Client: Configuration-based environment.consumerBuilder() .stream("my-stream") .name("application-1") .autoTrackingStrategy() // offset tracking .builder() .messageHandler((context, message) -> { }) .build();

Slide 49

Slide 49 text

Confidential │ ©2021 VMware, Inc. Java Client: Consumers spread across replicas RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica C C Connection Connection

Slide 50

Slide 50 text

Confidential │ ©2021 VMware, Inc. Java Client: Connections are re-used RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica C Connection Connection Connection C C C P P

Slide 51

Slide 51 text

Confidential │ ©2021 VMware, Inc. Current State RC! Ships in RabbitMQ 3.9 Try it!

Slide 52

Slide 52 text

Confidential │ ©2021 VMware, Inc. Depending on the feedback. No promises. Roadmap Partitioning Competing consumers

Slide 53

Slide 53 text

Confidential │ ©2021 VMware, Inc. Unlock new scenarios with RabbitMQ Streams: a new replicated and persistent log-type structure in RabbitMQ Large fan-outs Replay / Time travelling High Throughput Large logs Try it!

Slide 54

Slide 54 text

Confidential │ ©2021 VMware, Inc. ● Introductory Blog Post ● Stream Java Client Documentation ○ Good starting point, performance tool is included ○ Ongoing development, API stabilization ● Streams Documentation ● Stream Plugin Documentation ● Source code: ○ Stream Plugin ○ Stream Java Client ○ Stream Go Client Documentation & source code Resources

Slide 55

Slide 55 text

Confidential │ ©2021 VMware, Inc. ● Introductory Blog Post ○ https://blog.rabbitmq.com/posts/2021/07/rabbitmq-streams-overview/ ● Stream Java Client Documentation ○ Good starting point, performance tool is included ○ Ongoing development, API stabilization ○ https://rabbitmq.github.io/rabbitmq-stream-java-client/stable/htmlsingle/ ● Streams Documentation: https://rabbitmq.com/streams.html ● Stream Plugin Documentation: https://rabbitmq.com/stream.html ● Source code: ○ Stream Plugin: https://github.com/rabbitmq/rabbitmq-server/tree/master/deps/rabbitmq_stream ○ Stream Java Client: https://github.com/rabbitmq/rabbitmq-stream-java-client ○ Stream Go Client: https://github.com/rabbitmq/rabbitmq-stream-go-client Documentation & source code Resources

Slide 56

Slide 56 text

Confidential │ ©2021 VMware, Inc. Thank you!