RabbitMQ Streams Overview at RabbitMQ Summit 2021

RabbitMQ is driving an effort to implement a log-type data structure, with stream support on top of the AMQP 0-9-1 protocol and a new blazing fast protocol created for the occasion. Streams unlock scenarios that could have been tedious to implement in the old days of RabbitMQ: large fan-outs, replay & time travel, large logs, very high throughput (1 million messages per second on a 3-node cluster).

Transcript

1. Confidential │ ©2021 VMware, Inc. RabbitMQ Stream Overview July 13th, 2021 Arnaud Cogoluègnes - RabbitMQ Team
2. Confidential │ ©2021 VMware, Inc. Streams: a new type of data structure in RabbitMQ Models an append-only log Persistent and replicated Non-destructive consumer semantics AMQP 0.9.1 and new protocol
3. Confidential │ ©2021 VMware, Inc. Streams Strong Points Large fan-outs Replay / Time travelling High Throughput Large logs
4. Confidential │ ©2021 VMware, Inc. Why Streams in RabbitMQ? “Traditional” queues have limitations Streams complement queues Streams unlock new use cases
5. Confidential │ ©2021 VMware, Inc. Consuming with a traditional queue M1 M2 C
6. Confidential │ ©2021 VMware, Inc. Consuming with a traditional queue M1 M2 C M1
7. Confidential │ ©2021 VMware, Inc. Consuming with a traditional queue M2 C Ack M1
8. Confidential │ ©2021 VMware, Inc. Consuming with a traditional queue is destructive M1 M2 C M1 M2 C M1 M2 C Ack M1
9. Confidential │ ©2021 VMware, Inc. Fan-out with traditional queues M M C Exchange
10. Confidential │ ©2021 VMware, Inc. Fan-out with traditional queues M M M C C Exchange
11. Confidential │ ©2021 VMware, Inc. Fan-out with traditional queues can require many queues M M M M C C C Exchange
12. Confidential │ ©2021 VMware, Inc. The Log Abstraction • FIFO data structure • No destructive reading A stream models an append-only log M 0 M 1 M 2 M 3 M 4 M 5 M 6 Oldest message Offset Last message Next message will be inserted here
13. Confidential │ ©2021 VMware, Inc. Consuming from the beginning of a stream 0 1 C 2 3 4 5 6
14. Confidential │ ©2021 VMware, Inc. Consuming from the end of a stream 0 1 C 2 3 4 5 6
15. Confidential │ ©2021 VMware, Inc. Consuming from a point in the stream 0 1 C 2 3 4 5 6
16. Confidential │ ©2021 VMware, Inc. Any number of consumers can consume from a stream 0 1 C 2 3 4 5 6 C C
17. Confidential │ ©2021 VMware, Inc. A consumer can read the stream several times 0 1 C 2 3 4 5 6 C
18. Confidential │ ©2021 VMware, Inc. A stream is persistent and replicated RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica
19. Confidential │ ©2021 VMware, Inc. Well-behaved publishers and consumers RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica C C P Consume from replica Publish to leader Consume from replica
20. Confidential │ ©2021 VMware, Inc. Streams spread across the cluster RabbitMQ node Stream 1 Leader RabbitMQ node Stream 1 Replica RabbitMQ node Stream 1 Replica Stream 2 Replica Stream 2 Leader Stream 2 Replica Stream 3 Replica Stream 3 Replica Stream 3 Leader
21. Confidential │ ©2021 VMware, Inc. Stream Creation with any AMQP 0.9.1 Client channel.queueDeclare( "my-stream", true, // durable false, false, // not exclusive, not auto-delete Collections.singletonMap("x-queue-type", "stream") );
22. Confidential │ ©2021 VMware, Inc. Consuming from a Stream with any AMQP 0.9.1 Client channel.basicConsume( "my-stream", false, // not auto-ack Collections.singletonMap("x-stream-offset", 0), (s, delivery) -> { }, // delivery callback s -> { } // cancel callback );
23. Confidential │ ©2021 VMware, Inc. A stream is exposed as a regular AMQP 0.9.1 queue M M Classic Queue M Quorum Queue M Stream Exchange
24. Confidential │ ©2021 VMware, Inc. 3-node cluster (c2-standard-16 instances) Streams on AMQP Publishing Rate (messages / second) 39,000 64,600 Quorum Queue Stream on AMQP
25. Confidential │ ©2021 VMware, Inc. 3-node cluster (c2-standard-16 instances) Stream Protocol Publishing Rate (messages / second) 39,000 1,141,754 Quorum Queue Stream with stream protocol 64,600 Stream on AMQP
26. Confidential │ ©2021 VMware, Inc. Streams are also accessible through a new protocol Fast Plugin in core distribution Management integration
27. Confidential │ ©2021 VMware, Inc. Tooling Java client Go client (WIP) Performance tool
28. Confidential │ ©2021 VMware, Inc. Direct interaction between client and streams No exchange involved Stream Stream Client Stream Protocol
29. Confidential │ ©2021 VMware, Inc. Stream Interoperability A stream is accessible from AMQP 0.9.1, MQTT, STOMP, and the stream protocol Stream Stream Client Stream Protocol AMQP 0.9.1 Client AMQP 0.9.1
30. Confidential │ ©2021 VMware, Inc. Traditional Routing with AMQP AMQP 0.9.1 Queue Publisher Publisher Publisher Queue Queue << AMER >> << EMEA >> << APAC >> AMER processing EMEA processing APAC processing AMQP 0.9.1 AMQP 0.9.1 (Possibly) multi-protocol publishers
31. Confidential │ ©2021 VMware, Inc. Add Stream for Analytics AMQP 0.9.1 Queue Publisher Publisher Publisher Queue Queue << AMER >> << EMEA >> << APAC >> << * >> Stream AMER processing EMEA processing APAC processing World-wide analytics AMQP 0.9.1 AMQP 0.9.1 Stream Protocol (Possibly) multi-protocol publishers
32. Confidential │ ©2021 VMware, Inc. 3-node cluster (c2-standard-16 instances) Performance Rate 1.14 M msg/s 1 stream 33.3 MB/s 4.93 M msg/s 143.8 MB/s 5 streams 1 stream 5 streams
33. Confidential │ ©2021 VMware, Inc. 3-node cluster (c2-standard-16 instances) Performance (TLS) Rate 1.09 M msg/s 1 stream (TLS) 1.14 M msg/s 1 stream (plain)
34. Confidential │ ©2021 VMware, Inc. 3-node cluster (c2-standard-16 instances) Performance (sub-entry batching) Rate 4.14 M msg/s 1 stream 93.2 MB/s 16.97 M msg/s 382.3 MB/s 5 streams 1 stream 5 streams
35. Confidential │ ©2021 VMware, Inc.
36. Confidential │ ©2021 VMware, Inc.
37. Confidential │ ©2021 VMware, Inc. Guarantees At-least-once (no message loss) Message de-duplication (publishing) Flow control
38. Confidential │ ©2021 VMware, Inc. Other features Integer- and time-based offset index Retention policy (age or size) Server-side offset tracking Client/server TLS
39. Confidential │ ©2021 VMware, Inc. Stream Java client
40. Confidential │ ©2021 VMware, Inc. Java Client: Stream Management Environment environment = Environment.builder() .build(); environment.streamCreator().stream("my-stream") .create(); environment.deleteStream("my-stream");
41. Confidential │ ©2021 VMware, Inc. Client Environment Java Client: Resource Allocation RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica
42. Confidential │ ©2021 VMware, Inc. Java Client: Producer declaration Producer producer = environment.producerBuilder() .stream("my-stream") .build();
43. Confidential │ ©2021 VMware, Inc. Java Client: Sending a message Message message = producer.messageBuilder() .addData(messagePayload).build(); producer.send( message, confirmationStatus -> { // confirmation callback });
44. Confidential │ ©2021 VMware, Inc. Java Client: Publisher dispatched to stream leader RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica P Connection
45. Confidential │ ©2021 VMware, Inc. Java Client: Sending a message Consumer consumer = environment.consumerBuilder() .stream("my-stream") .offset(OffsetSpecification.first()) .messageHandler((offset, message) -> { // message processing callback }) .build();
46. Confidential │ ©2021 VMware, Inc. Java Client: Consumers dispatched to stream replicas RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica C Connection
47. Confidential │ ©2021 VMware, Inc. Java Client: Configuration-based Producer producer = environment.producerBuilder() .stream("my-stream") .batchSize(1000) // batching configuration .build();
48. Confidential │ ©2021 VMware, Inc. Java Client: Configuration-based environment.consumerBuilder() .stream("my-stream") .name("application-1") .autoTrackingStrategy() // offset tracking .builder() .messageHandler((context, message) -> { }) .build();
49. Confidential │ ©2021 VMware, Inc. Java Client: Consumers spread across replicas RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica C C Connection Connection
50. Confidential │ ©2021 VMware, Inc. Java Client: Connections are re-used RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica C Connection Connection Connection C C C P P
51. Confidential │ ©2021 VMware, Inc. Current State RC! Ships in RabbitMQ 3.9 Try it!
52. Confidential │ ©2021 VMware, Inc. Depending on the feedback. No promises. Roadmap Partitioning Competing consumers
53. Confidential │ ©2021 VMware, Inc. Unlock new scenarios with RabbitMQ Streams: a new replicated and persistent log-type structure in RabbitMQ Large fan-outs Replay / Time travelling High Throughput Large logs Try it!
54. Confidential │ ©2021 VMware, Inc. • Introductory Blog Post • Stream Java Client Documentation ◦ Good starting point, performance tool is included ◦ Ongoing development, API stabilization • Streams Documentation • Stream Plugin Documentation • Source code: ◦ Stream Plugin ◦ Stream Java Client ◦ Stream Go Client Documentation & source code Resources
55. Confidential │ ©2021 VMware, Inc. • Introductory Blog Post ◦ https://blog.rabbitmq.com/posts/2021/07/rabbitmq-streams-overview/ • Stream Java Client Documentation ◦ Good starting point, performance tool is included ◦ Ongoing development, API stabilization ◦ https://rabbitmq.github.io/rabbitmq-stream-java-client/stable/htmlsingle/ • Streams Documentation: https://rabbitmq.com/streams.html • Stream Plugin Documentation: https://rabbitmq.com/stream.html • Source code: ◦ Stream Plugin: https://github.com/rabbitmq/rabbitmq-server/tree/master/deps/rabbitmq_stream ◦ Stream Java Client: https://github.com/rabbitmq/rabbitmq-stream-java-client ◦ Stream Go Client: https://github.com/rabbitmq/rabbitmq-stream-go-client Documentation & source code Resources
56. Confidential │ ©2021 VMware, Inc. Thank you!