Upgrade to Pro — share decks privately, control downloads, hide ads and more …

RabbitMQ Streams Overview at RabbitMQ Summit 2021

RabbitMQ Streams Overview at RabbitMQ Summit 2021

RabbitMQ is driving an effort to implement a log-type data structure, with stream support on top of the AMQP 0-9-1 protocol and a new blazing fast protocol created for the occasion. Streams unlock scenarios that could have been tedious to implement in the old days of RabbitMQ: large fan-outs, replay & time travel, large logs, very high throughput (1 million messages per second on a 3-node cluster).

Fff70c641a9988db06414c9beb29029f?s=128

Arnaud Cogoluègnes

July 14, 2021
Tweet

Transcript

  1. Confidential │ ©2021 VMware, Inc. RabbitMQ Stream Overview July 13th,

    2021 Arnaud Cogoluègnes - RabbitMQ Team
  2. Confidential │ ©2021 VMware, Inc. Streams: a new type of

    data structure in RabbitMQ Models an append-only log Persistent and replicated Non-destructive consumer semantics AMQP 0.9.1 and new protocol
  3. Confidential │ ©2021 VMware, Inc. Streams Strong Points Large fan-outs

    Replay / Time travelling High Throughput Large logs
  4. Confidential │ ©2021 VMware, Inc. Why Streams in RabbitMQ? “Traditional”

    queues have limitations Streams complement queues Streams unlock new use cases
  5. Confidential │ ©2021 VMware, Inc. Consuming with a traditional queue

    M1 M2 C
  6. Confidential │ ©2021 VMware, Inc. Consuming with a traditional queue

    M1 M2 C M1
  7. Confidential │ ©2021 VMware, Inc. Consuming with a traditional queue

    M2 C Ack M1
  8. Confidential │ ©2021 VMware, Inc. Consuming with a traditional queue

    is destructive M1 M2 C M1 M2 C M1 M2 C Ack M1
  9. Confidential │ ©2021 VMware, Inc. Fan-out with traditional queues M

    M C Exchange
  10. Confidential │ ©2021 VMware, Inc. Fan-out with traditional queues M

    M M C C Exchange
  11. Confidential │ ©2021 VMware, Inc. Fan-out with traditional queues can

    require many queues M M M M C C C Exchange
  12. Confidential │ ©2021 VMware, Inc. The Log Abstraction • FIFO

    data structure • No destructive reading A stream models an append-only log M 0 M 1 M 2 M 3 M 4 M 5 M 6 Oldest message Offset Last message Next message will be inserted here
  13. Confidential │ ©2021 VMware, Inc. Consuming from the beginning of

    a stream 0 1 C 2 3 4 5 6
  14. Confidential │ ©2021 VMware, Inc. Consuming from the end of

    a stream 0 1 C 2 3 4 5 6
  15. Confidential │ ©2021 VMware, Inc. Consuming from a point in

    the stream 0 1 C 2 3 4 5 6
  16. Confidential │ ©2021 VMware, Inc. Any number of consumers can

    consume from a stream 0 1 C 2 3 4 5 6 C C
  17. Confidential │ ©2021 VMware, Inc. A consumer can read the

    stream several times 0 1 C 2 3 4 5 6 C
  18. Confidential │ ©2021 VMware, Inc. A stream is persistent and

    replicated RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica
  19. Confidential │ ©2021 VMware, Inc. Well-behaved publishers and consumers RabbitMQ

    node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica C C P Consume from replica Publish to leader Consume from replica
  20. Confidential │ ©2021 VMware, Inc. Streams spread across the cluster

    RabbitMQ node Stream 1 Leader RabbitMQ node Stream 1 Replica RabbitMQ node Stream 1 Replica Stream 2 Replica Stream 2 Leader Stream 2 Replica Stream 3 Replica Stream 3 Replica Stream 3 Leader
  21. Confidential │ ©2021 VMware, Inc. Stream Creation with any AMQP

    0.9.1 Client channel.queueDeclare( "my-stream", true, // durable false, false, // not exclusive, not auto-delete Collections.singletonMap("x-queue-type", "stream") );
  22. Confidential │ ©2021 VMware, Inc. Consuming from a Stream with

    any AMQP 0.9.1 Client channel.basicConsume( "my-stream", false, // not auto-ack Collections.singletonMap("x-stream-offset", 0), (s, delivery) -> { }, // delivery callback s -> { } // cancel callback );
  23. Confidential │ ©2021 VMware, Inc. A stream is exposed as

    a regular AMQP 0.9.1 queue M M Classic Queue M Quorum Queue M Stream Exchange
  24. Confidential │ ©2021 VMware, Inc. 3-node cluster (c2-standard-16 instances) Streams

    on AMQP Publishing Rate (messages / second) 39,000 64,600 Quorum Queue Stream on AMQP
  25. Confidential │ ©2021 VMware, Inc. 3-node cluster (c2-standard-16 instances) Stream

    Protocol Publishing Rate (messages / second) 39,000 1,141,754 Quorum Queue Stream with stream protocol 64,600 Stream on AMQP
  26. Confidential │ ©2021 VMware, Inc. Streams are also accessible through

    a new protocol Fast Plugin in core distribution Management integration
  27. Confidential │ ©2021 VMware, Inc. Tooling Java client Go client

    (WIP) Performance tool
  28. Confidential │ ©2021 VMware, Inc. Direct interaction between client and

    streams No exchange involved Stream Stream Client Stream Protocol
  29. Confidential │ ©2021 VMware, Inc. Stream Interoperability A stream is

    accessible from AMQP 0.9.1, MQTT, STOMP, and the stream protocol Stream Stream Client Stream Protocol AMQP 0.9.1 Client AMQP 0.9.1
  30. Confidential │ ©2021 VMware, Inc. Traditional Routing with AMQP AMQP

    0.9.1 Queue Publisher Publisher Publisher Queue Queue << AMER >> << EMEA >> << APAC >> AMER processing EMEA processing APAC processing AMQP 0.9.1 AMQP 0.9.1 (Possibly) multi-protocol publishers
  31. Confidential │ ©2021 VMware, Inc. Add Stream for Analytics AMQP

    0.9.1 Queue Publisher Publisher Publisher Queue Queue << AMER >> << EMEA >> << APAC >> << * >> Stream AMER processing EMEA processing APAC processing World-wide analytics AMQP 0.9.1 AMQP 0.9.1 Stream Protocol (Possibly) multi-protocol publishers
  32. Confidential │ ©2021 VMware, Inc. 3-node cluster (c2-standard-16 instances) Performance

    Rate 1.14 M msg/s 1 stream 33.3 MB/s 4.93 M msg/s 143.8 MB/s 5 streams 1 stream 5 streams
  33. Confidential │ ©2021 VMware, Inc. 3-node cluster (c2-standard-16 instances) Performance

    (TLS) Rate 1.09 M msg/s 1 stream (TLS) 1.14 M msg/s 1 stream (plain)
  34. Confidential │ ©2021 VMware, Inc. 3-node cluster (c2-standard-16 instances) Performance

    (sub-entry batching) Rate 4.14 M msg/s 1 stream 93.2 MB/s 16.97 M msg/s 382.3 MB/s 5 streams 1 stream 5 streams
  35. Confidential │ ©2021 VMware, Inc.

  36. Confidential │ ©2021 VMware, Inc.

  37. Confidential │ ©2021 VMware, Inc. Guarantees At-least-once (no message loss)

    Message de-duplication (publishing) Flow control
  38. Confidential │ ©2021 VMware, Inc. Other features Integer- and time-based

    offset index Retention policy (age or size) Server-side offset tracking Client/server TLS
  39. Confidential │ ©2021 VMware, Inc. Stream Java client

  40. Confidential │ ©2021 VMware, Inc. Java Client: Stream Management Environment

    environment = Environment.builder() .build(); environment.streamCreator().stream("my-stream") .create(); environment.deleteStream("my-stream");
  41. Confidential │ ©2021 VMware, Inc. Client Environment Java Client: Resource

    Allocation RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica
  42. Confidential │ ©2021 VMware, Inc. Java Client: Producer declaration Producer

    producer = environment.producerBuilder() .stream("my-stream") .build();
  43. Confidential │ ©2021 VMware, Inc. Java Client: Sending a message

    Message message = producer.messageBuilder() .addData(messagePayload).build(); producer.send( message, confirmationStatus -> { // confirmation callback });
  44. Confidential │ ©2021 VMware, Inc. Java Client: Publisher dispatched to

    stream leader RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica P Connection
  45. Confidential │ ©2021 VMware, Inc. Java Client: Sending a message

    Consumer consumer = environment.consumerBuilder() .stream("my-stream") .offset(OffsetSpecification.first()) .messageHandler((offset, message) -> { // message processing callback }) .build();
  46. Confidential │ ©2021 VMware, Inc. Java Client: Consumers dispatched to

    stream replicas RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica C Connection
  47. Confidential │ ©2021 VMware, Inc. Java Client: Configuration-based Producer producer

    = environment.producerBuilder() .stream("my-stream") .batchSize(1000) // batching configuration .build();
  48. Confidential │ ©2021 VMware, Inc. Java Client: Configuration-based environment.consumerBuilder() .stream("my-stream")

    .name("application-1") .autoTrackingStrategy() // offset tracking .builder() .messageHandler((context, message) -> { }) .build();
  49. Confidential │ ©2021 VMware, Inc. Java Client: Consumers spread across

    replicas RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica C C Connection Connection
  50. Confidential │ ©2021 VMware, Inc. Java Client: Connections are re-used

    RabbitMQ node Stream Replica RabbitMQ node Stream Leader RabbitMQ node Stream Replica C Connection Connection Connection C C C P P
  51. Confidential │ ©2021 VMware, Inc. Current State RC! Ships in

    RabbitMQ 3.9 Try it!
  52. Confidential │ ©2021 VMware, Inc. Depending on the feedback. No

    promises. Roadmap Partitioning Competing consumers
  53. Confidential │ ©2021 VMware, Inc. Unlock new scenarios with RabbitMQ

    Streams: a new replicated and persistent log-type structure in RabbitMQ Large fan-outs Replay / Time travelling High Throughput Large logs Try it!
  54. Confidential │ ©2021 VMware, Inc. • Introductory Blog Post •

    Stream Java Client Documentation ◦ Good starting point, performance tool is included ◦ Ongoing development, API stabilization • Streams Documentation • Stream Plugin Documentation • Source code: ◦ Stream Plugin ◦ Stream Java Client ◦ Stream Go Client Documentation & source code Resources
  55. Confidential │ ©2021 VMware, Inc. • Introductory Blog Post ◦

    https://blog.rabbitmq.com/posts/2021/07/rabbitmq-streams-overview/ • Stream Java Client Documentation ◦ Good starting point, performance tool is included ◦ Ongoing development, API stabilization ◦ https://rabbitmq.github.io/rabbitmq-stream-java-client/stable/htmlsingle/ • Streams Documentation: https://rabbitmq.com/streams.html • Stream Plugin Documentation: https://rabbitmq.com/stream.html • Source code: ◦ Stream Plugin: https://github.com/rabbitmq/rabbitmq-server/tree/master/deps/rabbitmq_stream ◦ Stream Java Client: https://github.com/rabbitmq/rabbitmq-stream-java-client ◦ Stream Go Client: https://github.com/rabbitmq/rabbitmq-stream-go-client Documentation & source code Resources
  56. Confidential │ ©2021 VMware, Inc. Thank you!