Join the Queue for Apache Kafka® Share Groups

Talk for Data Happy Hour, Minneapolis, Wednesday 10 September, 2025, hosted by SmartThings.
https://luma.com/7muq3c0q

Transcript

1. Paul Brebner Technology Evangelist NetApp Instaclustr Data Happy Hour Talk, Minneapolis, 10 September 2025 © 2025 NetApp, Inc. All rights reserved. JOIN THE QUEUE FOR APACHE KAFKA SHARE GROUPS
2. © 2025 NetApp, Inc. All rights reserved Would you like to experiment with Kafka Queues?!
3. © 2025 NetApp, Inc. All rights reserved. • Experimental (Early Access) in Kafka 4.0 • Uses share groups • Built on KIP-848 • the next generation consumer rebalance protocol • Enables • multiple consumers (> partitions) • to cooperatively process records from the same topic-partition concurrently Now you can! KIP-932: Queues for Kafka (share groups)
4. © 2025 NetApp, Inc. All rights reserved. Default Kafka partition-based consumes – multiple queues Problems? Consumers <= partitions and
5. © 2025 NetApp, Inc. All rights reserved. Default Kafka partition-based consumes – multiple queues Problems? Consumers <= partitions and slow consumers impact all records in partition
6. © 2025 NetApp, Inc. All rights reserved. Single queue with pooled consumers = share groups (concept) Most efficient queue system Unlimited consumers and no slow consumers problem
7. © 2025 NetApp, Inc. All rights reserved. Share groups in practice – partitions are still used Consumers in share groups to share the work to process records as fast as possible
8. © 2025 NetApp, Inc. All rights reserved. When to use Kafka Queues? Scaling for load spikes Consumers > partitions for higher throughput
9. © 2025 NetApp, Inc. All rights reserved. Internally share group consumers still have “queues” (polling batch size)
10. © 2025 NetApp, Inc. All rights reserved. When to use Kafka Queues? Very slow consumers (e.g. 100ms)? Use batch size 1 to reduce latency to minimum possible
11. © 2025 NetApp, Inc. All rights reserved. But watch out!
12. © 2025 NetApp, Inc. All rights reserved. • Batch size 1 will need lots of consumers • Order is now only partial • not per partition, but per batch per partition • With batch size 1 or errors & redelivery • NO ORDERING GURANTEES • In Kafka 4.0 queues must be enabled • unstable.api.versions.enable=true • group.coordinator.rebalance.protocols=classic,consumer,sh are • In consumer code • New constructor, KafkaShareConsumer() and use • Kafka 4.0.0 kafka-clients • Experimental in 4.0.0 • Not upgradeable • 4.1 preview • 4.2 GA But watch out!
13. MORE INFO https://www.instaclustr.com/blog/apache- kafka-4-0-share-groups-what-you-need- to-know-about-queues-for-kafka/ https://www.instaclustr.com/blog/top-use- cases-for-queues-for-kafka-unlocking-the- potential-of-apache-kafka-4-0-share- groups/ LinkedIn © 2025 NetApp, Inc. All rights reserved.