Devoxx Greece 2026 - Apache Kafka as a Queue: Bridging Event Streaming and Point-to-Point Messaging

Delivered at Devoxx Greece 2026 in Athens. Apache Kafka as a Queue: Bridging Event Streaming and Point-to-Point Messaging

Apache Kafka has established itself as the leading platform for event streaming, yet architects and engineers have often turned to other systems to fulfill point-to-point messaging needs. With KIP-932—Queues for Kafka—Kafka now natively supports queue-like semantics.

In this session, we’ll dive into the cooperative consumption model that brings message-level acknowledgments to Kafka via the new ShareConsumer API. We’ll trace the journey of a message and demonstrate how developers can programmatically manage message state. Queue-driven tasks are often long-running, with a non-deterministic return time - and the API has this use case covered, as well. Observability is essential in distributed systems - so we’ll also explore the metrics and configuration options that provide essential visibility. Finally, we’ll walk through some sample code - because slides never make it to production.

Queues for Kafka give developers a seamless way to meet messaging requirements using the Kafka protocol—without altering how messages are produced. This solution delivers point-to-point messaging with all the core guarantees of Apache Kafka, including durable storage, horizontal scalability, and fault tolerance. Discover how “queues the Kafka way” can elevate your messaging architecture.

Find the talk here -> https://bit.ly/48QhDLT

Transcript

1. Sandon Jacobs Bridging Event Streaming and Point-to-Point Messaging Apache Ka fk a as a Queue
2. Senior Developer Advocate, Con fl uent Sandon Jacobs • B a sed in R a leigh, North C a rolin a (USA) • N a me a JVM l a ngu a ge, I’ve kicked the tires…
3. Queues for Ka fk a •Event-Driven Architecture •Event Stre a ming a nd Point-to-Point Mess a ging •Queues for K af k a •Look a t some (simple) s a mple code Why are we here?
5. Event Streaming
6. What is an event? Producing Events He a ders (option a l) Key V a lue
7. Send it… Producing Events Topic P a rtition P a rtition P a rtition P a
8. Topic-P a rtition How data is stored… Inside a Topic-Partition 0 1 2 3 4 5 6 7 8 O ff set He a ders Key V a lue Older Newer
9. group.id=foo Understanding Partition Assignment Consumer Groups Topic Consumer Consumer Consumer P a rtition P a rtition P a rtition Consumer
10. How to connect… Consumer Group ID How often to commit o ff sets New Consumer Inst a nce Subscribe to topic(s)… Poll for events Process events
11. Consumer Groups Consumer Consumer Consumer P a rtition h a s ONE Consumer a t a time P a rellel processing Ordering gu a r a ntees
12. Point to Point Messaging
13. Queue The “work-queue” use case • Async processing of a t a sk • Mess a ge consumed by ONLY ONE process • Mess a ge-level a cknowledgment!!! • Delete on a cknowledgement
14. Not a great fi t for Apache Ka fk a Consumer Groups
15. Queues for Ka fk a
16. Share Partition • Sh a re group’s view of a topic-p a rtition • Owned by the p a rtition’s le a der • Responsible for: • Tr a cking a cknowledgements • M a n a ging record fetch/poll • M a n a ge locks a nd timeouts Where am I in the log? P a rtition Sh a re P a rtition Consumer Sh a re Consumer Sh a re Consumer Sh a re Consumer 1-to-1 rel a tionship Cooper a tive Consumption
17. share.acknowledgement.mode=implicit Message Acknowledgment • Def a ult a cknowledgment mode v a lue • API m a n a ges mess a ge a cknowledgement • Simil a r to a utom a tic o ff set commits in tr a dition a l K af k a Consumer cl a ss
18. share.acknowledgement.mode=explicit Message Acknowledgment • Gr a nul a r control of mess a ge a cknowledgments a nd lifecycle • Con f igure Sh a reConsumer • Must c a ll a cknowledge method for e a ch record with desired AcknowledgementType v a lue • Be speci f ic a bout di ff erent exception types Set this!!! Go-right p a th… Error, but we w a nt to retry this…
19. Go-Right Path… Topic P a rtition Sh a re P a rtition 0 1 2 3 200 201 205 202 203 204 O ff set St a te Delivery Count Ack Av Av Av Av Av Av a il a ble Acquired Acknowledged Archived 1 DC++ Av Av Av Av Aq Ack 1 SPSO SPEO SPEO SPSO
20. Bad Message Topic P a rtition Sh a re P a rtition Ack Av Av Av a il a ble Acquired Acknowledged Archived Ack 1 Av Av Av Av Av 1 DC++ Acq Av Arch 1 SPSO SPEO SPSO SPEO O ff set St a te Delivery Count 0 1 2 3 200 201 205 202 203 204
21. Retry Loop Topic P a rtition Sh a re P a rtition Ack Arch Av Av Av a il a ble Acquired Acknowledged Archived Ack 1 Av Av Av Av Av 1 DC++ Msg Rele a DC < 5? Acq 1 1 SPSO SPEO O ff set St a te Delivery Count 0 1 2 3 200 201 205 202 203 204
22. Retries Exhausted Topic P a rtition Sh a re P a rtition Ack Arch Av Av Av a il a ble Acquired Acknowledged Archived Ack 1 Av Av Av Av Av 1 DC++ Msg Rele a DC < 5? Acq 5 4 Arch 1 SPSO SPEO SPSO SPEO O ff set St a te Delivery Count 0 1 2 3 200 201 205 202 203 204
23. Long Running Tasks Topic P a rtition Sh a re P a rtition Ack Arch Av Av Av a il a ble Acquired Acknowledged Archived Ack 1 Av Av Av Av Av 1 DC++ Acq 4 3 1 SPSO SPEO SPSO SPEO Ack O ff set St a te Delivery Count 0 1 2 3 200 201 205 202 203 204
24. At some point in time… Topic P a rtition Sh a re P a rtition Ack Arch Av Ack 1 Av Acq Ack Av 1 5 Arch 1 1 1 SPSO SPEO In-Flight Records 5 Arch 2 O ff set St a te Delivery Count 0 1 2 3 200 201 401 202 203 400
25. Broker Con fi gurations • group.sh a re.delivery.count.limit How m a ny times we a ttempt to deliver a n event, def a ult is 5 • group.sh a re.record.lock.dur a tion.ms How long the a cquisition lock lives, def a ult is 30000 (30 seconds) • group.sh a re.p a rtition.m a x.record.locks How m a ny in- f light records per sh a re-p a rtition, def a ult is 200
26. Let’s see it in action…
27. Land the plane…
28. When can I expect to see this? Release timeline Ap a che K af k a 4.+ More Enh a ncements Q2 2026 a nd beyond Ap a che K af k a 4.2 Production Re a dy! J a n/Feb 2026 Ap a che K af k a 4.1 Preview Sept 2025
29. What’s (possibly) next: AK 4.3+ • De a d-letter queue (by con f igur a tion or convention) • Del a yed retry, i.e. exponenti a l b a cko ff • R a ck- a w a re p a rtition a ssignments • Key-b a sed ordering
30. In conclusion… • Altern a tive to tr a dition a l consumer groups • Consume from the s a me p a rtition in a cooper a tive m a nner • Sc a le consumers beyond number of topic-p a rtitions • Mess a ge-level a cknowledgement! • Built-in a retry mech a nism • No m a x queue depth issues! If you h a ve disk, you’re good! • No Producer ch a nges needed! • Process the s a me d a t a with stre a m processing.
31. More resources…