For the braves ones: Dive into the kafka wire protocol

This talk was given at Current London 2025. The code from the live coding part can be found here: https://github.com/tiffland/kafka-wire-protocol-current25

Transcript

1. THINKPORT For th e br ave s on es: Di ve i nto th e K afka w ire pr otoc ol T h o m a s I f f l a n d , C u r r e n t L o n d o n 2 0 2 5
2. A B O U T M E Thomas Iffland Cloud Architect @ Thinkport Consultancy in Germany & international Join us for the Kafka Meetup Frankfurt! https://www.linkedin.com/in/thomas-iffland/ THINKPORT
3. Join the Confluent community slack here!
4. THINKPORT Why? • It‘s fun! • It might give you a better understanding • Technologies using Kafka protocol: • WarpStream • RedPanda • AutoMQ • …
5. THINKPORT General • Communication over TCP • Request/response pairs • Bidirectional compatibility (be aware of KIP-896) • Supports batching • Order is guaranteed
6. THINKPORT Data type Description INT32 Integer value between -231 and 231-1. UNSIGNED_VARINT A positive integer encoded using varint (Google Protocol Buffers format) COMPACT_STRING The length of the string in bytes as UNSIGNED_VARINT + 1, followed by the payload. COMPACT_ARRAY The number of objects in the array as UNSIGNED_VARINT + 1, followed by the payload.
14. THINKPORT Follow up links • Kafka Protocol Guide: https://kafka.apache.org/protocol.html • Repo: https://github.com/tiffland/kafka-wire-protocol-current25 • Kafka protocol practical guide by Ivan Yurchenko Q&A