Everything I know about Kafka performance, and everything I don't know - but potentially could know if we had a Kafka performance model! Performance Engineering track keynote talk from Community over Code NA 2025 (including track intro).
streamlined "rocket" nose cone Australian locomotive (Wikipedia) + Fastest steam locomotive between Sydney and Newcastle (1964) + 1st steam locomotive to cross the Australian continent
Foga 2. C/C Asia Beijing 2023 - Willem Jiang 3. C/C NA Halifax 2023 - Roger Abelenda 4. C/C EU Bratislava 2024 - Stefan Vodita 5. C/C Asia Hangzhou 2024 - Yu Xiao 6. C/C NA Denver 2024 - Roger Abelenda 7. C/C NA Minneapolis 2025 – Roger Abelenda Approx 25% acceptance rate, 43 talks, 600+ attendees Talk acceptance algorithm = Performance Engineering + Apache Project (or open source value) + Interesting Thanks to co-chairs, reviewers, and volunteers and planners/conference PC
Goods moving around on 100’s of trucks between 100’s of warehouses • Real-time environmental and safety rules checking • Design choices • Many topics (one topic per location) and many consumer groups • Vs. One topic, single consumer group • Performance? “Kongo” - My 1st Kafka Application (2018) – a delivery/logistics demo Many topics
performs 155 x better Many topics, many consumer groups, 7200 Single topic, single consumer group, 1120000 0 200000 400000 600000 800000 1000000 1200000 Many topics Single topic TPS Why? Many topics = many partitions And high fan-out is expensive
Kafka + Kubernetes • (Too) easy to increase load with Kubernetes! • Hard to get good scalability Anomalia Machina – 2nd Application – real-time anomaly detection
consumers • Single-threaded • Consumers <= Partitions • Little’s Law predicts for higher throughput you need • More partitions/consumers • More parallelism per consumer • Or, reduced consumer latency • More resources for consumer servers • Also applies to Kafka Connectors • Number of Kafka Connector tasks (It’s good to avoid ) Slow Consumers Slow consumers are a problem (Source: Getty Images)
(2024) – cluster size (number of brokers) distribution Wide-range of cluster sizes → Kafka is horizontally scalable for many different workload sizes and types A few big clusters, many smaller → long-tailed power law → Zipf’s Law? 0 20 40 60 80 100 120 0 100 200 300 400 500 600 700 800 900 Size Cluster Cluster Size Distribution (largest to smallest)
that larger cluster sizes are possible (just like unseen Galaxies) Zipf’s law predicted that bigger galaxies would be detected in older parts of the universe and confirmed with James Webb telescope Visual size comparison of the six largest Local Group galaxies (Wikipedia) Image from NASA’s James Webb Space Telescope showing older and bigger galaxy clusters
that larger cluster sizes are possible (just like unseen Galaxies) Zipf’s law predicts that bigger Kafka clusters are also possible 0.1 1 10 100 1000 1 10 100 1000 Log rank Log size Kafka Clusters - Log size vs log rank Rank Predicted larger clusters Predicted larger clusters Larger
the size/TPS correlation Could use multi-variate regression to include more variables and increase the prediction accuracy But in reality, the top 10 clusters are very heterogenous and highly tuned for different workloads y = 71441x R² = 0.6853 0 2000000 4000000 6000000 8000000 10000000 12000000 0 10 20 30 40 50 60 70 80 90 TPS
at a time to minimize latency for long processing times Records that take 100ms to process introduce unacceptable latency when processed in batches Solution? Reduce batch size to 1 and process each record separately with many share group consumers Batch size 1 → 100ms latency 100ms x batch size
20 times slower • Micro-benchmarks and limited load testing • Is there a performance regression from 3.9.1 to 4.0? • Increase in linger.ms from 0 to 5ms (KIP-1030) • Now the default is for producers to batch records until the batch size (16,384 bytes) or 5ms is reached • What did we find? • No regression, but linger.ms increase definitely impacts latency for • Micro benchmarks with serialized producer→consumer pipelines • And normal benchmarks with lower throughput loads • Which the micro benchmarks and our latency benchmark exaggerated! • Watch what/how you are benchmarking! Kafka 3.9.1 to 4.0.0 had an increase in 90th percentile latency
higher impact of linger.ms=5 on latency For lower throughputs and more partitions 0 1 2 3 4 5 6 0 200000 400000 600000 800000 1000000 1200000 ms Msgs/s Throughput vs. Introduced latency per batch (ms) 1, 10 and 100 partitions latency per batch (1 partition) latency per batch (10 partitions) latency per batch (100 partitions)
storage just add more brokers (which also added CPU) Broker 6 CPU, RAM Disk Broker 1 CPU, RAM Disk Broker 2 CPU, RAM Disk Broker 5 CPU, RAM Disk Broker 3 CPU, RAM Disk Broker 4 CPU, RAM Disk Broker 7 CPU, RAM Disk Storage and CPU not independently scalable
Storage no need to add brokers Infinite cheap remote cloud storage (E.g. AWS S3) Local closed segments are copied to remote storage Remote segments can be retained for longer/cheaper For reads, if local segments have expired, then automatically retrieved from S3 Storage and CPU are independently scalable
• Producer write latency is not impacted by tiered storage • Copying to tiered storage is asynchronous • The “ugly” (watch out) • Tiered storage writes use more Kafka cluster resources • +10% CPU • Copying, and deletion to tiered storage • Can take time • Both are lazy, asynchronous • The bad! • Reading from tiered storage • Consumer latency and throughput impacted • Records from tiered storage don’t seem to be cached, so 2nd read is still slow • May need more partitions, consumers, and cluster resources (brokers) to increase tiered storage read throughput • Sensitive to segment size Kafka tiered storage in practice - performance Fake movie poster, ChatGPT
Kafka cluster with tiering enabled have to be bigger? Simple sizing calculator Given workloads in & out Computes IO and Network Load In MBytes/s Tiered storage with local SSDs
this? We can predict the minimum types/size/number of brokers Blue brokers have network bottleneck Green brokers have IO bottleneck A small increase in all cases BUT note that an existing cluster may have sufficient headroom for these already Lots of assumptions, and CPU not yet included – still need to benchmark in practice
Model Architecture – Performance Questions Producers Consumers Latency Throughput Maximum Capacity Lag Impact of configurations Steady-state vs. Dynamic Resources fixed/elastic Costs
Model Architecture – Performance Observations (Metrics) Producers Consumers Measured: Latency Throughput Resource Utilization Monitoring E.g. Monitoring with Prometheus Metrics Limited to measuring “What is” Performance of running application with actual workloads and resources
Model Architecture – Performance Benchmarking Producers Consumers Measured: Latency Throughput Lag Resource Utilization Maximum Capacity Impact of configurations Steady-state vs. Dynamic Resources fixed/elastic Costs Monitoring Optional monitoring Benchmark real or synthetic application with “What if” variations E.g. Higher loads, dynamic/elastic loads, different workload ratios, different versions or configurations, different resources, etc
Model Architecture – Performance Observations (Tracing) Producers Consumers Measured: Latency Throughput Lag Resource Utilization End to end distributed traces Topology and Spans (times) Monitoring Upgrade! Monitoring and tracing with OpenTelemetry Performance and tracing of running application with actual workloads and resources
Model Architecture – Performance Tracing Producers Consumers Measured: Latency Throughput Lag Resource Utilization End to end distributed traces Topology and Spans (times) Monitoring Currently, OpenTelemetry tracing only works with Kafka clients – the Kafka cluster is a black box ? Traces go through Kafka, but there’s no internal visibility (yet)
the OT instrumentation come from? 1. Manual code based 2. Automatic instrumentation – for specific languages, e.g. Java 3. Libraries 4. Agents In theory, Kafka brokers could eventually provide OpenTelemetry traces
Simple models need workloads, servers, service demands (times per workload per server) Workload 1 Server 1 Server 2 Server 3 Workload 2 Arrival Rate 1 Arrival Rate 2 Time 1, 1 Time 1, 2 Time 1, 3 Time 2, 1 Time 2, 2 Time 2, 3 1 server 3 servers 1 server Models can be solved using various techniques to predict latencies, throughput, utilization, etc E.g. Little’s Law, Queue Networks (Analytical, simulation), Custom tools TP RT Utilisation ?
= producers/consumers, servers are Net, CPU, IO Producers Network CPU IO Consumers Arrival Rate 1 Arrival Rate 2 Time 1, 1 Time 1, 2 Time 1, 3 Time 2, 1 Time 2, 2 Time 2, 3 1 server 3 servers 1 server TP RT Utilisation ? What open source modelling tool can we use and where do we get the service demands from?
Model Example CPU and Net resources for Producers and Consumers Net, CPU and IO resources for Kafka Brokers Need average service demands (times) per resource for each workload (Producers, Consumers) Including replication
to single components for Net, CPU and IO This simplifies the model Queue components can have multiple servers to model multiple brokers x cores E.g. 9 x 8 core brokers = 72 servers But how are average service demand times determined?
– parameterized with real cluster metrics 1M Msgs/s 5M Msgs/s 1,000 CPU Cores (Approx 60 brokers @ 16 cores/broker) Fan-out = 5 1,000 bytes/msg Service demands times = whatever gives the measured resource utilisation BUT There’s no way to distinguish times per workload Benchmarking each workload separately is the normal way this is done But time consuming
Dynatrace → Performance Predictions (2007-2016) Paul Brebner, Automatic Performance Modelling from Application Performance Management (APM) Data, 7th ACM/SPEC International Conference on Performance Engineering, ICPE 2016, Delft, The Netherlands, March 12-16, 2016 PurePath traces Latency and other predictions Automated model construction and parameterisation from distributed traces is the most accurate approach possible Traces must contain times per span per resource (CPU, IO, Network)
+ GUI – no longer maintained and Dynatrace specific Paul Brebner, Automatic Performance Modelling from Application Performance Management (APM) Data, 7th ACM/SPEC International Conference on Performance Engineering, ICPE 2016, Delft, The Netherlands, March 12-16, 2016 We modelled a diverse range of technologies including ESBs, distributed Middleware (E.g. BizTalk) and Kafka!
Open Source OT extension from www.retit.de Java Agent • Adds resource metrics to OT trace span attributes • io.retit.resource.demand.storage.bytes • io.retit.resource.demand.memory.bytes • io.retit.resource.demand.network.bytes • io.retit.resource.demand.cpu.ms Can we potentially do this with OpenTelemetry now?
trace processor • Takes input of (synthetically generated) Kafka client/broker traces/spans/resource demands • Computes workload metrics (type, arrival rate, average service demands per resource) • And outputs JMT model XML • For IO and Network • Converts demand in bytes to service demand times • Assuming IO/Network bandwidths • Likely underestimates times • Update August 2025 • Tested Kafka client OpenTelemetry instrumentation + RETIT resource plugin • Kafka client modelling is possible V0.1– mockup with client modelling “Lord of the Rings” Hobbiton movie set NZ The “houses” are just facades (Wikipedia)
-Dotel.traces.exporter=otlp -Dotel.resource.attributes=service.name=test_producer -Dio.retit.log.network.demand=true -Dotel.metrics.exporter=none -jar producer.jar Note exporter is now otlp not jaeger, and option to add network resources Kafka client OpenTelemetry + RETIT resource instrumentation example
paulbrebnertalks
shumei_ito
carta_engineering
kworkdev
sansantech
ishikawa_satoru
kentarofujii
y0hgi
terapyon
namiki64
takanorip
kairim0
akitoshiga
eileencodes
sarafernandez
katarinadahlin
mongodb
inesmontani
portentint
watany
tammyeverts
tmiket
mkilby
skipperchong
