Streaming More For Less With Apache Kafka Tiered Storage

Extended introduction to Apache Kafka tiered storage, including performance and cost modelling, with extra material covering recent "diskless" Kafka KIPs. Streaming track talk for Community over Code NA 2025.

Transcript

1. With Apache Kafka Tiered Storage Paul Brebner Instaclustr Technology Evangelist C/C NA 2025 Streaming Track © 2025 NetApp, Inc. All rights reserved. STREAMING MORE FOR LESS
2. + Kafka was simple + But streams are infinite! + Use cases for older data + Kafka tiered storage + In practice, performance and sizing + No local storage? KIP-1176 THE STORY © 2025 NetApp, Inc. All rights reserved
3. KAFKA WAS SIMPLE © 2025 NetApp, Inc. All rights reserved.
4. © 2025 NetApp, Inc. All rights reserved. Kafka Cluster = homogeneous brokers , with local CPU and storage Broker 6 CPU, RAM Disk Broker 1 CPU, RAM Disk Broker 2 CPU, RAM Disk Broker 5 CPU, RAM Disk Broker 3 CPU, RAM Disk Broker 4 CPU, RAM Disk Scalability Reliability Durability Availability
5. © 2025 NetApp, Inc. All rights reserved. Kafka Cluster → add brokers for more CPU and storage (both) Broker 6 CPU, RAM Disk Broker 1 CPU, RAM Disk Broker 2 CPU, RAM Disk Broker 5 CPU, RAM Disk Broker 3 CPU, RAM Disk Broker 4 CPU, RAM Disk Broker 7 CPU, RAM Disk From now we will simplify to single broker and topic In reality there are multiple brokers, topics, partitions, and replication!
6. © 2025 NetApp, Inc. All rights reserved. Kafka Was Simple! Records sent from Producers to Consumers End to end guaranteed delivery of records in close to real-time Kafka Producers Consumers Record 0 For subscribed consumers, records are delivered in close to real-time from producers, via Kafka, to consumers Guaranteed delivery, even if failures Produced, sent to Kafka
7. © 2025 NetApp, Inc. All rights reserved. Producers → Consumers End to end guaranteed delivery of records in close to real-time Kafka Producers Consumers Record 0 For subscribed consumers, records are delivered in close to real-time from producers, via Kafka, to consumers Guaranteed delivery, even if failures Via Kafka
8. © 2025 NetApp, Inc. All rights reserved. Producers → Consumers End to end guaranteed delivery of records in close to real-time Kafka Producers Consumers Record 0 For subscribed consumers, records are delivered in close to real-time from producers, via Kafka, to consumers Guaranteed delivery, even if failures Consumer polls Kafka Got it!
9. © 2025 NetApp, Inc. All rights reserved. Multiple consumers = multiple consumer groups – fan-out of records Kafka Producers Two consumer groups Multiple consumer groups receive each record How? Kafka does not delete records immediately so records can be consumed multiple times Record 0 Record 0 Record 0 Group 1 got it Group 2 got it
10. © 2025 NetApp, Inc. All rights reserved. Records are not deleted on delivery Kafka Producers Two consumer groups Records are initially retained in broker cache (active segment) Record 3 Record 2 Record 2 Record 0 Record 1 Record 2
11. © 2025 NetApp, Inc. All rights reserved. But, conceptually, streams are infinite The Milky Way seen from the Australia Outback (Paul Brebner)
12. © 2025 NetApp, Inc. All rights reserved. How do you fit infinite records in memory? Move to local disk? Kafka Producers Two consumer groups Records are moved to local disk when active segment is full (lots at once) Record 3 Record 1 Record 2 Record 2 Record 0 Record 2 Record 0 Record 1 Record 2
13. © 2025 NetApp, Inc. All rights reserved. Making room for new records in memory Kafka Producers Two consumer groups Record 4 Record 1 Record 3 Record 3 Record 0 Record 2 Record 3
14. © 2025 NetApp, Inc. All rights reserved. Time and space Kafka Producers Two consumer groups More time & records mean older records are available for consumption But more records take up more and more disk space When are older records consumed? Record 4 Record 1 Record 2 Record 0 Record 0 Record 2 Time Older records S p a c e Record 3
15. CONSUMER USE CASES © 2025 NetApp, Inc. All rights reserved. Backwards Time Travel!
16. © 2025 NetApp, Inc. All rights reserved. Why consume old records? Consumers decide which records to read - latest or from specific offset or time Pick a card, any card! (Adobe)
17. © 2025 NetApp, Inc. All rights reserved. Consumer use cases Real-time Slow consumers Buffered records Disconnected consumers New consumers Replaying Replication Migration
18. KAFKA TIERED STORAGE © 2025 NetApp, Inc. All rights reserved. Is Kafka tiered-storage more like a fountain or a dam? Latona Fountain (Source: Wikimedia) Hoover Dam (Source: Paul Brebner)
19. © 2025 NetApp, Inc. All rights reserved. What’s new? Tiered Storage = remote storage in the cloud Infinite cheap remote cloud storage What if you want Lots more storage? Cheaper storage? Then use cloud native storage How does this work?
20. © 2025 NetApp, Inc. All rights reserved. Tiered storage architecture – extra copies stored remotely Kafka Producers Record 4 Record 1 Record 0 Record 2 Record 3 Record 0 Record 1 Record 0 Record 2 3 - Records are also written to remote storage E.g. AWS S3 Multiple brokers and RF copies of records 1 - In cache (active segment) 2 - and then local storage
21. © 2025 NetApp, Inc. All rights reserved. Tiered storage architecture – but records retained for different times Kafka Producers Record 4 Record 2 Record 3 Record 0 Record 1 Record 0 Record 2 Records retained for a shorter time local.retention < remote.retention Remote records retained for a longer time remote.retention >> local.retention Older records (0 & 1) deleted from local storage But still available in remote storage
22. © 2025 NetApp, Inc. All rights reserved. Is Kafka tiered storage like a tiered fountain? Wikipedia
23. © 2025 NetApp, Inc. All rights reserved. Tiered Fountain? Local Storage – top tier Remote storage – bottom tier
24. © 2025 NetApp, Inc. All rights reserved. Tiered Fountain? Watch the duck - the oldest record – quack! Local Storage Remote storage
25. © 2025 NetApp, Inc. All rights reserved. Tiered Fountain? New Records Local Storage Remote storage
26. © 2025 NetApp, Inc. All rights reserved. Tiered Fountain? Local Storage Remote storage
27. © 2025 NetApp, Inc. All rights reserved. Tiered Fountain? Local Storage Remote storage
28. © 2025 NetApp, Inc. All rights reserved. Tiered Fountain? Local Storage Remote storage
29. © 2025 NetApp, Inc. All rights reserved. Tiered Fountain? Local Storage Remote storage Overflow!
30. © 2025 NetApp, Inc. All rights reserved. Tiered Fountain? Moved Moved Remote storage Local retention time up Overflow!
31. © 2025 NetApp, Inc. All rights reserved. Tiered Fountain? Local retention time up Remote storage
32. © 2025 NetApp, Inc. All rights reserved. Tiered Fountain? Remote storage
33. © 2025 NetApp, Inc. All rights reserved. Tiered Fountain? Remote storage
34. © 2025 NetApp, Inc. All rights reserved. Tiered Fountain? Remote storage
35. © 2025 NetApp, Inc. All rights reserved. Tiered Fountain? Deleted Deleted Remote storage time up
36. © 2025 NetApp, Inc. All rights reserved. Tiered Fountain? Deleted Deleted “Overflowing” of records from local to remote tier … Implies no time overlap of records in Local & Remote Storage – which there is
37. © 2025 NetApp, Inc. All rights reserved. This is not how it works Deleted Deleted
38. © 2025 NetApp, Inc. All rights reserved. More like a dam - water/records are quickly copied to remote storage The Hoover Dam wall with the outflow from the hydroelectric plant way down below (Source: Paul Brebner)
39. © 2025 NetApp, Inc. All rights reserved. More like a dam – local records/water spill (deleted) when time limit exceeded Hoover Dam Spillway (Source: Wikimedia) Spilling! (once, 1983) Public Domain
40. © 2025 NetApp, Inc. All rights reserved. Pumped hydro = upper and lower reservoir (Source: Adobe Stock) 1 Records are read from local storage, else 2 From remote storage (pump), else Error! 1 2
41. © 2025 NetApp, Inc. All rights reserved. Dam - upper and lower reservoir Upper reservoir Lower reservoir
42. © 2025 NetApp, Inc. All rights reserved. Dam - upper and lower reservoir Open segments are not copied
43. © 2025 NetApp, Inc. All rights reserved. Dam - upper and lower reservoir Eligible records (closed segments) are copied to remote storage
44. © 2025 NetApp, Inc. All rights reserved. Dam - upper and lower reservoir Reads are served from local storage if available
45. © 2025 NetApp, Inc. All rights reserved. Dam - upper and lower reservoir Reads are served from local storage if available
46. © 2025 NetApp, Inc. All rights reserved. Dam - upper and lower reservoir Reads are served from local storage if available
47. © 2025 NetApp, Inc. All rights reserved. Dam - upper and lower reservoir Reads are served from local storage if available
48. © 2025 NetApp, Inc. All rights reserved. Dam - upper and lower reservoir Local storage time up Deleted (spillway)
49. © 2025 NetApp, Inc. All rights reserved. Reads for older records are served from remote storage Pumped hydro
50. © 2025 NetApp, Inc. All rights reserved. Pumped hydro Reads for older records are served from remote storage
51. © 2025 NetApp, Inc. All rights reserved. Pumped hydro Reads for older records are served from remote storage
52. © 2025 NetApp, Inc. All rights reserved. Pumped hydro Reads for older records are served from remote storage
53. © 2025 NetApp, Inc. All rights reserved. Pumped hydro Remote storage time up Deleted
54. © 2025 NetApp, Inc. All rights reserved. Redrawing to show time overlap Time overlap Most/more records are in remote storage Reads are served from local storage Reads are served from remote storage
55. TIERED STORAGE IN PRACTICE © 2025 NetApp, Inc. All rights reserved.
56. © 2025 NetApp, Inc. All rights reserved. High level design – lots of changes Apache Kafka KIP-405
57. © 2025 NetApp, Inc. All rights reserved. • What do you need? • A Kafka cluster with tiered storage setup • E.g. Instaclustr managed kafka with AWS S3 tiered storage • BYO S3 bucket • Enable tiered storage for the cluster • Enable tiered storage for topics • With local.retention.ms (new config) << retention.ms (now remote retention time) • Write data • Wait a bit • Depending on segment size • Number of partitions etc • And then data will appear in S3 eventually Kafka tiered storage in practice – 1st steps
58. © 2025 NetApp, Inc. All rights reserved. • The good • Producer write latency is not impacted by tiered storage • Copying to tiered storage is asynchronous • he “ugly” (watch out) • Tiered storage writes use more Kafka cluster resources • +10% CPU • Copying, and deletion to tiered storage • Can take time • Both are lazy, asynchronous • The bad! • Reading from tiered storage • Consumer latency and throughput impacted • Records from tiered storage don’t seem to be cached, so 2nd read is still slow • May need more partitions, consumers, and cluster resources (brokers) to increase tiered storage read throughput • Sensitive to segment size Kafka tiered storage in practice - performance Fake movie poster, ChatGPT
59. © 2025 NetApp, Inc. All rights reserved. Consumer Throughput – 3.2M Records/s (local) vs. 1.2M Records/s (remote)
60. © 2025 NetApp, Inc. All rights reserved. Consumer latency – 5ms (local) vs. 20ms (remote)
61. © 2025 NetApp, Inc. All rights reserved. Does a Kafka cluster with tiering enabled have to be bigger? Simple sizing calculator Producer workload in Computes IO and Network Load In MBytes/s Tiered storage with local SSDs
62. © 2025 NetApp, Inc. All rights reserved. Screen shot JavaScript model Predicts IO, Network And TB/month $/month Storage costs https://github.com/instaclustr/code-samples/tree/main/Kafka/TieredStorage
63. © 2025 NetApp, Inc. All rights reserved. Overhead of tiered storage c.f. local storage? +11% for IO, +33% for Network 0 1,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000 9,000 Network MB/s IO MB/s Local vs Remote Storage Resource Usage Local Storage Remote Storage SSB local storage
64. © 2025 NetApp, Inc. All rights reserved. Include EBS local storage (networks SSDs) 0 5000 10000 15000 20000 25000 30000 35000 SSD local only SSD tiered storage 1/3 split SSD tiered storage x 10 remote workload EBS local only EBS tiered storage 1/3 split EBS tiered storage x 10 remote workload SSD vs. EBS IO (MB/s) Network (MB/s) Increasing the remote workload increases Network EBS configurations all have significantly increased Network
65. © 2025 NetApp, Inc. All rights reserved. What use is this? We can predict the minimum types/size/number of brokers Blue brokers have network bottleneck Green brokers have IO bottleneck A small increase in all cases BUT note that an existing cluster may have sufficient headroom for these already Lots of assumptions, and CPU not yet included – still need to benchmark in practice
66. © 2025 NetApp, Inc. All rights reserved. Finally, how much storage for local only (1 day) vs local/remote split (1 hour/1 day) 0 50 100 150 200 250 300 EBS local only - local retention 1 day EBS tiered storage - local retention 1 hour, remote 1 day EBS local vs tiered storage disk storage (TB) Local storage (TB) Remote storage (TB) 2.6x reduction
67. © 2025 NetApp, Inc. All rights reserved. And costs? 7x savings! That’s good 0 5000 10000 15000 20000 25000 EBS local only - local retention 1 day EBS tiered storage - local retention 1 hour, remote 1 day EBS local vs tiered storage $ per month EBS $/month S3 $/month
68. © 2025 NetApp, Inc. All rights reserved. What if we increase the remote retention to 1 month? 0 2000 4000 6000 8000 10000 EBS local only - local retention 1 month EBS tiered storage - local retention 1 hour, remote 1 month EBS local vs tiered storage disk storage (TB) - 1 month Local storage (TB) Remote storage (TB) 8,000 TB local vs. 2,500 TB remote
69. © 2025 NetApp, Inc. All rights reserved. What if we increase the remote retention to 1 month? 0 100000 200000 300000 400000 500000 600000 700000 EBS local only - local retention 1 month EBS tiered storage - local retention 1 hour, remote 1 month EBS local vs tiered storage $ per month EBS $/month S3 $/month And S3 is 10x cheaper
70. What’s new? Do we even need local storage? KIP—1176: Tiered Storage for Active Log Segment • https://cwiki.apache.org/confluence/display/KAFKA/KIP- 1176%3A+Tiered+Storage+for+Active+Log+Segment • Use fast object store in each AZ • Copy active segments to object store • Followers now • read from the object store (no inter-AZ cost) • not from the leader broker (inter-AZ cost) • Design goal • Reduce inter-AZ costs due to replication
71. Leader Follower 1 Follower 2 RF=3 AZ=3 Single AWS Region Write path P = Kafka producer AWS charges inter-AZ data transfer fees $0.01 per GB IN and OUT = $0.02/GB Replication cost = $0.04/GB BU If producer isn’t in Z with leader Then extra $0.02/GB! AZ 1 AZ 2 AZ 3 P Single AWS Region, 3 Availability Zones $0.02/GB $0.02/GB $0.02/GB Current inter-AZ costs
72. Leader Follower 1 RF=3 AZ=3 Single AWS Region Read path C = Kafka consumer Different AZ, inter-AZ cost Probability of different AZ to leader = 2/3 Inter-AZ for consumers is (2/3)*0.02 = $0.013/GB AZ 1 AZ 2 AZ 3 Single AWS Region, 3 Availability Zones P Follower 2 C $0.02/GB C $0.02/GB C Current inter-AZ costs Worst case
73. Leader Follower 1 RF=3 AZ=3 Single AWS Region Read path C = Kafka consumer But what if we use follower fetching (fetch from follower, KIP-392)? Reduces consumer inter-AZ cost to $0 C.f. worst case of $0.053/GB IN + $0.013/GB OUT =$0.066/GB (1:1 fan-out) AZ 1 AZ 2 AZ 3 Single AWS Region, 3 Availability Zones P Follower 2 C C C Current inter-AZ costs Follower fetching
74. Leader Follower 1 RF=3 AZ=3 Single AWS Region KIP-1176 Active segment copied to S3 And replicated from S3 Zero inter-AZ replication costs Some inter-AZ cost for producers in different AZ to leader Single AWS Region, 3 Availability Zones P Follower 2 S 3 KIP-1176 Given data replication in S3 → reduce RF???
75. 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 TOTAL Inter-AV $/GB 1 GB/s in, 1 GB/s out Default no follower fetching Default follower fetching KIP 1176 no follower fetching KIP 1176 follower fetching KIP 1176 follower fetching RF=2 AZ=3 KIP 1176 follower fetching RF=1 AZ=3 Reducing RF logically increases price to “1176 no follower price” RF=3 RF=2 RF=1 Same price
76. 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 Default no follower fetching Default follower fetching KIP 1176 no follower fetching KIP 1176 follower fetching KIP 1176 follower fetching RF=2 AZ=3 KIP 1176 follower fetching RF=1 AZ=3 Inter-AZ $/GB Sensitivty Analysis 1GB/s out vs 5GB/s out 1 GB Out 5 GB Out
77. 0 20000 40000 60000 80000 100000 120000 140000 160000 180000 200000 Total $/month $/month inter-AV costs Default no follower fetching Default follower fetching KIP 1176 no follower fetching KIP 1176 follower fetching
78. Observations • Not available yet • Inter-AZ cost is substantial for Kafka on AWS • KIP-1176 + follower fetching + cloud “cache” would offer substantial savings in inter-AZ costs • Up to 75% cost savings (or more with higher fan-out) • Note that you can’t reduce inter-AZ cost to zero due to inter-AZ producer costs • Cloud costs (e.g. S3 Express) • Contribute 25-33% of the cost + inter-AZ • Sensitivity to increase in fan-out • Minimal with follower fetching • Increased with reduced RF for KIP-1176 • Using follower fetching can increase latency slightly • As the replicas are slightly behind the leader • High watermark (offset up to which all replicas are in-sync) propagation delays which records are available
79. Leader KIP-1150 Diskless No local data storage No broker replication Relies on cloud object storage for replication Every broker is a “leader” Write and read from anywhere Data is accumulated across brokers Buffered and uploaded to object storage Latency >= 100ms to seconds Single AWS Region, 3 Availability Zones P S 3 Leader Leader P P Buffering KIP-1150 Alternative approach Major architectural changes
80. Leader KIP-1150 Diskless Reads from anywhere Get segments from object storage Single AWS Region, 3 Availability Zones S 3 Leader Leader C C C
81. Leader KIP-1150 Diskless Write and Reads No inter-AZ costs Read latency comparable to tiered storage 100-2000ms All segments are fetched from object storage Single AWS Region, 3 Availability Zones S 3 Leader Leader C C C P P P Not available yet
82. Observations • KIP-1150 diskless • Per topic • Can have both normal and diskless topics • Slow • 100-2000+ ms • Zero inter-AZ costs for AWS • Cloud storage costs? • Comparable to tiered storage probably • Depending on retention times • Complex architectural change • Brokers are now stateless and leaderless • Storage and replication pushed to object storage • N-tier architecture with compute and storage separated • Back to the 2000’s ;-)
83. © 2025 NetApp, Inc. All rights reserved. • Try it out! • NetApp Instaclustr Managed Kafka • 30 day free trial of developer sizes • Potential cost savings • Cost/performance tradeoffs • Watch out for: • Turning tiered storage on for existing topic with lots of local data • S3 buckets not automatically deleted if the cluster is deleted • New use cases with • more data, longer histories, different consumer behaviours, etc • Kafka as a database? Maybe! For some use cases • To scale remote workload may need to have • More partitions when writing data to remote storage • More consumers and resources • A bigger cluster for more CPU – easier, as now the cluster CPU and storage are independently scalable Kafka Tiered Storage Conclusions?
84. THANK YOU Blogs with more details Instaclustr.com/paul-brebner LinkedIn © 2025 NetApp, Inc. All rights reserved.