Incremental Change Processing with Apache Flink and Iceberg

Apache Iceberg is a robust foundation for large-scale data lakehouses, yet its current support for change data capture (CDC) is limited, making updates and deletes challenging for incremental processing. While the unreleased Iceberg V3 will introduce native CDC support, many production environments still run on V2 and require pragmatic workarounds.

In this talk, we’ll explore how to implement incremental change processing over Iceberg V2 using Apache Flink, by writing change data streams as append tables and reading append tables as change streams. We’ll also walk through trade-offs between append and upsert modes, and how to choose the right one for your workload.

Finally, we’ll preview what Iceberg V3 brings to the table with native CDC support, and how it shifts the design landscape for real-time pipelines. If you're building data pipelines on Iceberg, this session will provide you with pragmatic strategies to overcome existing limitations and scale efficiently while keeping the infrastructure simple.

Transcript

1. Incremental Change Processing with Apache Flink and Iceberg Sharon Xie Founding Engineer & Head of Product @Decodable 2025-07-23
2. Today we are going to learn
3. Change Data Capture (CDC) -> Change Stream The observer for your database.
4. Incremental Processing with Iceberg & Flink
5. How Does Flink Process Change Stream?
6. Processing Change Stream w/ Flink
7. Processing Change Stream w/ Flink (cont)
8. Processing Change Stream w/ Flink (cont) Flink has 4 record types: insert, update_before, update_after, delete
9. Processing Change Stream w/ Flink (cont)
10. Challenges with Iceberg V2
11. What is needed? • Flink can read the changes to a table as a stream • Flink can write change streams to a table
12. ❌ Iceberg CDC Streaming Reads
13. ⚠ CDC Streaming Writes Iceberg has upsert table. But, Flink produces high volume retractions (deletes and inserts) • Equality deletes ◦ Optimized for write performance ◦ But slow for query time • Lots of delete files ◦ Small files problem ◦ Slow for query time
14. Change Processing Doesn’t Really Work for Iceberg V2
15. The Trick • Store change events in append tables • Merge/Materialize changes in append tables on read
16. How Does It Work?
17. Change Stream -> Append Stream
18. Append Stream -> Append Table for Storing
19. What’s in the Append Table?
20. Querying the Iceberg Table Find the latest value per key
21. What about the query performance? • Records to merge grow over time • Can we do “compaction” similar to kafka compacted topics?
22. Streaming Job Writing to the Current Partition
23. Batch Job Merges Keys of Old Partition
24. Delete the Merged Partition
25. Putting it all together
26. Iceberg V3 - Row Lineage
27. Row Lineage • Tracks changes to individual rows as they are updated, deleted, or inserted. ◦ Foundation for incremental change processing • Not tracked for rows updated via equality deletes
28. What does this mean for Flink? • Writer side ◦ Flink uses equality deletes ◦ Can’t track row lineage information • Reader side ◦ Needs to derive a stream of changes from row lineage information ◦ No development yet • TL,DR: framework is there, but solution is not ready
29. 🎁 Wrapping Up
30. 1. Incremental change processing with Iceberg V2 requires workarounds: a. Write change streams to append-only tables b. Read append tables as change streams c. Schedule batch merges to maintain performance 2. Iceberg V3’s row lineage can make change processing easier a. But still needs more development in processing engines Key Takeaways
31. Thank You 🙏 Q&A Sharon Xie