Apache Paimon Demystified

Transcript

1. Apache Paimon Demysti f ied Zhiyan Xiao (LY Corp), 2025/3/26 A Realtime Lakehouse Table Format
2. Self Introduction 肖 志彦 / Zhiyan Xiao 🏠 中国重慶 / Chongqing, China 8D Magic City, Hot Pot 👨🎓 香 港中 文 大 学 / The Chinese University of Hong Kong Math, Information Engineering 👨💻 Software Engineer @ LY Corporation Streaming Data Pipeline, Spark, Iceberg ✨ Improving big data systems with Rust Hadoop Client, Hive Client, SASL / Kerberos, InnoFile, InnoTable
3. Purpose of Today To understand basic ideas of Paimon
4. Afterwards, we can • Deep dive internals of Paimon • Consider how to utilize Paimon in real business
5. Agenda What is Paimon? How does Paimon work? When to use Paimon?
6. What is Paimon?
7. A Lakehouse table format enabling realtime read and write operations
8. First, how to read and write big data?
9. Usage of File System & File Format
10. Issues in directly handling data f iles • No explicit schema • Schema evolution is not well de f ined • Full data scanning is needed when • reading records with particular f ilters • deleting or updating certain records • Missing advanced features like ACID, Time Travel, etc.
11. Bene f its of Table Format • De f ine schema explicitly • Support schema evolution • Accelerate querying by skipping unnecessary data f iles • Support other advanced features like ACID, Time Travel, etc.
12. Popular Table Formats • Hive • Lakehouse Table Format • Iceberg • Delta Lake • Hudi • Realtime Lakehouse (Streamhouse) Table Format • Paimon
13. Skipping unnecessary data f iles would signi f icantly accelerate data queries
14. Hive Table Format • Skip unnecessary data f iles with partition and bucket • Partition • Mapping “partition columns” to “data directories” • /table/year=2025/month=3 • Bucket • Mapping “bucket columns” to “data f iles with corresponding hash” • /table/year=2025/month=3/part-00000
15. Issues of Hive Table Format • Dif f icult for Hive Metastore to support huge amount of partitions • Query optimization is limited to partition and bucket level • Missing advanced features like ACID, full schema evolution, etc.
16. Lakehouse Table Format Take Iceberg as example • Skip unnecessary data f iles with metadata • metadata.json • manifest-list.avro (contains stats to skip manifest f iles) • manifest.avro (contains stats to skip data f iles) • data- f iles.parquet • Support advanced features like ACID, full schema evolution, etc.
17. Issues of Lakehouse Table Format For high-throughput streaming write, • Iceberg • Commits are atomic and inef f icient for frequent writes • Delta Lake, Hudi • Update/delete relies on Merge-on-Read with expensive compaction
18. Paimon Table Format • Skip unnecessary data f iles with metadata (manifest, index f ile) • Support high-throughput streaming write and low latency read • with LSM Tree (log-structured merge-tree) storage model
19. Above is the role Paimon plays as a Realtime Lakehouse Table Format
20. How does Paimon work?
21. Paimon Table Types • Table with Primary Key • Table without Primary Key • View • Format Table • Object Table • Materialized Table
22. Paimon File Layouts
23. Insert One Record INSERT INTO T VALUES (1, 10001, 'varchar00001', '20230501');
24. Insert Multiple Records In di ff erent partitions
25. Delete Records DELETE FROM T WHERE dt >= '20230503';
26. Compact Table CALL sys.compact('T');
27. Expire Snapshot
28. Flink Stream Write Before Flink checkpoint
29. Flink Stream Write During Flink checkpoint
30. Change Log Producer Take “lookup change log” as example
31. When to use Paimon?
32. Usage of Paimon • High-throughput streaming write • Streaming read from change log • Uni f ied streaming-batch pipeline • (Realtime Lakehouse) Streamhouse Architecture
33. Potential Future Actions • Comprehensively compare behaviors of Iceberg and Paimon • Create PoC with Paimon to demonstrate its bene f its • Smoothly upgrade from Lakehouse to Streamhouse
34. References • https://paimon.apache.org/ • https://github.com/facebook/rocksdb/wiki/Universal-Compaction
35. Fin