Introducing Data Lakehouse: Apache Iceberg

One of the topics on DevFest Cloud Bangkok 2023

Transcript

1. Introducing Data Lakehouse: Apache Iceberg Cloud Burasakorn Sabyeying (Mils) Data Engineer, CJ Express. Women Techmakers Ambassador, GDG Cloud Bangkok
2. เข้าใจ concept ของ Data Lakehouse Goal: (ผาน Iceberg)
3. What is Data Lakehouse? Cloud
4. 3 Generation of analytic platform Cr. Databricks Data warehouse - Database for analytics - ACID guarantees - Support only structured data
5. 3 Generation of analytic platform Cr. Databricks Data Lake • Store CSV, JSON, images, video, txt • Store data in Open Formats e.g. parquet, avro • Lower cost Lack of ACID Guarantees, metadata management, indexing, partitioning
6. 3 Generation of analytic platform Cr. Databricks Data Lakehouse • ACID guarantees • Lower cost • Fewer copies mean less storage costs • Undo mistakes by using Snapshot isolation • metadata management, indexing, partitioning
7. How to build Data Lakehouse Paid Services Open Source
8. What is Apache Iceberg ? Cloud
9. “Apache Iceberg is an open table format for huge analytic datasets” way to organize a dataset’s files to present them as a single “table”.
10. Iceberg table
11. Iceberg table catalog data metadata parquet avro
12. 1. making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time and works just like a SQL table. 2. Upsert data 3. Schema Evolution 4. Partition evolution 5. Time Travel and Rollback What can do ?
13. Create Apache Iceberg via Spark 1. PySpark Dataframe 2. Spark SQL
14. 1. Via PySpark Dataframe df.writeTo(tableName).create()
15. df.writeTo(tableName).overwrite() or df.writeTo(tableName).append()
16. 2. Via Spark SQL
17. Checkpoints ✅ making it possible for engines like Spark to work just like a SQL table. 💜 Upsert data 💜 Schema Evolution 💜 Partition evolution 💜 Time Travel and Rollback
18. Upsert Data (Insert + Update) Current data (target) new data (source) demo.nyc.new_data demo.nyc.taxis Demo.nyc.taxis (now)
19. Schema Evolution • Change column type • Add new column • Drop existing column • Reorder column • Rename existing column • Add column comment = No need to create and write to new table But you can do it in-place !
20. Rename Column
21. Add Column df = spark.table("demo_ja.nyc.taxis").show() Add ‘fare_per_distance’
22. Delete Column df = spark.table("demo.nyc.taxis").show() Delete `store_and_fwd_flag`
23. allows you to update your partitions for new data without rewriting data. Partition Evolution
24. Checkpoints ✅ making it possible for engines like Spark to work just like a SQL table. ✅ Upsert data ✅ Schema Evolution ✅ Partition evolution 💜 Time Travel and Rollback
25. Iceberg table catalog data metadata parquet avro
26. Iceberg table
27. v2.metadata.json
28. See snapshot id and manifest list
29. Rollback to specific snapshot before After
30. Checkpoints ✅ making it possible for engines like Spark to work just like a SQL table. ✅ Upsert data ✅ Schema Evolution ✅ Partition evolution ✅ Time Travel and Rollback
31. “Apache Iceberg is an open table format for huge analytic datasets” way to organize a dataset’s files to present them as a single “table”.
32. Apache Iceberg in BigQuery
33. Where to learn more - https://iceberg.apache.org/spark-quickstart/ - https://www.dremio.com/resources/guides/apache-iceberg/
34. Thank you ! Cloud https://bit.ly/burasakorn-mils