Data Lakehouse Architecture Evolution & Future

This talk i will start with sharing differecne and respective concept, benefits and drawbacks database, data lake and data warehouse. And based on these introduction, can quickly assist attendees pickup basic knowledge. And next, introduce architecture and importance of lakehouse. Then, i will prepare demo code (ex. python & delta lake) to present lakehouse operating principle so that attendees can deep understand it and image it. Lastly, describe one of usecase in my past production experience to know how to work in realworld.

By this talk. In addition to lakehouse application, i also introduce additional knowledge and concept, includes batch, streaming and MLOps. So attendees can fully learn and understand these skill and current data enginnering related important architecture and trend. Becuase have a good data architecture and pipeline, can do many interested and innovative application from data and lightening the load of data team member, like data analyst, data scientist or machine learning engineer so that establish the great data cycle.

Transcript

1. TrendMicro - Staff Data Engineer Mars Su Data Lakehouse Architecture Evolution & Future
2. Represent Service About Lakehouse Data & Datastore 01 02 TABLE OF CONTENTS 03 Demo & UseCase 04 Future & Conclusion 05 03
3. Data & DataStore 01
4. Data Categories Structured • Clearly define columns • Clearly define value type • Fixed column by record • Like Table Semi-Structured Unstructured • Flexible column by record • Mixed value type • JSON, XML • Ambiguously definition • Image, Audio, Video 05
5. Data Storage Data Warehouse Data Lake • Delta Lake • Apache IceBerg • Apache Hudi Database • RDB • NoSQL ◦ Key-Value ◦ Timeseries ◦ Column-family ◦ Document ◦ Graph • Clickhouse • AWS Redshift • Snowflake • HDFS • AWS S3 • GCP GCS • Azure Blob • Databricks FS Lakehouse 06
6. Data Storage 07
7. About Lakehouse 02
8. Lakehouse Unified Storage Using one storage to break Data Island, Reduce Cose Scaled traction data to ensure consistency ACID SImple Data Model Easy implement & Incremental ETL Data Government Control data and perform audits Data sharing & Lineage Maintain version history for dataset and track to data provenance. Adapt to changing business Easy to modify data schema or create table over time. 09
9. Represent Service 03
10. Top3 Lakehouse 11
11. Top3 Lakehouse • ACID Transaction • Data Version (Time travel) • Schema Evolution • Near-real-time Streaming/Batch 12
12. Delta Lake - metadata 13
13. Delta Lake - transaction log 14
14. Apache IceBerg - Partition Evolution Hidden Partition 15
15. Kappa Architecture Data Source Real-Time Layer Storage Real-Time App Batch App MS min/hour 16
16. Lambda Architecture Data Source Real-Time Layer Speed Real-time Query Mixed Query Batch Layer Batch Batch Query 17
17. Delta Architecture(Delta Lake) Data Source Real-Time Layer Real-time Query Mixed Query Batch Layer Batch Query 18
18. Lakehouse - 3 Zone Architecture Bronze Silver Golden Raw Ingestion Filtered, Cleaned Augmented Business-Level Aggregates Streaming Analytics AI & Report Apache Kafka AWS Kinesis GCP PubSub Azure Eventhub CSV, JSON… etc Lakehouse (Delta, Iceberg, Hudi) 19
19. Demo & UseCase 04
20. CREDITS: This presentation template was created by Slidesgo, including icons by Flaticon, and infographics & images by Freepik Demo
21. Delta Lake AWS S3 Real UseCase - 3 Zone Architecture Bronze Silver Golden Raw Ingestion Filtered, Cleaned Augmented Business-Level Aggregates Structured Streaming BU 1 BU 1 BU N 21 Schema Registry Great Expectation Great Expectation
22. Future & Conclusion 05
23. Lakehouse OverView STORAGE FORMATS Structured, Semi-Structured, Unstructured SECURITY & GOVERNMENT Data PREP SQL & BI AI / ML TIMESERIES GRAPH STEAMING … Analytics - SQL & BI SECURITY & GOVERNMENT TABLE METASTORE STORAGE FORMAT 23
24. Summary Data Central + Mart Lineage & Security Schema version to fit application demand, like analytics or ML. ETL + ELT Hybrid data pipeline for Flexibility and Scalability Centralized and Decentralized Manage overall data & control permission for security Iteratively application 24
25. CREDITS: This presentation template was created by Slidesgo, including icons by Flaticon, and infographics & images by Freepik THANKS