Apache Hive: Toward a Cloud Native Lakehouse

Slides used at OTFSG Osaka Meetup #2.

YouTube: https://youtu.be/xr3JQQJjjLE

Japanese Version: Apache Hive: そしてCloud Native Lakehouseへ

Transcript

2. About Me
3. Apache Hive: “Distributed Data Warehouse at Massive Scale” Spark / Flink / Trino / etc. Hive Metastore (Metadata Repository) RDBMS Backend Amazon S3 / HDFS / etc. HiveServer2 (SQL Gateway) YARN / Hadoop Hive on Tez Kubernetes Hive LLAP
4. Data Lake to Lakehouse On-prem Cloud Proprietary Open Data Warehouse Enterprise DWH Data Lake Big Data Cloud Data Warehouse Compute-Storage Separation Lakehouse Open Table Format
5. Data Lake to Lakehouse On-prem Cloud Proprietary Open Data Warehouse Enterprise DWH Data Lake Big Data Cloud Data Warehouse Compute-Storage Separation Lakehouse Open Table Format
6. Data Lake to Lakehouse On-prem Cloud Proprietary Open Data Warehouse Enterprise DWH Data Lake Big Data Cloud Data Warehouse Compute-Storage Separation Lakehouse Open Table Format
7. Agenda - Lakehouse-related Features - (1) Catalog - (2) Compute Engine - (3) Continuous Table Maintenance - Cloud-Native Readiness - (4) Kubernetes-Native Hive Disclaimer: This deck introduces some unreleased features
8. Lakehouse Catalog
9. Recap: General-Purpose Metadata Repository Hive Metastore (Stateless Java App) RDBMS Backend Files on a distributed storage Examples: HDFS / S3 / GCS / ADLS Files RDBMS / NoSQL Examples: PostgreSQL, MySQL, Apache HBase, Apache Kudu Databases Event stream Examples: Apache Kafka topics Event store Open Table Format Examples: Apache Iceberg, Apache Hudi, Delta Lake Lakehouse
10. Recap: De Facto Standard Thrift API Hive Metastore API Apache Spark Cloud Data Warehouse Trino - Thrift - Kerberos / LDAP - Any table format Apache Gravitino
11. Iceberg REST Catalog API: Another Emerging Standard Managed Services - Databricks Unity Catalog - Cloudera Iceberg REST Catalog - Snowflake Open Catalog - AWS Glue - Amazon S3 Tables - Google Cloud's Lakehouse - Microsoft Fabric OneLake - Dremio Open Catalog OSS - Unity Catalog - Apache Polaris - Apache Gravitino - Lakekeeper - Project Nessie
12. Iceberg REST Catalog API Backed by Hive Metastore “From Spec to Implementation: Iceberg REST Catalog with Hive Metastore” https://speakerdeck.com/okumin/from-spec-to-implementation-iceberg-rest-catalog-with-hive-metastore-english-version Trino REST Catalog Hive Metastore Iceberg REST API Hive Catalog API Server Iceberg REST API Hive Metastore Thrift API Trino REST Catalog REST API RDBMS RDBMS Thrift RPC REST API Embedded Mode Standalone Mode (>= 4.3)
13. HMS Thrift API Revisited: Generality Still Matters The most widely adopted general-purpose metadata API → Keep existing data useful, keep future options open Many Clients Real-World Data HMS Thrift API Apache Spark Trino Cloud DWH Federated Catalog Next-Generation Query Engine Hive Table CSV/TSV PostgreSQL Delta Lake Next-Generation Format
14. Summary: A Metastore Across Time Adapting to the Lakehouse Era Preserving the Past, Enabling the Future Iceberg REST API OAuth 2.0 Metrics Reporting Thrift API Existing Data Assets Future Innovation 🤝
15. Lakehouse Compute Engine
16. Principle: Multiple Engines, Single Copy Hive is one of many compute engines Batch → Hive, Spark Interactive → Trino, Hive LLAP Streaming → Flink, Spark ML Preprocessing → Spark, Polars ▦ The Same Iceberg Table
17. Examples of Iceberg Features Supported by Hive - Iceberg V1 & V2 - ✅ Schema Evolution - ✅ Partition Evolution - ✅ Hidden Partitioning - ✅ DELETE/UPDATE/MERGE - ✅ Time Travel - ✅ Branching / Tagging - Iceberg V3 - ✅ Row Lineage (for Parquet) - ✅ Deletion Vectors - ✅ Variant Type
18. Partition Aware Optimization “Hive: Bucket Map Join is now available for Iceberg users!” https://medium.com/@okumin/hive-bucket-map-join-is-now-available-for-iceberg-users-d3026a60a7bb
19. External Catalog Support “Expanding the Hive Ecosystem with Iceberg REST” https://medium.com/@dmitriy.fingerman/expanding-the-hive-ecosystem-with-iceberg-rest-f51da5019fe6 Hive on Tez S3 Tables Iceberg REST Catalog API Table Bucket Get Table Metadata Read Data Files
20. Summary: Hive = Scalable SQL Query Engine Distributed SQL Query Engine Choose the Right Engine for Each Workload Hive on Tez & LLAP Core Iceberg Features Advanced Optimizer Stream Processing Non-SQL Interfaces Machine Learning
21. Continuous Lakehouse Table Maintenance
22. Why Maintenance Matters Small Files Problem Accumulating Snapshots Physically Undeleted Data Poor Scan Performance Metadata Overhead GDPR Compliance Risk v1 v2 v… v99999 Data File Deletion Vector row1 row2 row3 row4 0 1 row5 1 0 0
23. Table Maintenance Feature Coverage - ✅ Minor Compaction: Merge Very Small Data Files - ✅ Major Compaction: Merge Small Data Files and Delete Files - ✅ Smart Compaction: Automatically Choose Minor or Major Compaction - ✅ Expire Snapshots - ✅ Remove Orphan Files
24. Built-In Continuous Table Maintenance in Hive A full Hive deployment can continuously maintain Iceberg tables Hive Metastore IcebergHouseKeeperService IcebergTableOptimizer HiveServer2 + Tez Enqueue Major / Minor Compaction Perform Compaction Expire Snapshots Iceberg Table w/ Old Snapshots Iceberg Table w/ Small Files HDFS, S3, etc.
25. Summary: Keeping Tables Healthy, Continuously Catalog + Query Engine → Integrated Maintenance Experience Hive Metastore - Understand Table State - Choose the Right Compaction - Expire Old Snapshots - Schedule Maintenance Tasks Hive Query Engine - Scalable Compaction Execution - Reliable Resource Pool
26. Kubernetes-Native Hive
27. Traditional Hive Deployment Architecture Apache Hive traditionally required Hadoop YARN, with no easy way to get started
28. Quickstart with Docker docker run \ --rm \ --env SERVICE_NAME=hiveserver2 \ --name hive \ apache/hive:4.2.0 HiveServer2 + Hive Metastore docker run \ --rm \ --name hms \ apache/hive:standalone-metastore-4.2.0 Hive Metastore w/ Iceberg REST Catalog API Official Docker images have been available since 2023 → Easy to try out
29. Object Storage Support Support any object store by adding the right JAR Hive Hadoop FileSystem API hdfs:// ofs:// s3a:// abfs:// HDFS Apache Ozone Amazon S3 Azure Data Lake Storage
30. Hive on Hadoop → Kubernetes YARN NodeManager Task Containers Hive LLAP Tez App Master OR Distributed Task Coordinator Launch YARN Containers and Schedule Tasks OR Schedule Tasks to Long-Running LLAP Daemons If Hive on Tez can run on Kubernetes, the whole Hive stack can run there too...
31. Hive on Hadoop → Kubernetes Tez Application Master Was the Final Barrier YARN NodeManager Task Containers Hive LLAP Tez App Master OR YARN Dependency: Strong YARN Dependency: Strong Not required if LLAP is available YARN Dependency: Weak (In theory, it could even run on EC2)
32. Hive on Kubernetes On the master branch, Hive on Tez now runs on Kubernetes in LLAP mode! Kubernetes Hive LLAP Tez App Master ZooKeeper Session Management Service Discovery Schedule Tez tasks to LLAP
33. Kubernetes Operator & Helm Chart # For Tez AMs and LLAP daemons helm install zookeeper bitnami/zookeeper ... # For Metastore backend helm install postgres bitnami/postgresql ... # Install Hive Metastore, HiveServer2, and LLAP daemons helm install hive oci://registry-1.docker.io/ayushtkn/hive-operator ... “Cloud-Native Apache Hive: A Deep Dive into the New Kubernetes Operator, Autoscaling, and Multi-Tenancy” https://medium.com/@ayushtkn/cloud-native-apache-hive-a-deep-dive-into-the-new-kubernetes-operator-autoscaling-and-6d3e4427121a
34. Apache Hive Is a Cloud Native Lakehouse Platform # For Tez AMs and LLAP daemons helm install zookeeper bitnami/zookeeper ... # For Metastore backend helm install postgres bitnami/postgresql ... # Install Hive Metastore, HiveServer2, and LLAP daemons helm install hive oci://registry-1.docker.io/ayushtkn/hive-operator ... Catalog + SQL Query Engine + Continuous Table Maintenance
35. Summary: Future Deployment Options Kubernetes Tez App Master ZooKeeper Hive LLAP Hadoop YARN Tez App Master Task Containers Hive LLAP Easy deployment with a Helm chart. Best for most users without an existing Hadoop cluster. Enterprise-grade scheduling with YARN. A proven choice for managing hundreds or thousands of tenants.
36. Apache Hive: Toward a Cloud Native Lakehouse - Rich built-in support for core Lakehouse capabilities - Docker and Kubernetes make Hive easier to try and deploy