Apache Fluss 101 — Introduction & Deep Dive into Real-Time Data Infrastructure for AI & Analytics

Transcript

6. Hot/Cold Tiered Storage Approach What is Streamhouse? A Streaming Lakehouse (or "Streamhouse") represents a unified architecture that merges streaming and batch systems under a single model, rather than stitching them together as separate components. ➔ Single Read/ Write • Automatic lifecycle management • Unified table abstraction • Sub-second freshness Key Streamhouse Concepts : Tier Role Storage Latency Hot Fluss Last 1 hour- 1 day of data NVMe/SSD Sub-second (milliseconds) Cold Iceberg) Hours to years of history S3/ object storage Minute-level
7. Tables, Not Topics - Native Iceberg Support 0.8 ➔ Zero ETL works through a stateless Flink job called the Tiering Service ➔ Built-in Compaction ➔ Log & Primary Key Table ➔ Simple DDL ➔ Union Read Mechanism
8. Debunking Union Read The Example: Consider a log table tracking all order events 1. Offset Coordination: Fluss CoordinatorServer provides Snapshot xx as the Iceberg boundary. At this snapshot, Iceberg contains events up to offset 2xx. 2. Data Distribution: 1. Iceberg contains: {Order#1, Order#2} (historical events) 2. Fluss contains: {Order#3,Order#4,Order#5} (new events after snapshot) 3. Union Read in Action: The query engine performs a union read: 1. Reads {Order#1, Order#2} from Iceberg (up to offset boundary) 2. Reads {Order#3, Order#4, Order#5} from Fluss (from offset boundary onwards) 4. Simple Concatenation: Results are concatenated in offset order (no deduplication needed): {(Giannis, 8), (Rhea, 14), (Yuxia, 24),(Mehul, 28), (Jark, 30)}
9. Traditional Architecture vs Streamhouse Architecture Replace Complexity with Fluss
10. Where Apache Fluss Meets Business Real Time Analytics/Dashboards Real Time context for agents ETL & Anomaly Detection Personalization & Recommendation
12. Thank You! For Joining Repository Workspace Apache Fluss Page Community Events ⭐
13. Union Read Primary Key Table Via Deletion Vector
14. In-Place Lakehouse Integration