Streaming Lakehouse to Agentic AI - Kafka, Flink & Iceberg meets MCP

Slides from the Singapore Apache Kafka meetup talk on September 18th 2025. Presents the concept of using MCP (Model Context Protocol) in a Data Lakehouse setup to enable natural language based conversational Data Analysis & Exploration in a Lakehouse using AI tools and LLM with MCP

Transcript

1. Streaming Lakehouse to Agentic AI : Kafka, Flink & Iceberg meets MCP
2. HELLO !! I’m Zabeer Farook Technical Architect, Credit Agricole CIB - Passionate about Data Architecture including Stream data processing & Event Driven Architecture as well as Cloud & DevOps. - Love travelling & exploring places
3. AGENDA 04 05 06 MCP’ing the Lakehouse Industry Use cases & Security best practices Demo Introduction to MCP (Model Context Protocol) 03 Streaming Lakehouse Recap 07 Closing & QA 01 02 Context & Background
4. Context & Background • Combines best from Data Warehouse & Data Lakes • Performance • Provides Transactional Guarantees • Cost Efficient & Scalable by leveraging Object Storage like S3 • Uses Opentable formats like Iceberg • Interoperability with different engines like Spark, Flink, Trino • Open Architecture without Vendor lock-in MCP (Model Context Protocol) brings Agentic AI capabilities to the Lakehouse and enables conversational data exploration & analysis Modern Data Platforms are increasingly being built with Lakehouse architecture with Open Table Formats
5. Streaming Lakehouse Recap Ingest Store & Process Consume AI/ML & Data Science Batch & Real time Analytics Reports & BI Data Lake Raw Data Cleansed Data Stream Batch Structured, Semi Structured & Unstructured Data Metadata Layer with Data Governance, Indexing and Data Management • Real time ingestion layer to ingest data from real time sources like Kafka • Storage layer with Object storage like Minio, AWS S3, GCS. Data itself will be stored in formats like parquet, avro or orc. • Metadata layer is made of open table formats like Iceberg with features like ACID compliance & Time travel. • Raw data can be further processed in batch or streaming mode with engines like Flink • Serving layer with query engines like Trino providing query and API capabilities to the consumption layer for AI/ML, Reporting, Analytics & Visualization Streaming Lakehouse API Layer Serve REST CATALOG
6. Streaming Lakehouse Recap Slide Deck of Streaming Lakehouse Talk Recording of Streaming Lakehouse Talk
7. Introduction to MCP - Evolution of LLM’s Simple LLM’s (pure text generation with limited context window) RAG Systems (Realtime knowledge retrieval from external sources) Tool calling Agents (Function calling & external tool integration) Agentic RAG (Intelligent retrieval decisions & query planning) Multi Agent Systems (Agent Collaboration & distributed problem solving)
8. Introduction to MCP - Why & What? MCP (Model Context Protocol) is an open, standardized interface that enables LLMs to • Interact seamlessly and securely to external systems, API’s and data sources • Have Agentic AI capabilities which makes them an intelligent operator who can take action based on natural language inputs Communication Protocol -> JSON-RPC Transport Protocol -> Stdio / SSE / Streamable HTTP Integrating different tools with multiple AI systems poses complex integration challenges and frictions without a standard communication & data exchange protocol between AI Systems & external tools. Imagine the situation of not having a USB - C or Type - C port in your computer or not having a Universal Travel Adapter while travelling
9. Introduction to MCP - Flow Illustration
10. MCP’ing the Lakehouse • Exposing the Lakehouse tools (Kafka, Flink, Iceberg etc.) through MCP • Natural language based operations and explorations in the lakehouse Sample Natural Language Data Exploration Prompts: • What are the different Flink jobs running in my Dev Cluster? • What are the top 10 products ordered today by customers? • What are the products low in stock?
11. Demo
12. Demo Code https://github.com/Zabi82/DataLakeHouseMCPJava Java Python https://github.com/Zabi82/DataLakeHouseMCP
13. Industry Use Cases • Data Platform Democratization ◦ Self Service Analytics without SQL knowledge for business users ◦ Automated data lineage • Intelligent Lakehouse Operations Management ◦ Automated Compaction, Snapshot Expiration, Orphan file cleanup ◦ Data quality monitoring and automated remediation • Automated Monitoring ◦ Real-time pipeline monitoring (Kafka lag, throughput) ◦ Ops Assistant for Kafka/Flink clusters • Performance & Cost Optimization ◦ Query Performance Analysis & Recommendations ◦ Resource Utilization & Right Sizing Potential Lakehouse / Data platform specific use cases with MCP and Agentic AI are....
14. Security & Guardrails • Strong auth & authz - not every user should be able to perform admin operations • Guardrails are critical - Scope the tools carefully • Security - Sanitize and Validate the inputs to prevent injection attacks • Privacy - Expose only necessary data fields & mask sensitive data • Cost Control - Avoid over querying LLM or large tables
15. Closing Thoughts “The future of data platforms is open, streaming and agentic - built on lakehouses that think and act with us“ MCP is powerful, but still evolving, adopt and adjust as it evolves
16. Q&A