Apache Spark Declarative Pipelines (SDP) Conference Talk

OSS Apache SDP and Databricks Lakeflow on Free Edition

Transcript

1. Apache Spark Declarative Pipelines (SDP) In Action Avionics Demo Frank Munz, Databricks 2026
2. Frank Munz, Databricks
3. About me • I am Frank Munz • Principal TM Engineer @ Databricks: Data and AI • My past: large scale data & compute • Based in • Built up AWS Tech Evangelism in Central Europe • SW architect, data scientist, published author etc. ⛰🥨🍻󰎲
4. Apache Spark ™ ©2022 Databricks Inc. — All rights reserved 4
5. Spark SQL and Dataframe API # PYTHON: read multiple JSON files with multiple datasets per file cust = spark.read.json('/data/customers/') cust.filter(cust['car_make']== "Audi").show() cust.createOrReplaceTempView("customers") %sql select * from customers where car_make = "Audi" -- PYTHON: -- spark.sql('select * from customers where car_make ="Audi"').show() ©2022 Databricks Inc. — All rights reserved
6. Automatic Parallelization and Scaling ©2022 Databricks Inc. — All rights reserved
7. Reading from Files vs Reading from Streams # read file(s) cust = spark.read.format("json").path('/data/customers/') # read from messaging platform sales = spark.readStream.format("kafka|kinesis|socket").load() ©2022 Databricks Inc. — All rights reserved
8. Data Engineering: The destination is simple. The journey is hard Data Sources Data Consumers Message Queues Real-time Operations Enterprise Apps Databases Unstructured Data Cloud Storage Data Lake ??? Agents Apps Dashboards Analytics
9. >>10y of Simplifying Apache Spark™ ©2022 Databricks Inc. — All rights reserved 9
10. What is declarative? ©2022 Databricks Inc. — All rights reserved 10
11. Declarative Declare a plan Execute it Query orders table 11
12. What is … a data pipeline ? ©2022 Databricks Inc. — All rights reserved 12
13. data pipeline Something that produces and updates a set of datasets
14. Query Query Dataset Query Query Dataset Dataset Query Dataset Query Dataset Dataset
15. Why make pipelines declarative? ©2022 Databricks Inc. — All rights reserved 15
16. Developing and operating data pipelines is hard
17. Surprisingly hard: multiple queries/tables Query customers Query Query raw_orders fact_orders
18. Sequential Execution could run at the same time
19. Multithreading
20. Things get very complicated quickly Query table1 Query Query table3 Query table2 Query table4 table5
21. What if there is an error? ©2022 Databricks Inc. — All rights reserved 21
22. Analysis Error: Catchable before execution ⚠
23. automatic retries at runtime
24. Manual Retry (command-line args)
25. Add an Orchestrator / Workflow System ? Code DAG object
26. What if we want to run streams continuously? Run every 15 minutes Batch Query customers table Streaming Query raw_orders table Run continuously Streaming Query fact_orders table Run continuously
28. Problem: first pipeline run • • • • raw_orders streaming query starts async fact_orders streaming query starts fact_orders streaming query tries to read from raw_orders table raw_orders table doesn’t exist yet -> ⚠error⚠ Streaming Query raw_orders table Streaming Query fact_orders table
29. What if we make the whole pipeline declarative? Build plan -> execute it ©2022 Databricks Inc. — All rights reserved 29
30. Spark Declarative Pipelines (SDP) ©2022 Databricks Inc. — All rights reserved 30
31. Streaming Table (ST) What is it? use it for… • Built for incremental processing • Source row handled exactly-once (remembers internal progress) • Append-only sources /event stream Real-time, incremental data ingestion • Low latency streaming transformations • Bronze/silver layers •
32. Materialized View (MV) What is it? use it for… • Precomputed query result (joins, aggregations, anything SQL can express) • Servering layer: gold, BI, dashboards, app views • Guaranteed correct at every refresh • Incremental when possible, full recompute when needed
33. Declarative Pipeline with STs and MVs Putting things together Declarative Pipelines customers_raw Streaming Table customers_clean Streaming Table customer_orders Materialized View orders_raw Streaming Table orders_clean Streaming Table Bronze Layer Silver Layer Gold Layer
34. SQL API
35. Python API
36. Flows (not needed for this talk/demo) What is it? use it for… • Flows are implicit for ST and MV • AUTO CDC • A flow is the processing unit: moves data to table -> how • Table is the target -> what • Flows support batch and streaming • ST with several flows "Stream 2 Kafka topics into a ST" • Syntax: CREATE FLOW F AS [AUTO CDC | … more to come]
37. ⚙ Spark Declarative Pipelines • Transformations are Python or SQL files • Initialize via • Execute with spark‑pipelines init --name orders‑pl spark‑pipelines [dry-]run • SDP automatically handles pre-validation, dependencies, retries & parallelization. • Triggered/batch and continuous/streaming mode
38. It's demo time! ©2022 Databricks Inc. — All rights reserved 38
39. Build this SDP in your next coffee break OSS or Lakeflow (try both!) ©2022 Databricks Inc. — All rights reserved
40. Getting Started with Spark Declarative Pipelines: OSS + Lakeflow OSS and Lakeflow Tutorials (realtime flight data) Create a Databricks forever Free Edition account, and add the avionics SDP demo to your portfolio on LinkedIn ©2022 Databricks Inc. — All rights reserved 40
41. Custom Python Datasource for OpenSky API ©2022 Databricks Inc. — All rights reserved 41
42. 🚀 Python Data Source API spark.readstream.format("opensky") • Use familiar spark.read.format() and write.format() • OSS • simple pip install • Supports batch + streaming
43. Lakeflow ©2022 Databricks Inc. — All rights reserved 43
44. Lakeflow Spark Declarative Pipelines Efficiently clean, transform and join data Simplified pipeline development Build batch and streaming pipelines with a declarative approach Reliable production infrastructure Automated pipeline configuration with reduces maintenance burden Built on an open standard Fully compatible with the open source Spark Declarative Pipelines ©2025 Databricks Inc. — All rights reserved
45. Genie Code An autonomous AI agent for Data Science, Engineering and Analytics • Natural language prompt to create, debug, explain and document • Creates, manages and debugs data pipelines • Genie Code Agent skills for Claude Code • Can be extended with Skills.md and MCP servers
46. Spark & SDP Real-time Mode ©2022 Databricks Inc. — All rights reserved 46
47. SDP with Real-time Mode (RTM) • 5 ms latency • Built on concurrent stages, streaming shuffle, and continuous data flow • Same Spark platform -> makes external systems like Flink redundant • Building blocks: ◦ a source (message bus, rate) ◦ sinks (message bus, Lakebase, console) ◦ transformation use @dp.update_flow(...)
48. Air Traffic Control System Spark Streaming, Rea-time mode and Databrick Apps (REC)
49. Convert Messy Sales Data to AI insights databricks.com/demos Use Lakeflow Connect, Jobs and SDP to create a marketing solution with AI for a global food corporation. Video Walkthrough with GitHub Repo + DABs 49
50. Get to know Genie Code Genie Code data engineering video demo: Use Genie Code to create a complete SDP pipeline from a prompt. With Auto Loader, JSON ingestion, and medallion architecture. Genie Code Step by Step Guide 50
51. Zerobus Ingest Demo Zerobus Ingest demo: ingest streaming IoT events without Kafka from all over the world