Streaming Data on the Lakehouse (Devoxx Conference ATH)

©2022 Databricks Inc. — All rights reserved
Streaming Data
on the Lakehouse
- a talk for everyone who ❤ data
Devoxx Greece 2023
1
Frank Munz, Ph.D.
May2023

View Slide

About me
• Principal @Databricks. Data, Analytics and AI products
• All things large scale data & compute
•
Based in Munich,
🍻 ⛰ 🥨 󰎲
• Twitter: @frankmunz
• Formerly AWS Tech Evangelist, SW architect, data scientist,
published author etc.

View Slide

©2022 Databricks Inc. — All rights reserved 3
What do you love most about
Apache Spark?

View Slide

Spark SQL and Dataframe API
# PYTHON: read multiple JSON files with multiple datasets per file
cust = spark.read.json('/data/customers/')
# cust.createOrReplaceTempView("customers")
cust.filter(cust['car_make']== "Audi").show()
%sql
select * from customers where car_make = "Audi"
-- PYTHON:
-- spark.sql('select * from customers where car_make ="Audi"').show()

View Slide

Automatic Parallelization and Scale

View Slide

Reading from Files vs Reading from Streams
# read file(s)
cust = spark.read.format("json").path('/data/customers/')
# read from messaging platform
sales = spark.readStream.format("kafka|kinesis|socket").load()

View Slide

Stream Processing is
continuous and unbounded
Stream Processing
7
Traditional Processing is
one-off and bounded 1
Data Source
2
Processing
Data Source Processing

View Slide

Technical Advantages
A more intuitive way of capturing and processing
continuous and unbounded data
Lower latency for time sensitive applications and use cases
Better fault-tolerance through checkpointing
Better compute utilization and scalability through
continuous and incremental processing
8

View Slide

Business Beneﬁts
9
BI and SQL
Analytics
Fresher
and faster
insights
Quicker and
better business
decisions
Data
Engineering
Sooner
availability of
cleaned data
More business
use cases
Data Science
and ML
More frequent
model update
and inference
Better model
efﬁcacy
Event Driven
Application
Faster customized
response
and action
Better and
differentiated
customer
experience

View Slide

Streaming
Misconceptions

View Slide

Misconception #1
Stream processing is only for low latency use cases
X
spark.readStream
.format("delta")
.option("maxFilesPerTrigger", "1")
.load(inputDir)
.writeStream
.trigger(Trigger.AvailableNow)
.option("checkpointLocation",chkdir)
.start()
11
Stream processing can
be applied to use cases
of any latency
“Batch” is a special case
of streaming

View Slide

Misconception #2
Latency
Accuracy Cost
Choose the right
latency, accuracy, and
cost tradeoff for each
speciﬁc use case
12
The lower the latency, the "better"
X

View Slide

Streaming is about the
programming paradigm

View Slide

Spark Structured
Streaming

View Slide

Structured Streaming
15
A scalable and
fault-tolerant stream
processing engine built
on the Spark SQL engine
+Project Lightspeed: predictable low latencies

View Slide

Structured Streaming
16
● Read from an initial
offset position
● Keep tracking offset
position as
processing makes
progress
Source
● Apply the same
transformations
using a standard
dataframe
Transformation
● Write to a target
● Keep updating
checkpoint as
processing makes
progress
Sink
Trigger

View Slide

source
transformation
sink
config
spark.readStream.format("kafka|kinesis|socket|...")
.option(<>,<>)...
.load()
.select(cast("string").alias("jsonData"))
.select(from_json($"jsonData",jsonSchema).alias("payload"))
.writeStream
.format("delta")
.option("path",...)
.trigger("30 seconds")
.option("checkpointLocation",...)
.start()
Streaming ETL
17

View Slide

Trigger Types
18
● Default: Process as soon as the previous batch has been processed
● Fixed interval: Process at a user-speciﬁed time interval
● One-time: Process all of the available data and then stop

View Slide

Output Modes
19
● Append (Default): Only new rows added to the result
table since the last trigger will be output to the sink
● Complete: The whole result table will be output to the
sink after each trigger
● Update: Only the rows updated in the result table since
the last trigger will be output to the sink

View Slide

Structured Streaming Benefits
High
Throughput
Optimized for high
throughput and
low cost
Rich Connector
Ecosystem
Streaming
connectors ranging
from message
buses to object
storage services
Exactly Once
Semantics
Fault-tolerance and
exactly once
semantics
guarantee
correctness
Unified Batch and
Streaming
Unified API makes
development and
maintenance
simple
20

View Slide

The Data Lakehouse

View Slide

Data Maturity Curve
Data + AI Maturity
Competitive Advantage
Reports
Clean Data
Ad Hoc
Queries
Data
Exploration
Predictive
Modeling
Prescriptive
Analytics
Automated
Decision Making
Data Lake
for AI
Data Warehouse
for BI
Data Maturity Curve
What will happen?
What happened?
22

View Slide

Business
Intelligence
SQL
Analytics
Data Science
& ML
Data
Streaming
Two disparate, incompatible data platforms
Structured tables Unstructured ﬁles:
logs, text, images, video,
Data Warehouse Data Lake
Governance and Security
Table ACLs
Governance and Security
Files and Blobs
Copy subsets of data
Disjointed
and duplicative
data silos
Incompatible
security and
governance models
Incomplete
support for
use cases
23

View Slide

Databricks
Lakehouse Platform
Lakehouse Platform
Data
Warehousing
Data
Engineering
Data Science
and ML
Data
Streaming
All structured and unstructured data
Cloud Data Lake
Unity Catalog
Fine-grained governance for data and AI
Delta Lake
Data reliability and performance
Simple
Unify your data warehousing and AI
use cases on a single platform
Open
Built on open source and open standards
Multicloud
One consistent data platform across clouds

View Slide

Streaming on the Lakehouse
Streaming on the Lakehouse
Lakehouse Platform
All structured and unstructured data
Cloud Data Lake
Unity Catalog
Fine-grained governance for data and AI
Delta Lake
Data reliability and performance
Streaming
Ingesting
Streaming
ETL
Event
Processing
Event Driven
Application
ML Inference
Live Dashboard
Near Real-time
Query
Alert
Fraud Prevention
Dynamic UI
Dynamic Ads
Dynamic Pricing
Device Control
Game Scene
Update
Etc.
DB Change
Data Feeds
Clickstreams
Machine &
Application Logs
Mobile & IoT Data
Application Events

View Slide

Lakehouse Differentiations
26
Uniﬁed Batch and
Streaming
No overhead of
learning, developing
on, or maintaining
two sets of APIs and
data processing
stacks
Favorite Tools
Provide diverse users
with their favorite tools
to work with streaming
data, enabling the
broader organization to
take advantage of
streaming
End-to-End
Streaming
Has everything you
need, no need to
stitch together
different streaming
technology stacks
or tune them to work
together
Optimal Cost
Structure
Easily conﬁgure the
right latency-cost
tradeoff for each of
your streaming
workloads

View Slide

but what is the
foundation of the
Lakehouse Platform?

View Slide

One source of truth for all your data
Open format Delta Lake is the foundation of the Lakehouse
• open source format Delta Lake, based on Parquet
• adds quality, reliability, and performance
to your existing data lakes
• Provides one common data management
framework for Batch & Streaming, ETL, Analytics & ML
✓ ACID Transactions
✓ Time Travel
✓ Schema Enforcement
✓ Identity Columns
✓ Advanced Indexing
✓ Caching
✓ Auto-tuning
✓ Python, SQL, R, Scala Support
All Open Source

View Slide

Lakehouse: Delta Lake Publication
Link to PDF

View Slide

Delta’s expanding ecosystem of connectors
N
ew
!
Coming Soon!
10x
Monthly
download
growth in
just one
year
>8 Million
monthly downloads

View Slide

Under the Hood
my_table/ ← table name
_delta_log/ ← delta log
00000.json ← delta for tx
00001.json
…
00010.json
00010.chkpt.parquet ← checkpoint files
date=2022-01-01/ ← optional partition
File-1.parquet ← data in parquet format
Write ahead log with optimistic concurrency provides
serializable ACID transactions:

View Slide

Delta Lake Quickstart
pyspark --packages io.delta:delta-core_2.12:1.0.0 \
--conf "spark.sql.extensions= \
io.delta.sql.DeltaSparkSessionExtension" \
--conf "spark.sql.catalog.spark_catalog= \
org.apache.spark.sql.delta.catalog.DeltaCatalog"
https://github.com/delta-io/delta

View Slide

Spark: Convert to Delta
33
#convert to delta format
cust = spark.read.json('/data/customers/')
cust.write.format("delta").saveAsTable(table_name)

View Slide

All of Delta Lake 2.0 is open
ACID
Transactions
Scalable
Metadata
Time Travel Open Source
Uniﬁed
Batch/Streaming
Schema Evolution
/Enforcement
Audit History DML Operations
OPTIMIZE
Compaction
OPTIMIZE
ZORDER
Change data
feed
Table Restore S3 Multi-cluster
writes
MERGE
Enhancements
Stream
Enhancements
Simpliﬁed
Logstore
Data Skipping
via Column
Stats
Multi-part checkpoint
writes
Generated
Columns
Column
Mapping
Generated
column support
w/ partitioning
Identity
Columns
Subqueries in
deletes and
updates
Clones
Iceberg to Delta
converter
Fast metadata
only deletes
Coming Soon!

View Slide

Without Delta Lake: No UPDATE

View Slide

Delta Examples (in SQL for brevity)
36
SELECT * FROM student TIMESTAMP AS OF "2022-05-28"
CREATE TABLE test_student SHALLOW CLONE student
DESCRIBE HISTORY student
UPDATE student SET lastname = "Miller" WHERE id = 2805
OPTIMIZE student ZORDER BY age
VACUUM student RETAIN 12 HOURS
Update/Upsert
Time travel
Clones
Table History
Co-locate
Compact Files

View Slide

… but how does this perform for
DWH workloads?

View Slide

Databricks SQL Photon
Serverless
Eliminate compute
infrastructure management
Instant, Elastic Compute
Zero Management
Lower TCO
Vectorized C++ execution engine with
Apache Spark API
https://dbricks.co/benchmark
TPC-DS Benchmark 100 TB

View Slide

… and streaming?

View Slide

Streaming: Built into the foundation
40
display(spark.readStream.format("delta").table("heart.bpm").
groupBy("bpm",window("time", "1 minute"))
.avg("bpm").orderBy("window",ascending=True))
Delta Tables can be
streaming
sources and sinks

View Slide


View Slide

How can you simplify
Streaming Ingestion and
Data Pipelines?

View Slide

Auto Loader
• Python & Scala (and SQL in Delta Live Tables!)
• Streaming data source with incremental loading
• Exactly once ingestion
• Scales to large amounts of data
• Designed for structured, semi-structured and unstructured data
• Schema inference, enforcement with data rescue, and evolution
df = spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "json")
.load("/path/to/table")
43

View Slide

“We ingest more than
5 petabytes per day
with Auto Loader”
44

View Slide

Delta Live Tables
45
CREATE STREAMING LIVE TABLE raw_data
AS SELECT *
FROM cloud_files ("/raw_data", "json")
CREATE LIVE TABLE clean_data
AS SELECT …
FROM LIVE .raw_data
Reliable, declarative, streaming data pipelines in SQL or Python
Accelerate ETL development
Declare SQL or Python and DLT automatically
orchestrates the DAG, handles retries, changing data
Automatically manage your infrastructure
Automates complex tedious activities like recovery,
auto-scaling, and performance optimization
Ensure high data quality
Deliver reliable data with built-in quality controls,
testing, monitoring, and enforcement
Unify batch and streaming
Get the simplicity of SQL with freshness
of streaming with one uniﬁed API

View Slide

COPY INTO
• SQL command
• Idempotent and incremental
• Great when source directory contains ~ thousands of ﬁles
• Schema automatically inferred
COPY INTO my_delta_table
FROM 's3://my-bucket/path/to/csv_files'
FILEFORMAT = CSV
FORMAT_OPTIONS ('header'='true','inferSchema'='true')
46

View Slide

Build sophisticated workﬂows inside your Databricks workspace
with a few clicks, or connect to your favorite IDE.
Simple Workﬂows
47

View Slide

10k View

View Slide

Lakehouse Platform: Streaming Options
49
Messaging Systems
(Apache Kafka, Kinesis, …)
Files / Object Stores
(S3, ADLS, GCS, …)
Apache Kafka
Connector for
SSStreaming
Databricks
Auto Loader
Spark Structured Streaming
Delta Live Tables (SQL/Python)
Delta Lake
Data
Consumers
Delta Lake
Sink Connector

View Slide

It's demo time!

View Slide

Demo 1

View Slide

Demo: Data Donation Project
https://corona-datenspende.de/science/en/

View Slide

System Architecture

View Slide

DLT: Directly Ingest Streaming Data

View Slide

Demo 2

View Slide

Delta Live Tables
Twitter Sentiment Analysis

View Slide

Tweepy API: Streaming Twitter Feed

View Slide

Auto Loader: Streaming Data Ingestion
Ingest Streaming Data in a Delta Live Table pipeline

View Slide

Declarative, auto scaling Data Pipelines in SQL
CTAS Pattern: Create Table As Select …

View Slide

Declarative, auto scaling Data Pipelines

View Slide

DWH / SQL Persona

View Slide

ML/DS: Hugging Face Transformer
62

View Slide

Resources

View Slide

Demos on Databricks Github
https://github.com/databricks/delta-live-tables-notebooks

View Slide

Databricks Blog: DLT with Apache Kafka
https://www.databricks.com/blog/2022/08/09/low-latency-streaming-data-pipelines-with-delta-live-tables-and-apache-kafka.html

View Slide

Resources
• Spark on Databricks
• Databricks Demo Hub
• Databricks Blog
• Databricks Community / Forum (join to ask your tech questions!)
• Training & Certiﬁcation

View Slide

Data + AI Summit: Sign up now

View Slide

Thank You!
https://speakerdeck.com/fmunz
@frankmunz
Try
Databricks
free

View Slide

Streaming Data on the Lakehouse (Devoxx Conference ATH)

Streaming Data on the Lakehouse (Devoxx Conference ATH)

Frank Munz

More Decks by Frank Munz

Other Decks in Programming

Featured

Transcript