Streaming Data on the Lakehouse (Devoxx Conference ATH)

Slide 1

Slide 1 text

Slide 2

Slide 2 text

©2022 Databricks Inc. — All rights reserved About me • Principal @Databricks. Data, Analytics and AI products • All things large scale data & compute • Based in Munich, 🍻 ⛰ 🥨 󰎲 • Twitter: @frankmunz • Formerly AWS Tech Evangelist, SW architect, data scientist, published author etc.

Slide 3

Slide 3 text

Slide 4

Slide 4 text

©2022 Databricks Inc. — All rights reserved Spark SQL and Dataframe API # PYTHON: read multiple JSON files with multiple datasets per file cust = spark.read.json('/data/customers/') # cust.createOrReplaceTempView("customers") cust.filter(cust['car_make']== "Audi").show() %sql select * from customers where car_make = "Audi" -- PYTHON: -- spark.sql('select * from customers where car_make ="Audi"').show()

Slide 5

Slide 5 text

Slide 6

Slide 6 text

©2022 Databricks Inc. — All rights reserved Reading from Files vs Reading from Streams # read file(s) cust = spark.read.format("json").path('/data/customers/') # read from messaging platform sales = spark.readStream.format("kafka|kinesis|socket").load()

Slide 7

Slide 7 text

©2022 Databricks Inc. — All rights reserved Stream Processing is continuous and unbounded Stream Processing 7 Traditional Processing is one-off and bounded 1 Data Source 2 Processing Data Source Processing

Slide 8

Slide 8 text

©2022 Databricks Inc. — All rights reserved Technical Advantages A more intuitive way of capturing and processing continuous and unbounded data Lower latency for time sensitive applications and use cases Better fault-tolerance through checkpointing Better compute utilization and scalability through continuous and incremental processing 8

Slide 9

Slide 9 text

©2022 Databricks Inc. — All rights reserved Business Beneﬁts 9 BI and SQL Analytics Fresher and faster insights Quicker and better business decisions Data Engineering Sooner availability of cleaned data More business use cases Data Science and ML More frequent model update and inference Better model efﬁcacy Event Driven Application Faster customized response and action Better and differentiated customer experience

Slide 10

Slide 10 text

Slide 11

Slide 11 text

©2022 Databricks Inc. — All rights reserved Misconception #1 Stream processing is only for low latency use cases X spark.readStream .format("delta") .option("maxFilesPerTrigger", "1") .load(inputDir) .writeStream .trigger(Trigger.AvailableNow) .option("checkpointLocation",chkdir) .start() 11 Stream processing can be applied to use cases of any latency “Batch” is a special case of streaming

Slide 12

Slide 12 text

©2022 Databricks Inc. — All rights reserved Misconception #2 Latency Accuracy Cost Choose the right latency, accuracy, and cost tradeoff for each speciﬁc use case 12 The lower the latency, the "better" X

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

©2022 Databricks Inc. — All rights reserved Structured Streaming 16 ● Read from an initial offset position ● Keep tracking offset position as processing makes progress Source ● Apply the same transformations using a standard dataframe Transformation ● Write to a target ● Keep updating checkpoint as processing makes progress Sink Trigger

Slide 17

Slide 17 text

©2022 Databricks Inc. — All rights reserved source transformation sink config spark.readStream.format("kafka|kinesis|socket|...") .option(<>,<>)... .load() .select(cast("string").alias("jsonData")) .select(from_json($"jsonData",jsonSchema).alias("payload")) .writeStream .format("delta") .option("path",...) .trigger("30 seconds") .option("checkpointLocation",...) .start() Streaming ETL 17

Slide 18

Slide 18 text

©2022 Databricks Inc. — All rights reserved Trigger Types 18 ● Default: Process as soon as the previous batch has been processed ● Fixed interval: Process at a user-speciﬁed time interval ● One-time: Process all of the available data and then stop

Slide 19

Slide 19 text

©2022 Databricks Inc. — All rights reserved Output Modes 19 ● Append (Default): Only new rows added to the result table since the last trigger will be output to the sink ● Complete: The whole result table will be output to the sink after each trigger ● Update: Only the rows updated in the result table since the last trigger will be output to the sink

Slide 20

Slide 20 text

©2022 Databricks Inc. — All rights reserved Structured Streaming Benefits High Throughput Optimized for high throughput and low cost Rich Connector Ecosystem Streaming connectors ranging from message buses to object storage services Exactly Once Semantics Fault-tolerance and exactly once semantics guarantee correctness Unified Batch and Streaming Unified API makes development and maintenance simple 20

Slide 21

Slide 21 text

Slide 22

Slide 22 text

©2022 Databricks Inc. — All rights reserved Data Maturity Curve Data + AI Maturity Competitive Advantage Reports Clean Data Ad Hoc Queries Data Exploration Predictive Modeling Prescriptive Analytics Automated Decision Making Data Lake for AI Data Warehouse for BI Data Maturity Curve What will happen? What happened? 22

Slide 23

Slide 23 text

©2022 Databricks Inc. — All rights reserved Business Intelligence SQL Analytics Data Science & ML Data Streaming Two disparate, incompatible data platforms Structured tables Unstructured ﬁles: logs, text, images, video, Data Warehouse Data Lake Governance and Security Table ACLs Governance and Security Files and Blobs Copy subsets of data Disjointed and duplicative data silos Incompatible security and governance models Incomplete support for use cases 23

Slide 24

Slide 24 text

©2022 Databricks Inc. — All rights reserved 24 Databricks Lakehouse Platform Lakehouse Platform Data Warehousing Data Engineering Data Science and ML Data Streaming All structured and unstructured data Cloud Data Lake Unity Catalog Fine-grained governance for data and AI Delta Lake Data reliability and performance Simple Unify your data warehousing and AI use cases on a single platform Open Built on open source and open standards Multicloud One consistent data platform across clouds

Slide 25

Slide 25 text

©2022 Databricks Inc. — All rights reserved 25 Streaming on the Lakehouse Streaming on the Lakehouse Lakehouse Platform All structured and unstructured data Cloud Data Lake Unity Catalog Fine-grained governance for data and AI Delta Lake Data reliability and performance Streaming Ingesting Streaming ETL Event Processing Event Driven Application ML Inference Live Dashboard Near Real-time Query Alert Fraud Prevention Dynamic UI Dynamic Ads Dynamic Pricing Device Control Game Scene Update Etc. DB Change Data Feeds Clickstreams Machine & Application Logs Mobile & IoT Data Application Events

Slide 26

Slide 26 text

©2022 Databricks Inc. — All rights reserved Lakehouse Differentiations 26 Uniﬁed Batch and Streaming No overhead of learning, developing on, or maintaining two sets of APIs and data processing stacks Favorite Tools Provide diverse users with their favorite tools to work with streaming data, enabling the broader organization to take advantage of streaming End-to-End Streaming Has everything you need, no need to stitch together different streaming technology stacks or tune them to work together Optimal Cost Structure Easily conﬁgure the right latency-cost tradeoff for each of your streaming workloads

Slide 27

Slide 27 text

Slide 28

Slide 28 text

©2022 Databricks Inc. — All rights reserved One source of truth for all your data Open format Delta Lake is the foundation of the Lakehouse • open source format Delta Lake, based on Parquet • adds quality, reliability, and performance to your existing data lakes • Provides one common data management framework for Batch & Streaming, ETL, Analytics & ML ✓ ACID Transactions ✓ Time Travel ✓ Schema Enforcement ✓ Identity Columns ✓ Advanced Indexing ✓ Caching ✓ Auto-tuning ✓ Python, SQL, R, Scala Support All Open Source

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

©2022 Databricks Inc. — All rights reserved Under the Hood my_table/ ← table name _delta_log/ ← delta log 00000.json ← delta for tx 00001.json … 00010.json 00010.chkpt.parquet ← checkpoint files date=2022-01-01/ ← optional partition File-1.parquet ← data in parquet format Write ahead log with optimistic concurrency provides serializable ACID transactions:

Slide 32

Slide 32 text

©2022 Databricks Inc. — All rights reserved Delta Lake Quickstart pyspark --packages io.delta:delta-core_2.12:1.0.0 \ --conf "spark.sql.extensions= \ io.delta.sql.DeltaSparkSessionExtension" \ --conf "spark.sql.catalog.spark_catalog= \ org.apache.spark.sql.delta.catalog.DeltaCatalog" https://github.com/delta-io/delta

Slide 33

Slide 33 text

Slide 34

Slide 34 text

©2022 Databricks Inc. — All rights reserved All of Delta Lake 2.0 is open ACID Transactions Scalable Metadata Time Travel Open Source Uniﬁed Batch/Streaming Schema Evolution /Enforcement Audit History DML Operations OPTIMIZE Compaction OPTIMIZE ZORDER Change data feed Table Restore S3 Multi-cluster writes MERGE Enhancements Stream Enhancements Simpliﬁed Logstore Data Skipping via Column Stats Multi-part checkpoint writes Generated Columns Column Mapping Generated column support w/ partitioning Identity Columns Subqueries in deletes and updates Clones Iceberg to Delta converter Fast metadata only deletes Coming Soon!

Slide 35

Slide 35 text

Slide 36

Slide 36 text

©2022 Databricks Inc. — All rights reserved Delta Examples (in SQL for brevity) 36 SELECT * FROM student TIMESTAMP AS OF "2022-05-28" CREATE TABLE test_student SHALLOW CLONE student DESCRIBE HISTORY student UPDATE student SET lastname = "Miller" WHERE id = 2805 OPTIMIZE student ZORDER BY age VACUUM student RETAIN 12 HOURS Update/Upsert Time travel Clones Table History Co-locate Compact Files

Slide 37

Slide 37 text

Slide 38

Slide 38 text

©2022 Databricks Inc. — All rights reserved Databricks SQL Photon Serverless Eliminate compute infrastructure management Instant, Elastic Compute Zero Management Lower TCO Vectorized C++ execution engine with Apache Spark API https://dbricks.co/benchmark TPC-DS Benchmark 100 TB

Slide 39

Slide 39 text

Slide 40

Slide 40 text

©2022 Databricks Inc. — All rights reserved Streaming: Built into the foundation 40 display(spark.readStream.format("delta").table("heart.bpm"). groupBy("bpm",window("time", "1 minute")) .avg("bpm").orderBy("window",ascending=True)) Delta Tables can be streaming sources and sinks

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Slide 43

Slide 43 text

©2022 Databricks Inc. — All rights reserved Auto Loader • Python & Scala (and SQL in Delta Live Tables!) • Streaming data source with incremental loading • Exactly once ingestion • Scales to large amounts of data • Designed for structured, semi-structured and unstructured data • Schema inference, enforcement with data rescue, and evolution df = spark.readStream.format("cloudFiles") .option("cloudFiles.format", "json") .load("/path/to/table") 43

Slide 44

Slide 44 text

Slide 45

Slide 45 text

©2022 Databricks Inc. — All rights reserved Delta Live Tables 45 CREATE STREAMING LIVE TABLE raw_data AS SELECT * FROM cloud_files ("/raw_data", "json") CREATE LIVE TABLE clean_data AS SELECT … FROM LIVE .raw_data Reliable, declarative, streaming data pipelines in SQL or Python Accelerate ETL development Declare SQL or Python and DLT automatically orchestrates the DAG, handles retries, changing data Automatically manage your infrastructure Automates complex tedious activities like recovery, auto-scaling, and performance optimization Ensure high data quality Deliver reliable data with built-in quality controls, testing, monitoring, and enforcement Unify batch and streaming Get the simplicity of SQL with freshness of streaming with one uniﬁed API

Slide 46

Slide 46 text

©2022 Databricks Inc. — All rights reserved COPY INTO • SQL command • Idempotent and incremental • Great when source directory contains ~ thousands of ﬁles • Schema automatically inferred COPY INTO my_delta_table FROM 's3://my-bucket/path/to/csv_files' FILEFORMAT = CSV FORMAT_OPTIONS ('header'='true','inferSchema'='true') 46

Slide 47

Slide 47 text

Slide 48

Slide 48 text

Slide 49

Slide 49 text

©2022 Databricks Inc. — All rights reserved Lakehouse Platform: Streaming Options 49 Messaging Systems (Apache Kafka, Kinesis, …) Files / Object Stores (S3, ADLS, GCS, …) Apache Kafka Connector for SSStreaming Databricks Auto Loader Spark Structured Streaming Delta Live Tables (SQL/Python) Delta Lake Data Consumers Delta Lake Sink Connector

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Slide 55

Slide 55 text

Slide 56

Slide 56 text

Slide 57

Slide 57 text

Slide 58

Slide 58 text

Slide 59

Slide 59 text

Slide 60

Slide 60 text

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Slide 63

Slide 63 text

Slide 64

Slide 64 text

Slide 65

Slide 65 text

©2022 Databricks Inc. — All rights reserved Databricks Blog: DLT with Apache Kafka https://www.databricks.com/blog/2022/08/09/low-latency-streaming-data-pipelines-with-delta-live-tables-and-apache-kafka.html

Slide 66

Slide 66 text

©2022 Databricks Inc. — All rights reserved Resources • Spark on Databricks • Databricks Demo Hub • Databricks Blog • Databricks Community / Forum (join to ask your tech questions!) • Training & Certiﬁcation

Slide 67

Slide 67 text

Slide 68

Slide 68 text