Slide 1

Slide 1 text

Timing is Everything Understanding Event-Time Processing in Flink SQL Sharon Xie,Flink Babysitter Founding Engineer @ Decodable

Slide 2

Slide 2 text

Agenda - Flink’s event time processing model - Use cases in Flink SQL

Slide 3

Slide 3 text

What is Apache Flink Stateful Computations over Data Streams ● Highly Scalable ● Exactly-once processing semantics ● Event time semantics and watermarks ● Layered APIs: Streaming SQL (easy to use) ↔ DataStream (expressive)

Slide 4

Slide 4 text

Context ● Flink streaming mode ● Flink SQL ● Current Flink version - 1.18 1.19

Slide 5

Slide 5 text

When is event time used? ● Monitoring and alerting ● Time-based compute or analytics

Slide 6

Slide 6 text

Event Immutable record containing the detail of something that happened at some point in time.

Slide 7

Slide 7 text

Time in Flink Event Time ● The time at which the event happened Processing Time ● The time at which the event is observed by Flink

Slide 8

Slide 8 text

Event time vs Processing Time ● Event time is < processing time ● The lag is arbitrary ● Events can be out-of-order

Slide 9

Slide 9 text

Challenges How do you know when all of the events are received for a particular window?

Slide 10

Slide 10 text

Watermark ● Measures the progress of event time ● Tracks the maximum event time seen ● Indicates the completeness of the event time

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

Create table sensors ( id bigint, `value` integer _time timestamp(3), watermark for _time as _time - interval '3' minutes ) WITH ( 'scan.watermark.emit.strategy'='on-event', ... ); Define Watermark

Slide 13

Slide 13 text

Watermark Generation (on-event)

Slide 14

Slide 14 text

There is a window that ends at 1:05. When can the window close? Quiz

Slide 15

Slide 15 text

There is a window that ends at 1:05. When can the window close? Quiz - Answer

Slide 16

Slide 16 text

Multiple sources/partitions

Slide 17

Slide 17 text

Idle source/partition ● If a partition is idle (no events), the watermark will not advance ● No result will be produced ● Solutions ○ Configure source idle timeout ■ set table.exec.source.idle-timeout = 1m ○ Balance the partitions

Slide 18

Slide 18 text

Implications ● Tradeoff between Correctness and Latency ● Latency ○ Results of a window is only seen after the window closes ● Correctness ○ Late arriving events are discarded after the window is closed

Slide 19

Slide 19 text

Correctness VS Latency In general: Alerting and monitoring: latency Timely analytics: correctness

Slide 20

Slide 20 text

But…can I have both? ● Yes! Flink can process & emit “updates” (changelog) ● No watermark is needed ● Downstream system must support “updates” ● It’s costly - need to store global state

Slide 21

Slide 21 text

Trade-offs

Slide 22

Slide 22 text

Quick Summary ● Timely response & analytics are based on event time ● Flink uses watermark to account for out-of-order events ● Watermark allows trade-off between accuracy and latency

Slide 23

Slide 23 text

Flink SQL Event-time Usage ● Windowed Aggregations ● Window join ● Temporal join

Slide 24

Slide 24 text

Windowing Put unbounded events into finite-sized temporal buckets, over which computation is applied.

Slide 25

Slide 25 text

Window Types ● Tumble / Fixed ● Hop / Sliding ● Cumulative ● Session

Slide 26

Slide 26 text

Window Types - Tumble/Fixed Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/window-tvf/ ● Fixed window size ● No overlapping ● Each event belongs to exactly 1 window

Slide 27

Slide 27 text

Flink SQL (Window TVF) ● TVF - Table-Valued Function ● Returns a new relation with all columns of original stream and additional 3 columns: ○ window_start, window_end, window_time

Slide 28

Slide 28 text

Example - Tumble Window

Slide 29

Slide 29 text

Window Types - Hop/Slide Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/window-tvf/ ● Fixed window size ● Overlaps when slide < window size

Slide 30

Slide 30 text

Window Types - Cumulative Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/window-tvf/ ● Similar to tumble window, but with early firing at the defined interval ● Defined by max window size and window step

Slide 31

Slide 31 text

Window Types - Session 😃 Supported in Flink 1.19 ● A new window is created when the consecutive event time > session gap Ref: https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/queries/window-tvf/#session

Slide 32

Slide 32 text

Window Join ● A window join adds the dimension of time into the join criteria themselves. ● Use case: compute click-through events

Slide 33

Slide 33 text

Example - tumble window join

Slide 34

Slide 34 text

Example - hop window join

Slide 35

Slide 35 text

Temporal Join ● Enrich a stream with the value of the joined record at the event time. ● Example: Continuously computing the price for each order based on the exchange rate happened when the order is placed

Slide 36

Slide 36 text

Example - temporal join Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/joins/#temporal-joins

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

Summary ● Event time is essential for timely response and analytics ● Watermark and windowing are the key concepts ● Flink SQL simplifies event time processing

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

Q&A @sharon_rxie