Timing is Everything
Understanding Event-Time Processing in Flink SQL
Sharon Xie，Flink Babysitter
Founding Engineer @ Decodable
Slide 2
Slide 2 text
Agenda
- Flink’s event time processing model
- Use cases in Flink SQL
Slide 3
Slide 3 text
What is Apache Flink
Stateful Computations over Data Streams
● Highly Scalable
● Exactly-once processing semantics
● Event time semantics and watermarks
● Layered APIs: Streaming SQL (easy to use) ↔ DataStream (expressive)
Slide 4
Slide 4 text
Context
● Flink streaming mode
● Flink SQL
● Current Flink version - 1.18 1.19
Slide 5
Slide 5 text
When is event time used?
● Monitoring and alerting
● Time-based compute or analytics
Slide 6
Slide 6 text
Event
Immutable record containing
the detail of something that
happened at some point in
time.
Slide 7
Slide 7 text
Time in Flink
Event Time
● The time at which the event happened
Processing Time
● The time at which the event is observed by
Flink
Slide 8
Slide 8 text
Event time vs Processing Time
● Event time is <
processing time
● The lag is arbitrary
● Events can be
out-of-order
Slide 9
Slide 9 text
Challenges
How do you know
when all of the events
are received for a
particular window?
Slide 10
Slide 10 text
Watermark
● Measures the progress of event time
● Tracks the maximum event time seen
● Indicates the completeness of the event time
Slide 11
Slide 11 text
No content
Slide 12
Slide 12 text
Create table sensors (
id bigint,
`value` integer
_time timestamp(3),
watermark for _time as _time - interval '3' minutes
) WITH (
'scan.watermark.emit.strategy'='on-event',
...
);
Define Watermark
Slide 13
Slide 13 text
Watermark Generation (on-event)
Slide 14
Slide 14 text
There is a
window that
ends at 1:05.
When can the
window close?
Quiz
Slide 15
Slide 15 text
There is a
window that
ends at 1:05.
When can the
window close?
Quiz - Answer
Slide 16
Slide 16 text
Multiple sources/partitions
Slide 17
Slide 17 text
Idle source/partition
● If a partition is idle (no events), the watermark
will not advance
● No result will be produced
● Solutions
○ Configure source idle timeout
■ set table.exec.source.idle-timeout = 1m
○ Balance the partitions
Slide 18
Slide 18 text
Implications
● Tradeoff between Correctness and Latency
● Latency
○ Results of a window is only seen after the window
closes
● Correctness
○ Late arriving events are discarded after the window
is closed
Slide 19
Slide 19 text
Correctness VS Latency
In general:
Alerting and monitoring: latency
Timely analytics: correctness
Slide 20
Slide 20 text
But…can I have both?
● Yes! Flink can process & emit
“updates” (changelog)
● No watermark is needed
● Downstream system must support
“updates”
● It’s costly - need to store global state
Slide 21
Slide 21 text
Trade-offs
Slide 22
Slide 22 text
Quick Summary
● Timely response & analytics are based on event time
● Flink uses watermark to account for out-of-order
events
● Watermark allows trade-off between accuracy and
latency
Window Types - Tumble/Fixed
Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/window-tvf/
● Fixed window
size
● No overlapping
● Each event
belongs to
exactly 1 window
Slide 27
Slide 27 text
Flink SQL (Window TVF)
● TVF - Table-Valued Function
● Returns a new relation with all columns of original stream and
additional 3 columns:
○ window_start, window_end, window_time
Window Types - Cumulative
Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/window-tvf/
● Similar to tumble
window, but with
early firing at the
defined interval
● Defined by max
window size and
window step
Slide 31
Slide 31 text
Window Types - Session
😃 Supported in Flink
1.19
● A new window is
created when the
consecutive event
time > session gap
Ref: https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/queries/window-tvf/#session
Slide 32
Slide 32 text
Window Join
● A window join adds the dimension of time into
the join criteria themselves.
● Use case: compute click-through events
Slide 33
Slide 33 text
Example - tumble window join
Slide 34
Slide 34 text
Example - hop window join
Slide 35
Slide 35 text
Temporal Join
● Enrich a stream with the value of the joined
record at the event time.
● Example: Continuously computing the price for
each order based on the exchange rate
happened when the order is placed
Slide 36
Slide 36 text
Example - temporal join
Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/joins/#temporal-joins
Slide 37
Slide 37 text
No content
Slide 38
Slide 38 text
Summary
● Event time is essential for timely response and
analytics
● Watermark and windowing are the key concepts
● Flink SQL simplifies event time processing