Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Stream Processing with Apache Flink
Search
Kristian Kottke
September 13, 2018
Programming
0
160
Stream Processing with Apache Flink
Kristian Kottke
September 13, 2018
Tweet
Share
More Decks by Kristian Kottke
See All by Kristian Kottke
Jeder wie er will, aber so nicht
kkottke
0
32
Turmbau_zu_Babel.pdf
kkottke
0
110
Reactive Microservices based on Vert.x
kkottke
0
210
Graph Processing using Apache Flink
kkottke
0
110
Other Decks in Programming
See All in Programming
Compose Multiplatform × AI で作る、次世代アプリ開発支援ツールの設計と実装
thagikura
0
160
Swift Updates - Learn Languages 2025
koher
2
480
もうちょっといいRubyプロファイラを作りたい (2025)
osyoyu
1
440
CJK and Unicode From a PHP Committer
youkidearitai
PRO
0
110
RDoc meets YARD
okuramasafumi
4
170
Reading Rails 1.0 Source Code
okuramasafumi
0
240
Ruby Parser progress report 2025
yui_knk
1
440
ユーザーも開発者も悩ませない TV アプリ開発 ~Compose の内部実装から学ぶフォーカス制御~
taked137
0
180
実用的なGOCACHEPROG実装をするために / golang.tokyo #40
mazrean
1
280
機能追加とリーダー業務の類似性
rinchoku
2
1.3k
そのAPI、誰のため? Androidライブラリ設計における利用者目線の実践テクニック
mkeeda
2
310
testingを眺める
matumoto
1
140
Featured
See All Featured
Fantastic passwords and where to find them - at NoRuKo
philnash
52
3.4k
A better future with KSS
kneath
239
17k
Unsuck your backbone
ammeep
671
58k
Making Projects Easy
brettharned
117
6.4k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.4k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Faster Mobile Websites
deanohume
309
31k
Context Engineering - Making Every Token Count
addyosmani
3
48
Imperfection Machines: The Place of Print at Facebook
scottboms
268
13k
BBQ
matthewcrist
89
9.8k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Embracing the Ebb and Flow
colly
87
4.8k
Transcript
Java Forum Nord Kristian Kottke From one Stream Stream Processing
with Apache Flink
©iteratec Whoami Kristian Kottke › Senior Software Engineer -> iteratec
Interests › Software Architecture › Big Data Technologies
[email protected]
github.com/kkottke xing.to/kkottke speakerdeck.com/kkottke 2
©iteratec 4
©iteratec Batch Processing 5
©iteratec Stream Processor
©iteratec Lambda Architecture 7
©iteratec Lambda Architecture 8
©iteratec Streaming Architecture 9
©iteratec Streaming Architecture 10
©iteratec Stream Processing
©iteratec Streams following: https://flink.apache.org/flink-architecture.html ← bounded stream → ← bounded
stream → now start of the stream past future unbounded stream 12
©iteratec State following: https://ci.apache.org/projects/flink/flink-docs-release-1.6/ Local State Remote State Periodic Checkpoint
13
©iteratec Time
©iteratec Time Event Time Processing Time Ingestion Time 15
©iteratec Windows
©iteratec Window Tumbling Key 1 12:00 12:10 12:20 12:30 12:40
12:50 Key 2 Key 3 17
©iteratec Window Sliding Key 1 12:00 12:10 12:20 12:30 12:40
12:50 Key 2 Key 3 18
©iteratec Window Session Key 1 12:00 12:10 12:20 12:30 12:40
12:50 Key 2 Key 3 19
©iteratec 20 20 Window › Watermark › Trigger › Late
Data › Discard › Redirect into separate Stream › Update result Key 1
©iteratec 22 22 Guarantees › At most once › At
least once › Exactly once › Processor State › End-2-End Exactly once › Resettable / Replayable Source & Sink › Idempotency Source Sink State
©iteratec 24
©iteratec Apache Flink Databases Stream following: https://ci.apache.org/projects/flink/flink-docs-release-1.6/ Storage Application Streams
Historic Data Transactions Logs IoT Clicks ..... ...framework and distributed processing engine for stateful computations over unbounded and bounded data streams 25
©iteratec Apache Flink Files, HDFS, S3, JDBC, Kafka, ... Local
Cluster Cloud DataStream API FlinkML Gelly Table & SQL CEP Table & SQL Storage Deployment Runtime API Libraries following: https://ci.apache.org/projects/flink/flink-docs-release-1.6/ DataSet API 26
©iteratec Apache Flink DataStream<String> messages = env.addSource( new FlinkKafkaConsumer<>(...)); DataStream<Tick>
ticks = messages.map( Tick::parse); DataStream<Tick> maxValues = ticks .keyBy(„id“) .timeWindow(Time.seconds(10)) .maxBy(„value“); stats.addSink(new BucketingSink(„/path/to/dir“)); OP OP OP OP Transformation Transformation Source Sink 28
©iteratec Code
©iteratec DataStream<String> inputStream = env.addSource(new FlinkKafkaConsumer<>(...)); DataStream<Tick> ticks = inputStream
.map(Tick::parse) .assignTimestampsAndWatermarks(new PeriodicAssigner(Time.seconds(5))); DataStream<Tick> maxValues = ticks .keyBy("id") .timeWindow(Time.seconds(10)) .maxBy("value"); Window Functions 33
©iteratec DataStream<Tick> performanceValues = ticks .keyBy("id") .timeWindow(Time.seconds(10)) .trigger(new ThresholdTrigger(10d)) .process(new
PerformanceFunction()); public void process( Tuple key, Context ctx, Iterable<Tick> ticks, Collector<Tick> out) { /* calculate min / max value */ out.collect(tick); } Window Functions 34
©iteratec public void processElement(Tick tick, Context ctx, Collector<Tick> out) {
... ctx.timerService().registerEventTimeTimer(timerTimestamp); ... } public void onTimer(long timestamp, OnTimerContext ctx, Collector<Tick> out) { ... ctx.output(outputTag, ctx.getCurrentKey()); ... } Timer Service 36
©iteratec DataStream<Tick> priceAlerts = ticks .keyBy("id") .flatMap(new PriceAlertFunction(10d)); public void
open(Configuration parameters) { // ... previousPriceState = getRuntimeContext().getState(previousPriceDescriptor); } public void flatMap(Tick tick, Collector<Tick> out) throws Exception { if (Math.abs(tick.value - previousPriceState.value()) > threshold) { out.collect(tick); } previousPriceState.update(tick.value); } Value State 38
©iteratec DataStream<Threshold> thresholds = env.addSource(...); BroadcastStream<Threshold> thresholdBroadcast = thresholds.broadcast(thresholdsDescriptor); DataStream<Tick>
priceAlerts = ticks .keyBy("id") .connect(thresholdBroadcast) .process(new UpdatablePriceDiffFunction()); Broadcast State 39
©iteratec
©iteratec Queryable State 43 TaskManager TaskManager TaskManager
©iteratec Complex Event Processing Stream Pattern Pattern Stream 44
©iteratec Table & SQL Dynamic Table Dynamic Table Stream Stream
Continuous Query State 45
©iteratec Alternatives source: https://commons.wikimedia.org 46
©iteratec Wrap Up › Data usually occur in streams ›
Batch Processing doesn’t meet the modern requirements regarding continuous data streams › Stream Processing › Powerful › Higher / manageable complexity › Real-time / low latency › Intuitiveness 47
www.iteratec.de Contact Kristian Kottke
[email protected]
github.com/kkottke xing.to/kkottke speakerdeck.com/kkottke