Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Stream Processing with Apache Flink
Search
Kristian Kottke
September 13, 2018
Programming
0
130
Stream Processing with Apache Flink
Kristian Kottke
September 13, 2018
Tweet
Share
More Decks by Kristian Kottke
See All by Kristian Kottke
Jeder wie er will, aber so nicht
kkottke
0
9
Turmbau_zu_Babel.pdf
kkottke
0
110
Reactive Microservices based on Vert.x
kkottke
0
150
Graph Processing using Apache Flink
kkottke
0
100
Other Decks in Programming
See All in Programming
Polars入門
daikikatsuragawa
1
160
Code Reviews
bkuhlmann
4
890
Apache Hive 4 on Treasure Data
ryukobayashi
1
410
Sheets API使ってみた
toshi0383
2
160
Site Reliability Engineering for GMO
pyama86
8
1.1k
ゆるい個人開発のススメ
kuroppe1819
10
1k
『Railsオワコン』と言われる時代に、なぜブルーモ証券はRailsを選ぶのか
free_world21
1
350
2 週間で Twitter Bot を作ってみた
contour_gara
0
760
Fragment Composition of GraphQL
quramy
13
1.4k
スクラムガイドのスプリントレトロスペクティブを改めて読みかえしてみた / Re-reading the Sprint Retrospective Section in the Scrum Guide
mackey0225
3
480
大規模UIKitベースアプリへのTCAの段階的導入/gradual-adoption-of-tca-in-a-large-scale-uikit-based-app
takehilo
2
200
From Spring Boot 2 to Spring Boot 3 with Java 21 and Jakarta EE
ivargrimstad
0
430
Featured
See All Featured
The Mythical Team-Month
searls
216
42k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
79
43k
Git: the NoSQL Database
bkeepers
PRO
423
63k
The Cult of Friendly URLs
andyhume
74
5.7k
Designing with Data
zakiwarfel
96
4.8k
In The Pink: A Labor of Love
frogandcode
138
21k
Typedesign – Prime Four
hannesfritz
36
2.1k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
26
2.3k
Being A Developer After 40
akosma
66
580k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
245
20k
The Straight Up "How To Draw Better" Workshop
denniskardys
228
130k
Bash Introduction
62gerente
605
210k
Transcript
Java Forum Nord Kristian Kottke From one Stream Stream Processing
with Apache Flink
©iteratec Whoami Kristian Kottke › Senior Software Engineer -> iteratec
Interests › Software Architecture › Big Data Technologies
[email protected]
github.com/kkottke xing.to/kkottke speakerdeck.com/kkottke 2
©iteratec 4
©iteratec Batch Processing 5
©iteratec Stream Processor
©iteratec Lambda Architecture 7
©iteratec Lambda Architecture 8
©iteratec Streaming Architecture 9
©iteratec Streaming Architecture 10
©iteratec Stream Processing
©iteratec Streams following: https://flink.apache.org/flink-architecture.html ← bounded stream → ← bounded
stream → now start of the stream past future unbounded stream 12
©iteratec State following: https://ci.apache.org/projects/flink/flink-docs-release-1.6/ Local State Remote State Periodic Checkpoint
13
©iteratec Time
©iteratec Time Event Time Processing Time Ingestion Time 15
©iteratec Windows
©iteratec Window Tumbling Key 1 12:00 12:10 12:20 12:30 12:40
12:50 Key 2 Key 3 17
©iteratec Window Sliding Key 1 12:00 12:10 12:20 12:30 12:40
12:50 Key 2 Key 3 18
©iteratec Window Session Key 1 12:00 12:10 12:20 12:30 12:40
12:50 Key 2 Key 3 19
©iteratec 20 20 Window › Watermark › Trigger › Late
Data › Discard › Redirect into separate Stream › Update result Key 1
©iteratec 22 22 Guarantees › At most once › At
least once › Exactly once › Processor State › End-2-End Exactly once › Resettable / Replayable Source & Sink › Idempotency Source Sink State
©iteratec 24
©iteratec Apache Flink Databases Stream following: https://ci.apache.org/projects/flink/flink-docs-release-1.6/ Storage Application Streams
Historic Data Transactions Logs IoT Clicks ..... ...framework and distributed processing engine for stateful computations over unbounded and bounded data streams 25
©iteratec Apache Flink Files, HDFS, S3, JDBC, Kafka, ... Local
Cluster Cloud DataStream API FlinkML Gelly Table & SQL CEP Table & SQL Storage Deployment Runtime API Libraries following: https://ci.apache.org/projects/flink/flink-docs-release-1.6/ DataSet API 26
©iteratec Apache Flink DataStream<String> messages = env.addSource( new FlinkKafkaConsumer<>(...)); DataStream<Tick>
ticks = messages.map( Tick::parse); DataStream<Tick> maxValues = ticks .keyBy(„id“) .timeWindow(Time.seconds(10)) .maxBy(„value“); stats.addSink(new BucketingSink(„/path/to/dir“)); OP OP OP OP Transformation Transformation Source Sink 28
©iteratec Code
©iteratec DataStream<String> inputStream = env.addSource(new FlinkKafkaConsumer<>(...)); DataStream<Tick> ticks = inputStream
.map(Tick::parse) .assignTimestampsAndWatermarks(new PeriodicAssigner(Time.seconds(5))); DataStream<Tick> maxValues = ticks .keyBy("id") .timeWindow(Time.seconds(10)) .maxBy("value"); Window Functions 33
©iteratec DataStream<Tick> performanceValues = ticks .keyBy("id") .timeWindow(Time.seconds(10)) .trigger(new ThresholdTrigger(10d)) .process(new
PerformanceFunction()); public void process( Tuple key, Context ctx, Iterable<Tick> ticks, Collector<Tick> out) { /* calculate min / max value */ out.collect(tick); } Window Functions 34
©iteratec public void processElement(Tick tick, Context ctx, Collector<Tick> out) {
... ctx.timerService().registerEventTimeTimer(timerTimestamp); ... } public void onTimer(long timestamp, OnTimerContext ctx, Collector<Tick> out) { ... ctx.output(outputTag, ctx.getCurrentKey()); ... } Timer Service 36
©iteratec DataStream<Tick> priceAlerts = ticks .keyBy("id") .flatMap(new PriceAlertFunction(10d)); public void
open(Configuration parameters) { // ... previousPriceState = getRuntimeContext().getState(previousPriceDescriptor); } public void flatMap(Tick tick, Collector<Tick> out) throws Exception { if (Math.abs(tick.value - previousPriceState.value()) > threshold) { out.collect(tick); } previousPriceState.update(tick.value); } Value State 38
©iteratec DataStream<Threshold> thresholds = env.addSource(...); BroadcastStream<Threshold> thresholdBroadcast = thresholds.broadcast(thresholdsDescriptor); DataStream<Tick>
priceAlerts = ticks .keyBy("id") .connect(thresholdBroadcast) .process(new UpdatablePriceDiffFunction()); Broadcast State 39
©iteratec
©iteratec Queryable State 43 TaskManager TaskManager TaskManager
©iteratec Complex Event Processing Stream Pattern Pattern Stream 44
©iteratec Table & SQL Dynamic Table Dynamic Table Stream Stream
Continuous Query State 45
©iteratec Alternatives source: https://commons.wikimedia.org 46
©iteratec Wrap Up › Data usually occur in streams ›
Batch Processing doesn’t meet the modern requirements regarding continuous data streams › Stream Processing › Powerful › Higher / manageable complexity › Real-time / low latency › Intuitiveness 47
www.iteratec.de Contact Kristian Kottke
[email protected]
github.com/kkottke xing.to/kkottke speakerdeck.com/kkottke