Stream Processing with Apache Flink

Transcript

1. Java Forum Nord Kristian Kottke From one Stream Stream Processing with Apache Flink
2. ©iteratec Whoami Kristian Kottke › Senior Software Engineer -> iteratec Interests › Software Architecture › Big Data Technologies Kristian.Kottke@iteratec.de github.com/kkottke xing.to/kkottke speakerdeck.com/kkottke 2
3. ©iteratec 4
4. ©iteratec Batch Processing 5
5. ©iteratec Stream Processor
6. ©iteratec Lambda Architecture 7
7. ©iteratec Lambda Architecture 8
8. ©iteratec Streaming Architecture 9
9. ©iteratec Streaming Architecture 10
10. ©iteratec Stream Processing
11. ©iteratec Streams following: https://flink.apache.org/flink-architecture.html ← bounded stream → ← bounded stream → now start of the stream past future unbounded stream 12
12. ©iteratec State following: https://ci.apache.org/projects/flink/flink-docs-release-1.6/ Local State Remote State Periodic Checkpoint 13
13. ©iteratec Time
14. ©iteratec Time Event Time Processing Time Ingestion Time 15
15. ©iteratec Windows
16. ©iteratec Window Tumbling Key 1 12:00 12:10 12:20 12:30 12:40 12:50 Key 2 Key 3 17
17. ©iteratec Window Sliding Key 1 12:00 12:10 12:20 12:30 12:40 12:50 Key 2 Key 3 18
18. ©iteratec Window Session Key 1 12:00 12:10 12:20 12:30 12:40 12:50 Key 2 Key 3 19
19. ©iteratec 20 20 Window › Watermark › Trigger › Late Data › Discard › Redirect into separate Stream › Update result Key 1
20. ©iteratec 22 22 Guarantees › At most once › At least once › Exactly once › Processor State › End-2-End Exactly once › Resettable / Replayable Source & Sink › Idempotency Source Sink State
21. ©iteratec 24
22. ©iteratec Apache Flink Databases Stream following: https://ci.apache.org/projects/flink/flink-docs-release-1.6/ Storage Application Streams Historic Data Transactions Logs IoT Clicks ..... ...framework and distributed processing engine for stateful computations over unbounded and bounded data streams 25
23. ©iteratec Apache Flink Files, HDFS, S3, JDBC, Kafka, ... Local Cluster Cloud DataStream API FlinkML Gelly Table & SQL CEP Table & SQL Storage Deployment Runtime API Libraries following: https://ci.apache.org/projects/flink/flink-docs-release-1.6/ DataSet API 26
24. ©iteratec Apache Flink DataStream<String> messages = env.addSource( new FlinkKafkaConsumer<>(...)); DataStream<Tick> ticks = messages.map( Tick::parse); DataStream<Tick> maxValues = ticks .keyBy(„id“) .timeWindow(Time.seconds(10)) .maxBy(„value“); stats.addSink(new BucketingSink(„/path/to/dir“)); OP OP OP OP Transformation Transformation Source Sink 28
25. ©iteratec Code
26. ©iteratec DataStream<String> inputStream = env.addSource(new FlinkKafkaConsumer<>(...)); DataStream<Tick> ticks = inputStream .map(Tick::parse) .assignTimestampsAndWatermarks(new PeriodicAssigner(Time.seconds(5))); DataStream<Tick> maxValues = ticks .keyBy("id") .timeWindow(Time.seconds(10)) .maxBy("value"); Window Functions 33
27. ©iteratec DataStream<Tick> performanceValues = ticks .keyBy("id") .timeWindow(Time.seconds(10)) .trigger(new ThresholdTrigger(10d)) .process(new PerformanceFunction()); public void process( Tuple key, Context ctx, Iterable<Tick> ticks, Collector<Tick> out) { /* calculate min / max value */ out.collect(tick); } Window Functions 34
28. ©iteratec public void processElement(Tick tick, Context ctx, Collector<Tick> out) { ... ctx.timerService().registerEventTimeTimer(timerTimestamp); ... } public void onTimer(long timestamp, OnTimerContext ctx, Collector<Tick> out) { ... ctx.output(outputTag, ctx.getCurrentKey()); ... } Timer Service 36
29. ©iteratec DataStream<Tick> priceAlerts = ticks .keyBy("id") .flatMap(new PriceAlertFunction(10d)); public void open(Configuration parameters) { // ... previousPriceState = getRuntimeContext().getState(previousPriceDescriptor); } public void flatMap(Tick tick, Collector<Tick> out) throws Exception { if (Math.abs(tick.value - previousPriceState.value()) > threshold) { out.collect(tick); } previousPriceState.update(tick.value); } Value State 38
30. ©iteratec DataStream<Threshold> thresholds = env.addSource(...); BroadcastStream<Threshold> thresholdBroadcast = thresholds.broadcast(thresholdsDescriptor); DataStream<Tick> priceAlerts = ticks .keyBy("id") .connect(thresholdBroadcast) .process(new UpdatablePriceDiffFunction()); Broadcast State 39
31. ©iteratec
32. ©iteratec Queryable State 43 TaskManager TaskManager TaskManager
33. ©iteratec Complex Event Processing Stream Pattern Pattern Stream 44
34. ©iteratec Table & SQL Dynamic Table Dynamic Table Stream Stream Continuous Query State 45
35. ©iteratec Alternatives source: https://commons.wikimedia.org 46
36. ©iteratec Wrap Up › Data usually occur in streams › Batch Processing doesn’t meet the modern requirements regarding continuous data streams › Stream Processing › Powerful › Higher / manageable complexity › Real-time / low latency › Intuitiveness 47
37. www.iteratec.de Contact Kristian Kottke Kristian.Kottke@iteratec.de github.com/kkottke xing.to/kkottke speakerdeck.com/kkottke