Large scale stream processing with Apache Flink

In today’s world it’s no longer enough to build systems that process big volumes of information. We now need applications that can handle large continuous streams of data with very low latency so we can react to the ever-changing environment around us. To efficiently handle such problems we need to deploy a stream processing solution. During the talk we’ll explore one of the most popular frameworks for stream processing – Apache Flink. We’ll see what unique capabilities it provides and how they apply to some real world problems. And we’ll also explore how it works under the hood and how to get the scalable and fault-tolerant stream processing that Flink provides.

Transcript

1. Large scale stream processing with Apache Flink Nikolay Stoitsev Sr. Software Engineer at Uber Tech Sofia
2. Stream Processing?
3. Stream Processing? User Interaction Logs
4. Stream Processing? User Interaction Logs Application Logs
5. Stream Processing? User Interaction Logs Application Logs Sensor Data
6. Stream Processing? User Interaction Logs Application Logs Sensor Data Database Commit Logs
7. Infinite Dataset
8. Producer Stream
9. Producer Stream HDFS
10. Producer Stream HDFS Hive
11. Producer Stream HDFS Hive Big Latency
12. Producer Stream HDFS Real-time service
13. Apache Storm storm.apache.org
14. High-latency & accurate vs. Low-latency & approximation
15. Lambda architecture
16. https://www.oreilly.com/ideas/questioning-the-lambda-architecture
17. Kappa Architecture
18. Use Apache Kafka Durable, scalable, fault-tolerant
19. Producer Kafka Stream Processor
22. Metrics we want to track Net payout Daily items sold Weekly items sold Order acceptance rate Order preparation speed Item rating
23. Real time
24. Scalable
25. Granular
26. Highly available
27. Order Stream Payment Stream User Rating Stream
28. Order Stream Payment Stream User Rating Stream Stream Processor OLAP
29. samza.apache.org
30. Apache Flink flink.apache.org
31. Everything is a batch vs. Everything is a stream
32. Single JVM Cluster Cloud Runtime DataSet API DataStream API
33. Dataflow graph
34. Source Source Operator Operator Operator Sinc OLAP
35. https://ci.apache.org/projects/flink/flink-docs-release-1.6/concepts/programming-model.html
36. https://ci.apache.org/projects/flink/flink-docs-release-1.6/concepts/programming-model.html
37. https://ci.apache.org/projects/flink/flink-docs-release-1.6/concepts/programming-model.html
38. Flink Program Optimizer Graph Builder Client
39. Flink Program Optimizer Graph Builder Client Job Manager Task Manager Task Manager
40. Flink Program Optimizer Graph Builder Client Job Manager Task Manager Task Manager Snapshot Store
41. Fault tolerant
42. Flink Program Optimizer Graph Builder Client Job Manager Task Manager Task Manager Snapshot Store
43. Lightweight Asynchronous Snapshots for Distributed Dataflows Paris Carbone, Gyula Fóra, Stephan Ewen Seif Haridi Kostas Tzoumas
44. Barrier Msg Msg Barrier Msg Msg Barrier Operator
45. Barrier Msg Msg Barrier Msg Msg Operator Msg Snapshot Store
46. Exactly Once Processing
47. Can handle very large state
48. Flink Program Optimizer Graph Builder Client Job Manager Task Manager Task Manager Snapshot Store
49. Flink Program Optimizer Graph Builder Client Job Manager Task Manager Task Manager Snapshot Store Job Manager Job Manager Zookeeper
50. Flink Program Optimizer Graph Builder Client Job Manager Task Manager Task Manager Snapshot Store Job Manager Job Manager Zookeeper
51. Flink Program Optimizer Graph Builder Client Task Manager Task Manager Snapshot Store Job Manager Job Manager Zookeeper
52. Joining Streams
53. Order Stream User Rating Stream
54. Order Stream User Rating Stream
55. Order Stream User Rating Stream Local Join Local Join
56. Order Stream User Rating Stream Local Join Local Join
57. Apache Flink • Can join streams • Fault tolerant • Exactly Once Processing • Combines stream and batch processing
58. … but it requires Java/Scala code
59. Scalable, efficient and robust
60. github.com/uber/AthenaX
61. SQL → what data to analyze Flink → how to analyze it
69. Resource estimation and auto scaling
70. Monitoring and automatic failure recovery
71. eng.uber.com/athenax
72. Thanks! Nikolay Stoitsev @ Uber