Flink in two nutshells

An introduction and analysis to the complexity of real-time data processing with Apache Flink, with the intent to offer an additional layer of abstraction, in order to facilitate the conceptual understanding of Apache Flink and data streaming in general.

Transcript

2. WHO? Who: Fränk Lotzkes • Long time independent • App/Web/Fullstack dev • Now Solution Architect • Cloud • Real time processing • @ Posedio • Passion for abstraction
3. WHAT? What? • Apache Flink • Open-source tool • For Data streaming
4. WHY? Why? • Extensive POC • From scratch • Lots of new XP & hindsight • To share with newcomers
5. HOW? 1. The Problem I. Streaming is complicated II. Flink is a complex III. Data & real-time are abstract 2. The Approach I. It can’t be THAT complicated? II. Can it? 3. The Solution (attempt) I. Layman's abstraction II. Flink in two nutshells
6. THE PROBLEM 1. I. Streaming is complicated II. Flink is complex III. Data & real-time are abstract
7. 7 1. THE PROBLEM I. Streaming is complicated • Streaming is built on top of complicated technology
8. 8 1. THE PROBLEM I. Streaming is complicated • Streaming is built on top of complicated technology • Flink tries to make streaming easy
9. 9 1. THE PROBLEM I. Streaming is complicated • Streaming is built on top of complicated technology • Flink tries to make streaming easy • With a lot of abstraction
10. 10 1. THE PROBLEM I. Streaming is complicated • Streaming is built on top of complicated technology • Flink tries to make streaming easy • With a lot of abstraction • Meanwhile, in the “learn Flink” intro:
11. 11 1. THE PROBLEM I. Streaming is complicated • Streaming is built on top of complicated technology • Flink tries to make streaming easy • With a lot of abstraction • Meanwhile, in the “learn Flink” intro: (q.e.d.)
12. 12 1. THE PROBLEM I. Streaming is complicated • Streaming is built on top of complicated technology • Flink tries to make streaming easy • With a lot of abstraction • Meanwhile, in the “learn Flink” intro: (q.e.d.) ÞLet’s take a step back: What exactly is Flink?
13. 13 1. THE PROBLEM II. Flink is complex
14. 14 1. THE PROBLEM II. Flink is complex • Flink tries to keep it simple
15. 15 1. THE PROBLEM II. Flink is complex • Flink “tries” to keep it simple Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.
16. 16 1. THE PROBLEM II. Flink is complex • Flink “tries” to keep it simple • With complex & detailed documentation • and some abstraction Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.
17. 17 1. THE PROBLEM II. Flink is complex • Flink “tries” to keep it simple • With complex & detailed documentation • and some abstraction • and multiple layers of abstraction Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.
18. 18 1. THE PROBLEM II. Flink is complex • Flink “tries” to keep it simple • With complex & detailed documentation • and some abstraction • and multiple layers of abstraction • …for Flink’s API… Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.
19. 19 1. THE PROBLEM II. Flink is complex • Flink “tries” to keep it simple • With complex & detailed documentation • and some abstraction • and multiple layers of abstraction • Wait… WDYM “Flink’s API”??? Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.
20. 20 1. THE PROBLEM II. Flink is complex • Flink “tries” to keep it simple • With complex & detailed documentation • and some abstraction • and multiple layers of abstraction • Wait… WDYM “Flink’s API”??? • …and for Flink’s cluster…. Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.
21. 21 1. THE PROBLEM II. Flink is complex • Flink “tries” to keep it simple • With complex & detailed documentation • and some abstraction • and multiple layers of abstraction • Wait… WDYM “Flink’s API”??? • Wait… WDYM “Flink’s Cluster”??? Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.
22. 22 1. THE PROBLEM II. Flink is complex • Flink “tries” to keep it simple • With complex & detailed documentation • and some abstraction • and multiple layers of abstraction • Wait… WDYM “Flink’s API”??? • Wait… WDYM “Flink’s Cluster”??? • Flink “forgets” to explain what it really is Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.
23. 23 1. THE PROBLEM II. Flink is complex • Flink “tries” to keep it simple • With complex & detailed documentation • and some abstraction • and multiple layers of abstraction • Wait… WDYM “Flink’s API”??? • Wait… WDYM “Flink’s Cluster”??? • Flink “forgets” to explain what it really is Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.
24. 24 1. THE PROBLEM II. Flink is complex • Flink “tries” to keep it simple • With complex & detailed documentation • and some abstraction • and multiple layers of abstraction • Wait… WDYM “Flink’s API”??? • Wait… WDYM “Flink’s Cluster”??? • Flink “forgets” to explain what it really is Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Þ Let’s take another step back: Þ What does Flink do? Simplified?
25. 25 1. THE PROBLEM III. Data & real-time are abstract 1. Flink takes some data & deserializes it
26. 26 1. THE PROBLEM III. Data & real-time are abstract 1. Flink takes some data & deserializes it 2. Does some transformation
27. 27 1. THE PROBLEM III. Data & real-time are abstract 1. Flink takes some data & deserializes it 2. Does some transformation 3. Serializes it again
28. 28 1. THE PROBLEM III. Data & real-time are abstract 1. Flink takes some data & deserializes it 2. Does some transformation 3. Serializes it again 4. And puts the data somewhere
29. 29 1. THE PROBLEM III. Data & real-time are abstract 1. Flink takes some data & deserializes it 2. Does some transformation 3. Serializes it again 4. And puts the data somewhere 5. All in real-time • BUT: • Surely not ANY data(de/serialization)? • Surely not ANY transformation? • Surely not ANYWHERE? • Also how real is real-time?
30. 30 1. THE PROBLEM III. Data & real-time are abstract 1. Flink takes some data & deserializes it 2. Does some transformation 3. Serializes it again 4. And puts the data somewhere 5. All in real-time • BUT: • Surely not ANY data(de/serialization)? • Surely not ANY transformation? • Surely not ANYWHERE? • Also how real is real-time?
31. 31 1. THE PROBLEM III. Data & real-time are abstract 1. Flink takes some data & deserializes it 2. Does some transformation 3. Serializes it again 4. And puts the data somewhere 5. All in real-time • BUT: • Surely not ANY data(de/serialization)? • Surely not ANY transformation? • Surely not ANYWHERE? • Also how real is real-time?
32. 32 1. THE PROBLEM III. Data & real-time are abstract 1. Flink takes some data & deserializes it 2. Does some transformation 3. Serializes it again 4. And puts the data somewhere 5. All in real-time • BUT: • Surely not ANY data(de/serialization)? • Surely not ANY transformation? • Surely not ANYWHERE? • Also how real is real-time? Þ OKOK, we get it! It’s complicated. But now what??
33. WHAT? 1. The Problem I. Streaming is complicated II. Flink is a complex III. Data & real-time are abstract 2.The Approach I. It can’t be THAT complicated? II. Can it? 3. The Solution (attempt) I. Layman's abstraction II. Flink in two nutshells
34. THE APPROACH 2. I. It can’t be THAT complicated? II. Can it?
35. 35 2. THE APPROACH I. It can’t be THAT complicated? • Getting an example to run seems fast & easy
36. 36 2. THE APPROACH I. It can’t be THAT complicated? • Getting an example to run seems fast & easy • From Flink’s first steps: 1. Have Java 11 2. Download Flink 3. Start local cluster (1 command) 4. Submit premade job (1 command)
37. 37 2. THE APPROACH I. It can’t be THAT complicated? • Getting an example to run seems fast & easy • From Flink’s first steps: 1. Have Java 11 2. Download Flink 3. Start local cluster (1 command) 4. Submit premade job (1 command)
38. 38 2. THE APPROACH I. It can’t be THAT complicated? • Getting an example to run seems fast & easy • From Flink’s first steps: 1. Have Java 11 2. Download Flink 3. Start local cluster (1 command) 4. Submit premade job (1 command) • => SUCCESS – with only a few steps
39. 39 2. THE APPROACH II. It can’t be THAT complicated? Can it?
40. 40 2. THE APPROACH II. It can’t be THAT complicated? Can it? • Despite the fast result: already some issues
41. 41 2. THE APPROACH II. It can’t be THAT complicated? Can it? • Despite the fast result: already some issues A) Java 11? Also, Flink version? -> my experience:
42. 42 2. THE APPROACH II. It can’t be THAT complicated? Can it? • Despite the fast result: already some issues A) Java 11? Also, Flink version?: • At the time of POC • Flink 1.20: work in progress • Java 17 + Flink 1.19 not 100% stable • Flink 1.18.1 + Java 11 = stable
43. 43 2. THE APPROACH II. It can’t be THAT complicated? Can it? • Despite the fast result: already some issues A) Java 11? Also, Flink version?: • At the time of POC • Flink 1.20: work in progress • Java 17 + Flink 1.19 not 100% stable • Flink 1.18.1 + Java 11 = stable
44. 44 2. THE APPROACH II. It can’t be THAT complicated? Can it? • Despite the fast result: already some issues A) Java 11? Also, Flink version?: • At the time of POC • Flink 1.20: work in progress • Java 17 + Flink 1.19 not 100% stable • Flink 1.18.1 + Java 11 = stable ÞSolution: Flink 1.18.1 + Java 11 + upgrade soon
45. 45 2. THE APPROACH II. It can’t be THAT complicated? Can it? • Despite the fast result: already some issues B) A local cluster
46. 46 2. THE APPROACH II. It can’t be THAT complicated? Can it? • Despite the fast result: already some issues B) A local cluster • Clusters require know-how • Luckily Flink provides simple solutions • Easy Standalone or native K8s setup • And a UI
47. 47 2. THE APPROACH II. It can’t be THAT complicated? Can it? • Despite the fast result: already some issues B) A local cluster • Clusters require know-how • Luckily Flink provides simple solutions • Easy Standalone or native K8s setup • And a UI => Solution: Use Flinks abstraction to set up a cluster and start the UI
48. 48 2. THE APPROACH II. It can’t be THAT complicated? Can it? • Despite the fast result: already some issues C) A premade job
49. 49 2. THE APPROACH II. It can’t be THAT complicated? Can it? • Despite the fast result: already some issues C) A premade job • What is a job? • The Source-Transformation-Sink pipeline • Some Java(/…) code • written with Flink libraries • from the framework
50. 50 2. THE APPROACH II. It can’t be THAT complicated? Can it? • Despite the fast result: already some issues C) A premade job • What is a job? • The Source-Transformation-Sink pipeline • Some Java(/…) code • written with Flink libraries • from the framework
51. 51 2. THE APPROACH II. It can’t be THAT complicated? Can it? • Despite the fast result: already some issues C) A premade job • What is a job? • The Source-Transformation-Sink pipeline • Some Java(/…) code • written with Flink libraries • from the framework => Solution: embrace the complexity
52. WHAT? 1. The Problem I. Streaming is complicated II. Flink is a complex III. Data & real-time are abstract 2. The Approach I. It can’t be THAT complicated? II. Can it? 3.The Solution (attempt) I. Layman's abstraction II. Flink in two nutshells
53. THE SOLUTION 3. I. Layman’s Abstraction
54. 54 3. THE SOLUTION I. Layman’s abstraction • Simple approach might not be simple enough • For a smooth jumpstart into Flink
55. 55 3. THE SOLUTION I. Layman’s abstraction • Simple approach might not be simple enough • For a smooth jumpstart into Flink • Issues-summary: A) Evolving ecosystem with many versions B) Some cluster know-how needed C) Data processing and it’s integration into the cluster still complex
56. 56 3. THE SOLUTION I. Layman’s abstraction • Simple approach might not be simple enough • For a smooth jumpstart into Flink • Issues-summary: A) Evolving ecosystem with many versions B) Some cluster know-how needed C) Data processing and it’s integration into the cluster still complex • A) -> easy solution: use the stable versions ✅
57. 57 3. THE SOLUTION I. Layman’s abstraction • Simple approach might not be simple enough • For a smooth jumpstart into Flink • Issues-summary: A) Evolving ecosystem with many versions B) Some cluster know-how needed C) Data processing and it’s integration into the cluster still complex • A) -> easy solution: use the stable versions ✅ • B) & C) -> I suggest a gross oversimplification
58. THE SOLUTION 3. II. Flink in two nutshells
59. 59 3. THE SOLUTION II. Flink in two nutshells • So, what is Flink if not “[…] a framework and distributed processing engine for stateful computations over unbounded and […]”
60. 60 3. THE SOLUTION II. Flink in two nutshells • So, what is Flink if not “[…] a framework and […]” • In a nutshell, Flink is…
61. 61 3. THE SOLUTION II. Flink in two nutshells • So, what is Flink if not “[…] a framework and […]” • In a nutshell, Flink is 2 nutshells
62. 62 3. THE SOLUTION II. Flink in two nutshells • So, what is Flink if not “[…] a framework and […]” • In a nutshell, Flink is 2 nutshells • Nutshell 1: • A toolbox for coding • With building blocks • To define WHAT will be happening to data • The Framework/Libraries/APIs/Job
63. 63 3. THE SOLUTION II. Flink in two nutshells • So, what is Flink if not “[…] a framework and […]” • In a nutshell, Flink is 2 nutshells • Nutshell 2: • A bigger toolbox • With commands and a UI • To define HOW data is handled • Mostly automatically • The Engine resp. The Cluster
64. 64 3. THE SOLUTION II. Flink in two nutshells • So, what is Flink if not “[…] a framework and […]” • In a nutshell, Flink is 2 nutshells • Nutshell 1: The Framework • Nutshell 2: The Engine
65. 65 3. THE SOLUTION II. Flink in two nutshells • So, what is Flink if not “[…] a framework and […]” • In a nutshell, Flink is 2 nutshells • Nutshell 1: The Framework • Nutshell 2: The Engine • Simple principle: • Build it • Package it • Send it • Let it run
66. 66 3. THE SOLUTION II. Flink in two nutshells BUT DON’T FORGET! • The devil is in the detail
67. 67 3. THE SOLUTION II. Flink in two nutshells BUT DON’T FORGET! • The devil is in the detail • Every nutshell has many nested ones • DataStream API, Table API, Connectors • REST API, Watermarks, Checkpoints • Jobmanager, Taskmanager, Client
68. 68 3. THE SOLUTION II. Flink in two nutshells BUT DON’T FORGET! • The devil is in the detail • Every nutshell has many nested ones • DataStream API, Table API, Connectors • REST API, Watermarks, Checkpoints • Jobmanager, Taskmanager, Client • Often hidden & hard to crack
69. 69 3. THE SOLUTION II. Flink in two nutshells BUT DON’T FORGET! • The devil is in the detail • Every nutshell has many nested ones • DataStream API, Table API, Connectors • REST API, Watermarks, Checkpoints • Jobmanager, Taskmanager, Client • Often hidden & hard to crack • Typical squirrel
70. THANK YOU! POSEDIO GMBH contact@posedio.com Weyringergasse 1-3, 1040 Wien, Millenium Park 4, 6980 Lustenau www.posedio.com • References: • https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/learn- flink/overview/ • https://nightlies.apache.org/flink/flink-docs-release- 1.18/docs/concepts/overview/ • Connectors DS-API: https://nightlies.apache.org/flink/flink-docs- release-1.19/docs/connectors/datastream/overview/ • Formats DS-API: https://nightlies.apache.org/flink/flink-docs- release-1.19/docs/connectors/datastream/formats/overview/ • Connectors TableAPI: https://nightlies.apache.org/flink/flink-docs- release-1.19/docs/connectors/table/overview/ • Formats TableAPI: https://nightlies.apache.org/flink/flink-docs- release-1.19/docs/connectors/table/formats/overview/ • First steps: https://nightlies.apache.org/flink/flink-docs- master/docs/try-flink/local_installation/ • https://flink.apache.org/what-is-flink/roadmap/ • https://medium.com/big-data-processing/twitter-streaming-using- flink-d19504b676a5