Let's play Flink - Fun with streaming applications at InnoGames

Let's play Flink - Fun with streaming applications at InnoGames
Presented at: code.talks 2018

Chocolate, ice cream and games are perhaps 3 of the most popular universally understood words that can bring joy to anyone between 5-60 years of age!

At InnoGames we not only have all three of those things but in addition we build up a powerful data infrastructure because it’s expensive to run your business blind. And being able to evaluate key performance indicators fast to make good decisions and deliver personalized and relevant content to each and every gamer is essential to be successful and it is how a customer becomes a fan.

With a revenue of 130 million Euros in 2016, InnoGames is one of the world's leading developers and providers of online games. InnoGames has more than 200 million registered players and has scored major successes with games such as Tribal Wars, Forge of Empires and Elvenar.

Our data infrastructure mainly consists of a data pipeline that covers the streaming part and a data platform to perform batch processing. The latter is based on the Hadoop ecosystem using technologies such as Hive, Spark, Hue, R and more to give our data scientists a high flexibility. There were several evolutions of the data pipeline, starting with Kestrel and custom streaming applications. Later on we switched the base technologies to Apache Kafka and Apache Storm. Last year we recreated our streaming infrastructure based on Apache Flink which is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications.

Transcript

1. Volker Janz Let‘s play Flink Big Data
2. LET´S PLAY FLINK Fun with streaming applications
3. LET‘S GO ON A ROADTRIP
4. IMAGINE You are driving with your family on the backseat
5. IMAGINE There is a lot of traffic, you have to concentrate
6. IMAGINE And now, while driving, you are closing your eyes
8. HOW DO YOU FEEL?
9. WOULD YOU EVER DO THAT? NOPE
10. METAPHOR The car is your company, team or project The passengers are your colleagues
11. WOULD YOU EVER DO THAT? NOPE IN A METAPHORICAL SENSE
12. YOU HAVE TO PROCESS DATA ON TIME AS IT HAPPENS
13. BATCH PROCESSING… …might cause accidents Because…
14. https://mapr.com/ebooks/streaming-architecture/chapter-01-why-event-streaming.html © Ellen Friedman, Ted Dunning
16. SIMILARITIES MAKE US HAPPY
17. SIMILARITIES WHICH TASTES BETTER?
18. SIMILARITIES WHICH TASTES BETTER?
19. SIMILARITIES THE FIRST IMPRESSION COUNTS The moment the customer enters the shop or the player plays his first session is crucial HALO EFFECT When one trait of a person or thing is used to make an overall judgment of that person or thing
20. IN ORDER TO MAKE A POSITIVE IMPACT A RESPONSE NEEDS TO HAPPEN QUICKLY
21. TIME-VALUE OF INFORMATION
22. REAL-TIME USER REPORTS TRAFFIC GAS PRICE SPEED TRAPS
23. RESPOND TO LIFE AS IT HAPPENS
24. BACK TO OUR TOPIC Let‘s have fun with streaming applications
25. STREAM PROCESSING STREAMS OF DATA GPS DATA WEB INTERACTION SENSOR DATA
26. STREAM PROCESSING PROCESSING DATA IN MOTION
27. STREAM PROCESSING YOUR CODE SOURCE SINK OPERATOR
28. STREAM PROCESSING LAKE
29. STREAM PROCESSING LAKE Real-Time Processing Explained: A Survey of Storm, Samza, Spark & Flink Wolfram Wingerath Cinema 6 15:00
30. STREAM PROCESSING LAKE
31. APACHE FLINK
32. APACHE FLINK Framework and distributed process engine for stateful computations on unbounded and bounded data streams
33. EVERYTHING IS A STREAM UNBOUNDED STREAMS BOUNDED STREAMS
34. EVERYTHING IS A STREAM UNBOUNDED STREAMS BOUNDED STREAMS AKA BATCH PROCESSING
35. TIME IN STREAMING EPISODE I EPISODE II EPISODE III EPISODE IV EPISODE V EPISODE VI EPISODE VII EPISODE VIII EPISODE IX 1999 2002 2005 1977 1980 1983 2015 2017 2019 The Phantom Menace Attack of the Clones Revenge of the Sith A New Hope The Empire Strikes Back Return of the Jedi The Force Awakens The Last Jedi ? ORDERED BY EVENT TIME PROCESSING TIME
36. TIME IN STREAMING EPISODE I EPISODE II EPISODE III EPISODE IV EPISODE V EPISODE VI EPISODE VII EPISODE VIII EPISODE IX 1999 2002 2005 1977 1980 1983 2015 2017 2019 The Phantom Menace Attack of the Clones Revenge of the Sith A New Hope The Empire Strikes Back Return of the Jedi The Force Awakens The Last Jedi ? EVENT TIME ORDERED BY PROCESSING TIME
37. TUMBLING WINDOWS 9 1 3 2 6 8 1 3 9 8 4 5 9 1 3 2 6 8 1 3 9 8 4 5 15 18 26 SENSOR SUM
38. SLIDING WINDOWS 9 1 3 2 6 8 1 3 9 8 4 5 9 1 3 2 6 8 1 3 9 8 4 5 9 1 3 2 6 8 1 3 9 8 4 5 15 18 26 19 21 SENSOR SUM
39. EXACTLY-ONCE EXACTLY-ONCE IN FLINK Each incoming event affects the final result exactly once It does not necessarily mean that each event gets processed only once Achieved with distributed snapshot/state checkpointing
40. BUILDING BLOCKS DATA SOURCE TRANSFORMATION DATA SINK
41. BUILDING BLOCKS DATA SOURCE TRANSFORMATION DATA SINK API
42. BUILDING BLOCKS SQL / TABLE API DataStream API ProcessFunction APIs (dynamic tables) (streams, windows) (events, state, time) HIGH LEVEL ANALYTICS API STREAM AND BATCH DATA PROCESSING STATEFUL EVENT- DRIVEN APPLICATIONS CONCISENESS EXPRESSIVENESS
43. LET‘S HAVE A CLOSER LOOK
44. LET‘S HAVE A CLOSER LOOK final StreamExecutionEnvironment env = getExecutionEnvironment(); final DataStreamSource<Integer> stream = env.fromElements(1, 2, 3, 4); stream .map((MapFunction<Integer, Integer>) i -> i + 2) .filter((FilterFunction<Integer>) i -> i % 2 == 0) .print(); env.execute(); DATA SOURCE TRANSFORMATION DATA SINK
45. RUNTIME YOUR FLINK APP FLINK RUNTIME D E P LOY
46. RUNTIME
47. RABBIT HOLE
48. RUNTIME SOURCE MAP PRINT FILTER STREAMING DATAFLOW (CONDENSED VIEW) OPERATOR CHAIN OPERATOR OPERATOR TASK TASK TASK SOURCE MAP PRINT FILTER OPERATOR CHAIN OPERATOR OPERATOR SUBTASK SUBTASK TASK SOURCE MAP FILTER OPERATOR CHAIN OPERATOR SUBTASK SUBTASK STREAM PARTITIONS STREAMING DATAFLOW (PARALLELIZED VIEW)
49. RUNTIME SOURCE MAP PRINT FILTER OPERATOR CHAIN OPERATOR OPERATOR SUBTASK SUBTASK TASK SOURCE MAP FILTER OPERATOR CHAIN OPERATOR SUBTASK SUBTASK STREAM PARTITIONS STREAMING DATAFLOW (PARALLELIZED VIEW) A Flink cluster has a JOB MANAGER and multiple TASK MANAGERS. Each of those is a JVM.
50. RUNTIME Each Task Manager can manage MULTIPLE THREADS executing TASKS / SUBTASKS. SOURCE MAP PRINT FILTER OPERATOR CHAIN OPERATOR OPERATOR THREAD THREAD THREAD SUBTASK SUBTASK TASK SOURCE MAP FILTER OPERATOR CHAIN OPERATOR THREAD THREAD SUBTASK SUBTASK STREAM PARTITIONS STREAMING DATAFLOW (PARALLELIZED VIEW)
51. CHECKPOINTING checkpoint barrier n checkpoint barrier n-1 checkpoint n+1 checkpoint n checkpoint n-1 Consistent, incremental snapshots of distributed data stream and operator state Based on a paper from 1985, inspired by the Chandy-Lamport-Algorithm
52. STATE OPERATOR STATE KEYED STATE Bound only to an operator Bound to an operator and key PLUGGABLE BACKEND MULTIPLE PRIMITIVES SUPPORTED GUARANTEED CONSISTENCY IN CASE OF A FAILURE
53. INNOGAMES STREAMING
54. COMPANY SNAPSHOT More than 400 employees Founded 2007 in Germany Headquarter in Hamburg +160m EUR revenue made in 2017 7 live games >30 language versions
55. I AM LEGEND OUR PORTFOLIO Simulation Strategy RPG Browser Multi-device Mobile
56. EVENT TRACKING quest build fight invite
57. EVENT TRACKING 1.000.000.000 EVENTS PER DAY
58. DATA ARCHITECTURE DATA PIPELINE DATA PLATFORM milliseconds, seconds, minutes hours, days, years
59. DATA ARCHITECTURE EVENT CLIENT EVENT CLIENT EVENT CLIENT EVENT GATEWAY EVENT BUS STREAM PROCESSING DISTRIBUTED DATA STORE DISTRIBUTED BATCH PROCESSING BI
60. DATA ARCHITECTURE EVENT CLIENT EVENT CLIENT EVENT CLIENT EVENT GATEWAY EVENT BUS STREAM PROCESSING DISTRIBUTED DATA STORE DISTRIBUTED BATCH PROCESSING BI
62. DATA ARCHITECTURE EVENT CLIENT EVENT CLIENT EVENT CLIENT EVENT GATEWAY EVENT BUS DISTRIBUTED DATA STORE DISTRIBUTED BATCH PROCESSING BI STREAM PROCESSING
63. USE CASE EVENT METRICS
64. Metrics.java stream .map(streamEvent -> new Tuple2<>(streamEvent.getEventName(), 1)) .keyBy(0) .timeWindow(Time.minutes(1)) .sum(1) .addSink(graphiteSink).setParallelism(1).name("event_counts");
65. USE CASE EVENT METRICS
66. USE CASE LOG00 MONITOR
67. KeyedStream<StreamEvent, Integer> stream = events .filter(event -> Arrays.asList("reg", "login").contains(event.getEventName())) .keyBy((KeySelector<StreamEvent, Integer>) StreamEvent::getPlayerId); Log00.java
68. Pattern<StreamEvent, StreamEvent> pattern = Pattern.<StreamEvent>begin("reg").where(new SimpleCondition<StreamEvent>() { @Override public boolean filter(StreamEvent event) { return event.getEventName().equals("reg"); } }).followedBy("login").where(new SimpleCondition<StreamEvent>() { @Override public boolean filter(StreamEvent event) { return event.getEventName().equals("login"); } }).within(Time.seconds(60)); Log00.java
69. PatternStream<StreamEvent> patternStream = CEP.pattern(stream, pattern); DataStream<Either<PatternResult, PatternResult>> patternResultStream = patternStream.select( (p, ts) -> sendTimeoutToGraphite(p, ts), p -> sendSuccessToGraphite(p) ); Log00.java
70. USE CASE LOG00 MONITOR
71. USE CASE NEAR TIME CRM (NTCRM)
72. USE CASE NTCRM EVENT BUS EVENT CLIENT EVENT GATEWAY PLAYER DATA NTCRM React to events with interstitials in less than 10 seconds
73. USE CASE NTCRM Elvenar has a trading feature that sometimes causes confusion. With NTCRM we can react to this and show more details within interstitials exactly when the player needs it.
74. JUST DO IT DEMO TIME Check it out on Github: https://github.com/prenomenon/codetalks-flinkdemo
75. GET IN TOUCH InnoGames GmbH Friesenstrasse 13 20097 Hamburg http://www.innogames.com Volker Janz Senior Software Developer Corporate Systems - Analytics
76. GET IN TOUCH @prenomenon feedback appreciated
78. Great Flink training: http://training.data-artisans.com
79. THAT’S IT FOR NOW… @prenomenon feedback appreciated
80. BACKUP / DETAILS The following slides are not part of my talk but might give the reader more insights later
81. SQUIRREL TESTS <dependency <groupId>org.apache.flink</groupId> <artifactId>flink-test-utils_2.11</artifactId> <version>1.6.1</version> </dependency>
82. WINDOWING KEYED NON-KEYED TASK 1 TASK N SOURCE TASK 1 SOURCE KEY 1 KEY N ALL DATA
83. STATE SOURCE MAP DATA SINK SOURCE MAP SUM(C,D) OFFSET OFFSET SUM(A,B) AB CD