Squirrels and Elephants - The InnoGames Big Data and Streaming Infrastructure

Life doesn't happen in batches. You have to process data on time as it happens to make use of the time-value of information.

Apart from a general introduction to stream processing and Apache Flink, this presentation shows how we as a successful data-driven gaming company apply this concept with our data architecture and how we use Apache Flink to run several streaming applications.

Transcript

1. SQUIRRELS AND ELEPHANTS Big Data and Streaming at InnoGames
2. LET‘S GO ON A ROADTRIP
3. IMAGINE You are driving with your family on the backseat
4. IMAGINE There is a lot of traffic, you have to concentrate
5. IMAGINE And now, while driving, you are closing your eyes
7. HOW DO YOU FEEL?
8. WOULD YOU EVER DO THAT? NOPE
9. METAPHOR The car is your company, team or project The passengers are your colleagues
10. WOULD YOU EVER DO THAT? NOPE IN A METAPHORICAL SENSE
11. YOU HAVE TO PROCESS DATA ON TIME AS IT HAPPENS
12. BATCH PROCESSING… …might cause accidents Because…
13. LIFE DOESN’T HAPPEN IN BATCHES https://mapr.com/ebooks/streaming-architecture/chapter-01-why-event-streaming.html © Ellen Friedman, Ted Dunning
14. ICE CREAM AND GAMING?
15. SIMILARITIES MAKE US HAPPY
16. SIMILARITIES WHICH TASTES BETTER?
17. SIMILARITIES WHICH TASTES BETTER?
18. SIMILARITIES THE FIRST IMPRESSION COUNTS The moment the customer enters the shop or the player plays his first session is crucial HALO EFFECT When one trait of a person or thing is used to make an overall judgment of that person or thing
19. IN ORDER TO MAKE A POSITIVE IMPACT A RESPONSE NEEDS TO HAPPEN QUICKLY
20. TIME-VALUE OF INFORMATION
21. REAL-TIME USER REPORTS TRAFFIC GAS PRICE SPEED TRAPS
22. RESPOND TO LIFE AS IT HAPPENS
23. STREAM PROCESSING STREAMS OF DATA GPS DATA WEB INTERACTION SENSOR DATA
24. STREAM PROCESSING PROCESSING DATA IN MOTION
25. STREAM PROCESSING YOUR CODE SOURCE SINK OPERATOR
26. STREAM PROCESSING LAKE
27. STREAM PROCESSING LAKE
28. APACHE FLINK
29. APACHE FLINK Framework and distributed process engine for stateful computations on unbounded and bounded data streams
30. STREAMS TIME WINDOWS
31. EVERYTHING IS A STREAM UNBOUNDED STREAMS BOUNDED STREAMS
32. EVERYTHING IS A STREAM UNBOUNDED STREAMS BOUNDED STREAMS AKA BATCH PROCESSING
33. TIME IN STREAMING EPISODE I EPISODE II EPISODE III EPISODE IV EPISODE V EPISODE VI EPISODE VII EPISODE VIII EPISODE IX 1999 2002 2005 1977 1980 1983 2015 2017 2019 The Phantom Menace Attack of the Clones Revenge of the Sith A New Hope The Empire Strikes Back Return of the Jedi The Force Awakens The Last Jedi ? ORDERED BY EVENT TIME PROCESSING TIME
34. TIME IN STREAMING EPISODE I EPISODE II EPISODE III EPISODE IV EPISODE V EPISODE VI EPISODE VII EPISODE VIII EPISODE IX 1999 2002 2005 1977 1980 1983 2015 2017 2019 The Phantom Menace Attack of the Clones Revenge of the Sith A New Hope The Empire Strikes Back Return of the Jedi The Force Awakens The Last Jedi ? EVENT TIME ORDERED BY PROCESSING TIME
35. TUMBLING WINDOWS 9 1 3 2 6 8 1 3 9 8 4 5 9 1 3 2 6 8 1 3 9 8 4 5 15 18 26 SENSOR SUM
36. SLIDING WINDOWS 9 1 3 2 6 8 1 3 9 8 4 5 9 1 3 2 6 8 1 3 9 8 4 5 9 1 3 2 6 8 1 3 9 8 4 5 15 18 26 19 21 SENSOR SUM
37. BUILDING BLOCKS DATA SOURCE TRANSFORMATION DATA SINK
38. API BUILDING BLOCKS DATA SOURCE TRANSFORMATION DATA SINK
39. BUILDING BLOCKS SQL / TABLE API DataStream API ProcessFunction APIs (dynamic tables) (streams, windows) (events, state, time) HIGH LEVEL ANALYTICS API STREAM AND BATCH DATA PROCESSING STATEFUL EVENT- DRIVEN APPLICATIONS CONCISENESS EXPRESSIVENESS
40. LET‘S HAVE A CLOSER LOOK
41. LET‘S HAVE A CLOSER LOOK final StreamExecutionEnvironment env = getExecutionEnvironment(); final DataStreamSource<Integer> stream = env.fromElements(1, 2, 3, 4); stream .map((MapFunction<Integer, Integer>) i -> i + 2) .filter((FilterFunction<Integer>) i -> i % 2 == 0) .print(); env.execute(); DATA SOURCE TRANSFORMATION DATA SINK
42. RUNTIME YOUR FLINK APP FLINK RUNTIME D E P LOY
43. RUNTIME
44. BIG DATA AND STREAMING AT INNOGAMES
45. TEAM
46. TEAM BUSINESS INTELLIGENCE DATA ENGINEERING DATA SCIENCE OPERATIONS
47. EVENT TRACKING quest build fight invite
48. EVENT TRACKING 1.500.000.000 EVENTS PER DAY
49. DATA ARCHITECTURE DATA PIPELINE DATA PLATFORM milliseconds, seconds, minutes hours, days, years
50. DATA ARCHITECTURE SQUIRREL ELEPHANT
51. DATA ARCHITECTURE EVENT CLIENT EVENT CLIENT EVENT CLIENT EVENT GATEWAY EVENT BUS STREAM PROCESSING DISTRIBUTED DATA STORE DISTRIBUTED BATCH PROCESSING BUSINESS INTELLIGENCE
52. DATA ARCHITECTURE EVENT CLIENT EVENT CLIENT EVENT CLIENT EVENT GATEWAY EVENT BUS STREAM PROCESSING DISTRIBUTED DATA STORE DISTRIBUTED BATCH PROCESSING BUSINESS INTELLIGENCE
54. DATA ARCHITECTURE EVENT CLIENT EVENT CLIENT EVENT CLIENT EVENT GATEWAY EVENT BUS STREAM PROCESSING DISTRIBUTED DATA STORE DISTRIBUTED BATCH PROCESSING BUSINESS INTELLIGENCE STREAM PROCESSING
55. USE CASE EVENT METRICS
56. Metrics.java stream .map(streamEvent -> new Tuple2<>(streamEvent.getEventName(), 1)) .keyBy(0) .timeWindow(Time.minutes(1)) .sum(1) .addSink(graphiteSink).setParallelism(1).name("event_counts");
57. USE CASE EVENT METRICS
58. USE CASE LOG00 MONITOR
59. KeyedStream<StreamEvent, Integer> stream = events .filter(event -> Arrays.asList("reg", "login").contains(event.getEventName())) .keyBy((KeySelector<StreamEvent, Integer>) StreamEvent::getPlayerId); Log00.java
60. Pattern<StreamEvent, StreamEvent> pattern = Pattern.<StreamEvent>begin("reg").where(new SimpleCondition<StreamEvent>() { @Override public boolean filter(StreamEvent event) { return event.getEventName().equals("reg"); } }).followedBy("login").where(new SimpleCondition<StreamEvent>() { @Override public boolean filter(StreamEvent event) { return event.getEventName().equals("login"); } }).within(Time.seconds(60)); Log00.java
61. PatternStream<StreamEvent> patternStream = CEP.pattern(stream, pattern); DataStream<Either<PatternResult, PatternResult>> patternResultStream = patternStream.select( (p, ts) -> sendTimeoutToGraphite(p, ts), p -> sendSuccessToGraphite(p) ); Log00.java
62. USE CASE LOG00 MONITOR
63. USE CASE NEAR TIME CRM (NTCRM)
64. USE CASE NTCRM EVENT BUS EVENT CLIENT EVENT GATEWAY PLAYER DATA NTCRM React to events with interstitials in less than 10 seconds
65. USE CASE NTCRM Elvenar has a trading feature that sometimes causes confusion. With NTCRM we can react to this and show more details within interstitials exactly when the player needs it.
66. JUST DO IT DEMO TIME Check it out on Github: https://github.com/prenomenon/codetalks-flinkdemo
67. GET IN TOUCH InnoGames GmbH Friesenstrasse 13 20097 Hamburg https://www.innogames.com Volker Janz Senior Software Developer Corporate Systems - Analytics
68. GET IN TOUCH @prenomenon feedback appreciated
69. LIFE DOESN’T HAPPEN IN BATCHES
70. EAT ICE CREAM AND STREAM ON Great Flink training: http://training.data-artisans.com
71. NEXT UP EVENT CLIENT EVENT CLIENT EVENT CLIENT EVENT GATEWAY EVENT BUS STREAM PROCESSING DISTRIBUTED DATA STORE DISTRIBUTED BATCH PROCESSING BI BUSINESS INTELLIGENCE
72. THAT’S IT FOR NOW…
73. BACKUP / DETAILS The following slides are not part of my talk but might give the reader more insights later
74. COMPANY SNAPSHOT More than 400 employees Founded 2007 in Germany Headquarter in Hamburg +160m EUR revenue made in 2017 7 live games >30 language versions
75. I AM LEGEND OUR PORTFOLIO Simulation Strategy RPG Browser Multi-device Mobile
76. SQUIRREL TESTS <dependency <groupId>org.apache.flink</groupId> <artifactId>flink-test-utils_2.11</artifactId> <version>1.6.1</version> </dependency>
77. WINDOWING KEYED NON-KEYED TASK 1 TASK N SOURCE TASK 1 SOURCE KEY 1 KEY N ALL DATA
78. STATE SOURCE MAP DATA SINK SOURCE MAP SUM(C,D) OFFSET OFFSET SUM(A,B) AB CD
79. RABBIT HOLE
80. RUNTIME SOURCE MAP PRINT FILTER STREAMING DATAFLOW (CONDENSED VIEW) OPERATOR CHAIN OPERATOR OPERATOR TASK TASK TASK SOURCE MAP PRINT FILTER OPERATOR CHAIN OPERATOR OPERATOR SUBTASK SUBTASK TASK SOURCE MAP FILTER OPERATOR CHAIN OPERATOR SUBTASK SUBTASK STREAM PARTITIONS STREAMING DATAFLOW (PARALLELIZED VIEW)
81. RUNTIME SOURCE MAP PRINT FILTER OPERATOR CHAIN OPERATOR OPERATOR SUBTASK SUBTASK TASK SOURCE MAP FILTER OPERATOR CHAIN OPERATOR SUBTASK SUBTASK STREAM PARTITIONS STREAMING DATAFLOW (PARALLELIZED VIEW) A Flink cluster has a JOB MANAGER and multiple TASK MANAGERS. Each of those is a JVM.
82. RUNTIME Each Task Manager can manage MULTIPLE THREADS executing TASKS / SUBTASKS. SOURCE MAP PRINT FILTER OPERATOR CHAIN OPERATOR OPERATOR THREAD THREAD THREAD SUBTASK SUBTASK TASK SOURCE MAP FILTER OPERATOR CHAIN OPERATOR THREAD THREAD SUBTASK SUBTASK STREAM PARTITIONS STREAMING DATAFLOW (PARALLELIZED VIEW)
83. CHECKPOINTING checkpoint barrier n checkpoint barrier n-1 checkpoint n+1 checkpoint n checkpoint n-1 Consistent, incremental snapshots of distributed data stream and operator state Based on a paper from 1985, inspired by the Chandy-Lamport-Algorithm
84. STATE OPERATOR STATE KEYED STATE Bound only to an operator Bound to an operator and key PLUGGABLE BACKEND MULTIPLE PRIMITIVES SUPPORTED GUARANTEED CONSISTENCY IN CASE OF A FAILURE