Kafka jako strumień i co się z tym wiąże

Stream Processing staje się coraz bardziej popularnym paradygmatem przetwarzania danych.
Na przykładzie mobilnej gry w klocki, pokażę Wam na jakie ciekawe problemy ze świata przetwarzania strumieni możemy się natkąć i jak je rozwiązać.
Opowiem czym jest Apache Kafka i Kafka Streams oraz jak przy ich pomocy tworzyć systemy reaktywne.

Transcript

1. Kafka jako strumień i co się z tym wiąże Wojciech Marusarz // nexocode
3. Programiści...
4. Modyfikacja zapytań http:// { "totalTime": "1855", "result": "16" }
5. Kodowanie wartości { "totalTime": "gXe", "result": "bM" } { "totalTime": "1855", "result": "16" }
6. Obfuskacja
7. var _0xaed1 = ["\x6C\x65\x6E\x67\x74\x68","\x65\x6E\x75\x6D\x65\x72\x61\x62\x6C\x65","\x 63\x6F\x6E\x66\x69\x67\x75\x72\x61\x62\x6C\x65","\x76\x61\x6C\x75\x65","\ x77\x72\x69\x74\x61\x62\x6C\x65","\x6B\x65\x79","\x64\x65\x66\x69\x6E\x65 \x50\x72\x6F\x70\x65\x72\x74\x79", ...
8. Uparci programiści...
9. Przesyłanie modelu gry { "totalTime": "gXe", "result": "bM", "levels": [ { "rectangle": "dpPTVXT3XI7p" }, { "rectangle": "4ajhYLh23FdQ" } ] }
10. Walidacja modelu gry na serwerze { "totalTime": "750", "result": "2", "levels": [ { "rectangle": "[10,10,80,80]" } { "rectangle": "[0,0,100,100]" }, ] }
11. Brace yourself, gra nadchodzi...
13. { "totalTime": "12478", "result": "55", "levels": [ { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,0,100,100]" } ] }
14. Debug mode - ON !!! { "totalTime": "12478", "result": "55", "levels": [ { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,0,100,100]" } ] }
15. Debug mode - ON !!! { "totalTime": "12478", "result": "55", "levels": [ { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,0,100,100]" } ] }
16. Challenge accepted…
17. •Przesyłanie klocków pojedynczo
18. •Przesyłanie klocków pojedynczo •Przechowywanie klocków
19. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry
20. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry •Walidacja & wykrywanie nadużyć
21. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry •Walidacja & wykrywanie nadużyć •Zapis w bazie danych
22. Request/Response HTTP
23. Request/Response HTTP Progress
24. Request/Response Progress
25. Przechowywanie klocków
26. Przechowywanie klocków Przetwarzanie modelu gry
27. Przechowywanie klocków Przetwarzanie modelu gry Przechowywanie wyników
28. Over engineering ?
29. Po co Apache Kafka ?
30. Szyna danych z warstwą persystencji
31. Szyna danych z warstwą persystencji cechująca się wysoką dostępnością oraz szybkością działania 99.999%
32. Szyna danych z warstwą persystencji cechująca się wysoką dostępnością oraz szybkością działania
39. 12 000 /s
40. 500 /s
41. Technikalia
42. #1 Problem - Architektura
43. Producent Konsument Topic
44. Producent Konsument “blocks”
45. Broker A Producent Konsument Topic
46. Broker A Topic A B C D E ... A B C ... A B C D ... Partition_0 Partition_1 Partition_2 Producent Konsument
47. Topic Broker C Broker B Broker A A B C D E ... A B C ... Partition_0 Partition_1 A B C D E ... A B C D ... Partition_0 Partition_2 Producent Konsument A B C D ... Partition_2 A B C D E ... Partition_0
48. Topic Broker C Broker B Broker A A B C D E ... A B C D Partition_0 Partition_1 A B C D E ... Partition_0 Producent Konsument A B C D E ... Partition_0
49. Broker A A B C D E ... A B C ... Partition_0 Partition_1 Broker B Broker C A B C ... A B C D ... Partition_1 Partition_2 A C D E ... A B C D ... Partition_0 Partition_2 B
50. Broker B IP: 192.168.0.2 Broker C IP: 192.168.0.3 Broker B IP: 192.168.0.2 Broker A IP: 192.168.0.1 Zookeeper Broker A IP: 192.168.0.1 Broker C IP: 192.168.0.3
51. IP: 192.168.0.1 IP: 192.168.0.3 Broker C IP: 192.168.0.3 Broker B IP: 192.168.0.2 Broker A IP: 192.168.0.1 Zookeeper IP: 192.168.0.2 IP: 192.168.0.3 IP: 192.168.0.1 IP: 192.168.0.2
52. Broker A Producent Konsument TOPIC-PARTITION-ADDRESS 192.168.0.1 - blocks-partition_0 192.168.0.2 - blocks-partition_2 192.168.0.3 - blocks-partition_1
53. A B C D E Partition_0 Partition_1 B C D A E
54. B A E C D Partition_0 Partition_1 B C D A E
55. Partition_0 Partition_1 1 2 3 4 5 1 3 5 2 4
56. Partition_0 Partition_1 1 3 5 2 4 1 3 5 2 4
57. KEY-3 KEY-2 Partition_0 Partition_1 C A E B A E C D B A E C D Partition_0 Partition_1 A E KEY-1 B KEY-1 D D
58. A E Partition_0 Partition_1 E E A E E A Partition_0 Partition_1 A KEY-3
59. Partition_0 Partition_1 Klucz Modulo 2 1 2 3 4 5 1 3 5 2 4
60. Consumer Partition_0 Partition_1 1 3 5 2 4
61. Consumer Partition_0 Partition_1 1 3 5 2 4 1 2 3 4 5
62. Partition_0 Partition_1 Consumer Group Consumer Consumer 1 2 3 4 5 1 3 5 2 4
63. Partition_0 Partition_1 Consumer Group Consumer Consumer 1 2 3 4 5 2 4 1 3 5
64. F G H I J Partition_0 Partition_1 A D E B C
65. F G H I J Partition_0 Partition_1 A D E B C F H J G I
66. F G H I J Partition_0 Partition_1 A D E B C F H J G I blocks-Partition_0-3 blocks-Partition_1-2 topic-partition-from_begining Offset:
70. #1 Problem - Architektura
71. Po co Kafka Streams ?
72. App App App Consumers Producers Stream Processors Kafka Cluster
73. •Stream processing
74. •Stream processing •Request/Response •Batch processing
75. •Stream processing •Request/Response •Batch processing
76. •Stream processing •Request/Response •Batch processing
77. •Stream processing •Request/Response •Batch processing
78. #2 Problem - Stream Processing
79. Single event-processing
80. Single event-processing Filter Map
81. Single event-processing Nie przechowuje stanu
82. Processing with local state
83. Local-state Aggregate
84. Local-state Aggregate
86. Local-state Aggregate
87. Local-state Aggregate Restart
88. Local-state Aggregate
90. Stream Wyliczony stan Aggregate
91. Stream Wyliczony stan Aggregate
92. Duality of Streams and Tables
93. Stream, (as record Stream) (”anna.pawlak@mail.com”,1) (”jan.kowalski@mail.com”,1) (”anna.pawlak@mail.com”,1)
94. Stream, (as record Stream) (”anna.pawlak@mail.com”,1) (”jan.kowalski@mail.com”,1) (”anna.pawlak@mail.com”,1) Anna 1
95. Stream, (as record Stream) (”anna.pawlak@mail.com”,1) (”jan.kowalski@mail.com”,1) (”anna.pawlak@mail.com”,1) Anna 1 Anna 1 Jan 1
96. Stream, (as record Stream) (”anna.pawlak@mail.com”,1) (”jan.kowalski@mail.com”,1) (”anna.pawlak@mail.com”,1) Anna 1 Anna 1 Jan 1 Anna 2 Jan 1
97. Stream, (as record Stream) (”anna.pawlak@mail.com”,1) (”jan.kowalski@mail.com”,1) (”anna.pawlak@mail.com”,1) Anna 1 Anna 1 Jan 1 Anna 2 Jan 1 (”anna.pawlak@mail.com”,1) (”jan.kowalski@mail.com”,1)
98. Stream, (as record Stream) (”anna.pawlak@mail.com”,1) (”jan.kowalski@mail.com”,1) (”anna.pawlak@mail.com”,1) Anna 1 Anna 1 Jan 1 Anna 2 Jan 1 (”anna.pawlak@mail.com”,1) (”anna.pawlak@mail.com”,2) (”jan.kowalski@mail.com”,1)
99. Source code
100. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG, env.getClientId()); return props;
101. Zapis bloków
102. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG, env.getClientId()); return props; “localhost:9021,localhost:9022,localhost:9023”
103. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(), block) blocksProducer.send(record);
104. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(), block) blocksProducer.send(record);
105. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(), block) blocksProducer.send(record);
106. Topologia: przetwarzanie bloków
107. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new, (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));
108. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new, (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));
109. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new, (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));
110. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new, (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));
111. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new, (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));
112. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new, (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));
113. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new, (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));
114. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props); kafkaStreams.start();
115. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props); kafkaStreams.start();
116. Topologia: przetwarzanie gry
117. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);
118. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);
119. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);
120. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props); kafkaStreams.start();
121. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props); kafkaStreams.start();
123. #2 Problem - Stream Processing
124. System reaktywny
129. Kafka Stream s
130. #3 Problem - nadmiar danych
131. Kafka Stream s
132. http
133. http
134. http “blocks”
135. http “blocks” “games”
136. http “blocks” “games”
138. nexoblocks https://bit.ly/2WQcHmz
139. nexoblocks https://bit.ly/2WQcHmz
140. Documentation Reference Guide Developers Guide Kafka Reactor Kafka Kafka Streams Project Reactor Any Tool... +
141. nexoblocks https://bit.ly/2WQcHmz Dziękuję! Wojciech Marusarz // nexocode