Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kafka jako strumień i co się z tym wiąże

Avatar for Wojciech Marusarz Wojciech Marusarz
October 24, 2019
Programming
0
14

Kafka jako strumień i co się z tym wiąże

Avatar for Wojciech Marusarz

Wojciech Marusarz

October 24, 2019
Tweet

More Decks by Wojciech Marusarz

See All by Wojciech Marusarz
Kafka jako strumień i co się z tym wiąże
Avatar for Wojciech Marusarz marwoj
0
96

Other Decks in Programming

See All in Programming
GraphRAGの仕組みまるわかり
Avatar for とすり tosuri13
8
530
5つのアンチパターンから学ぶLT設計
Avatar for Narihara narihara
1
160
効率的な開発手段として VRTを活用する
Avatar for Yosuke Ishikawa ishkawa
0
130
猫と暮らす Google Nest Cam生活🐈 / WebRTC with Google Nest Cam
Avatar for Yutaro Muta yutailang0119
0
110
Modern Angular with Signals and Signal Store: New Rules for Your Architecture @enterJS Advanced Angular Day 2025
Avatar for Manfred Steyer manfredsteyer
PRO
0
210
ISUCON研修おかわり会 講義スライド
Avatar for Yuki Yata arfes0e2b3c
1
440
Azure AI Foundryではじめてのマルチエージェントワークフロー
Avatar for 瀬尾佳隆 seosoft
0
160
Google Agent Development Kit でLINE Botを作ってみた
Avatar for Kento.Yamada ymd65536
2
240
技術同人誌をMCP Serverにしてみた
Avatar for 74th(Atsushi Morimoto) 74th
1
630
GitHub Copilot and GitHub Codespaces Hands-on
Avatar for Kento.Yamada ymd65536
2
150
High-Level Programming Languages in AI Era -Human Thought and Mind-
Avatar for Hayato Ishida hayat01sh1da
PRO
0
770
ruby.wasmで多人数リアルタイム通信ゲームを作ろう
Avatar for lni_T lnit
3
460

Featured

See All Featured
Building Applications with DynamoDB
Avatar for Matt Wood mza
95
6.5k
Building Better People: How to give real-time feedback that sticks.
Avatar for Will Jessup wjessup
367
19k
The Invisible Side of Design
Avatar for Vitaly Friedman smashingmag
301
51k
Java REST API Framework Comparison - PWX 2021
Avatar for Matt Raible mraible
31
8.7k
Side Projects
Avatar for Sacha Greif sachag
455
42k
It's Worth the Effort
Avatar for 3n 3n
185
28k
StorybookのUI Testing Handbookを読んだ
Avatar for Shingo Yamazaki zakiyama
30
5.9k
Unsuck your backbone
Avatar for Amy Palamountain ammeep
671
58k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
Avatar for Rebecca Miller-Webster rmw
34
3.1k
Six Lessons from altMBA
Avatar for Skipper Chong Warson skipperchong
28
3.9k
How To Stay Up To Date on Web Technology
Avatar for Chris Coyier chriscoyier
790
250k
How to Think Like a Performance Engineer
Avatar for Harry Roberts csswizardry
25
1.7k

Transcript

  1. Kafka jako strumień i co się z tym wiąże Wojciech

    Marusarz // nexocode

  2. Kafka jako strumień i co się z tym wiąże Wojciech

    Marusarz // nexocode
  3. None

  4. Programiści...

  5. Modyfikacja zapytań http:// { "totalTime": "1855", "result": "16" }

  6. Kodowanie wartości { "totalTime": "gXe", "result": "bM" } { "totalTime":

    "1855", "result": "16" }

  7. Obfuskacja

  8. var _0xaed1 = ["\x6C\x65\x6E\x67\x74\x68","\x65\x6E\x75\x6D\x65\x72\x61\x62\x6C\x65","\x 63\x6F\x6E\x66\x69\x67\x75\x72\x61\x62\x6C\x65","\x76\x61\x6C\x75\x65","\ x77\x72\x69\x74\x61\x62\x6C\x65","\x6B\x65\x79","\x64\x65\x66\x69\x6E\x65 \x50\x72\x6F\x70\x65\x72\x74\x79", ...

  9. Uparci programiści...

  10. Przesyłanie modelu gry { "totalTime": "gXe", "result": "bM", "levels": [

    { "rectangle": "dpPTVXT3XI7p" }, { "rectangle": "4ajhYLh23FdQ" } ] }

  11. Walidacja modelu gry na serwerze { "totalTime": "750", "result": "2",

    "levels": [ { "rectangle": "[10,10,80,80]" } { "rectangle": "[0,0,100,100]" }, ] }

  12. Brace yourself, gra nadchodzi...

  13. None

  14. { "totalTime": "12478", "result": "2", "levels": [ { "rectangle": "[0,20,90,80]"

    }, { "rectangle": "[0,0,100,100]" } ] }

  15. Debug mode - ON !!! { "totalTime": "12478", "result": "3",

    "levels": [ { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,0,100,100]" } ] }

  16. Debug mode - ON !!! { "totalTime": "12478", "result": "4",

    "levels": [ { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,0,100,100]" } ] }

  17. Challenge accepted…

  18. •Przesyłanie klocków pojedynczo

  19. •Przesyłanie klocków pojedynczo •Przechowywanie klocków

  20. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry

  21. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry •Walidacja &

    wykrywanie nadużyć

  22. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry •Walidacja &

    wykrywanie nadużyć •Zapis w bazie danych

  23. Request/Response HTTP

  24. Request/Response HTTP Progress

  25. Request/Response Progress

  26. Przechowywanie klocków

  27. Przechowywanie klocków Przetwarzanie modelu gry

  28. Przechowywanie klocków Przetwarzanie modelu gry Przechowywanie wyników

  29. Czy takie rozwiązanie ma sens?

  30. Po co Apache Kafka ?

  31. Szyna danych z warstwą persystencji

  32. Szyna danych z warstwą persystencji cechująca się wysoką dostępnością oraz

    szybkością działania 99.999%

  33. Szyna danych z warstwą persystencji cechująca się wysoką dostępnością oraz

    szybkością działania
  34. None
  35. None
  36. None
  37. None
  38. None
  39. None

  40. 12 000 /s

  41. 500 /s

  42. Technikalia

  43. #1 Problem - Architektura

  44. Producent Konsument Topic

  45. Producent Konsument “blocks”

  46. Broker A Producent Konsument Topic

  47. Broker A Topic A B C D E ... A

    B C ... A B C D ... Partition_0 Partition_1 Partition_2 Producent Konsument

  48. Topic Broker C Broker B Broker A A B C

    D E ... A B C ... Partition_0 Partition_1 A B C D E ... A B C D ... Partition_0 Partition_2 Producent Konsument A B C D ... Partition_2 A B C D E ... Partition_0

  49. Topic Broker C Broker B Broker A A B C

    D E ... A B C D Partition_0 Partition_1 A B C D E ... Partition_0 Producent Konsument A B C D E ... Partition_0

  50. Broker A A B C D E ... A B

    C ... Partition_0 Partition_1 Broker B Broker C A B C ... A B C D ... Partition_1 Partition_2 A C D E ... A B C D ... Partition_0 Partition_2 B

  51. Broker A A B C D E ... A B

    C ... Partition_0 Partition_1 Broker B Broker C A B C ... A B C D ... Partition_1 Partition_2 A C D E ... A B C D ... Partition_0 Partition_2 B

  52. Broker B IP: 192.168.0.2 Broker C IP: 192.168.0.3 Broker B

    IP: 192.168.0.2 Broker A IP: 192.168.0.1 Zookeeper Broker A IP: 192.168.0.1 Broker C IP: 192.168.0.3

  53. IP: 192.168.0.1 IP: 192.168.0.3 Broker C IP: 192.168.0.3 Broker B

    IP: 192.168.0.2 Broker A IP: 192.168.0.1 Zookeeper IP: 192.168.0.2 IP: 192.168.0.3 IP: 192.168.0.1 IP: 192.168.0.2

  54. Broker A Producent Konsument TOPIC-PARTITION-ADDRESS 192.168.0.1 - blocks-partition_0 192.168.0.2 -

    blocks-partition_2 192.168.0.3 - blocks-partition_1

  55. A B C D E Partition_0 Partition_1 B C D

    A E

  56. B A E C D Partition_0 Partition_1 B C D

    A E

  57. Partition_0 Partition_1 1 2 3 4 5 1 3 5

    2 4

  58. Partition_0 Partition_1 1 3 5 2 4 1 3 5

    2 4

  59. KEY-2 KEY-1 Partition_0 Partition_1 C A E B A E

    C D B A E C D Partition_0 Partition_1 A E KEY-1 B KEY-1 D D

  60. A E Partition_0 Partition_1 E E A E E A

    Partition_0 Partition_1 A KEY-2

  61. Partition_0 Partition_1 Klucz Modulo 2 1 2 3 4 5

    1 3 5 2 4

  62. Consumer Partition_0 Partition_1 1 3 5 2 4

  63. Consumer Partition_0 Partition_1 1 3 5 2 4 1 2

    3 4 5

  64. Partition_0 Partition_1 Consumer Group Consumer Consumer 1 2 3 4

    5 1 3 5 2 4

  65. Partition_0 Partition_1 Consumer Group Consumer Consumer 1 2 3 4

    5 2 4 1 3 5
  66. None
  67. None

  68. #1 Problem - Architektura

  69. Po co Kafka Streams ?

  70. App App App Consumers Producers Stream Processors Kafka Cluster

  71. •Stream processing

  72. •Stream processing •Request/Response •Batch processing

  73. •Stream processing •Request/Response •Batch processing

  74. •Stream processing •Request/Response •Batch processing

  75. •Stream processing •Request/Response •Batch processing

  76. #2 Problem - Stream Processing

  77. Single event-processing

  78. Single event-processing Filter Map

  79. Single event-processing Nie przechowuje stanu

  80. Processing with local state

  81. Local-state Aggregate

  82. Local-state Aggregate

  83. None

  84. Local-state Aggregate

  85. Local-state Aggregate Restart

  86. topic-partition-from_begining Offset: blocks-Partition_0-3 Partition_0

  87. Local-state Aggregate

  88. None

  89. Stream Wyliczony stan Aggregate

  90. Stream Wyliczony stan Aggregate

  91. Duality of Streams and Tables

  92. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1)

  93. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1

  94. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 [email protected]

    1 [email protected] 1

  95. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 [email protected]

    1 [email protected] 1 [email protected] 2 [email protected] 1

  96. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 (”[email protected]”,1)

    (”[email protected]”,1) [email protected] 1 [email protected] 1 [email protected] 2 [email protected] 1

  97. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 (”[email protected]”,1)

    (”[email protected]”,2) (”[email protected]”,1) [email protected] 1 [email protected] 1 [email protected] 2 [email protected] 1

  98. Source code

  99. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG,

    env.getClientId()); return props;

  100. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG,

    env.getClientId()); return props; “localhost:9021,localhost:9022,localhost:9023”

  101. Przechowywanie bloków

  102. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(),

    block) blocksProducer.send(record);

  103. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(),

    block) blocksProducer.send(record);

  104. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(),

    block) blocksProducer.send(record);

  105. Topologia: przetwarzanie bloków

  106. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  107. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  108. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  109. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  110. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  111. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  112. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  113. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();

  114. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();

  115. Topologia: przetwarzanie gry

  116. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);

  117. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);

  118. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);

  119. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();

  120. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();
  121. None

  122. #2 Problem - Stream Processing

  123. System reaktywny

  124. None
  125. None
  126. None
  127. None

  128. Kafka Stream s

  129. #3 Problem - nadmiar danych

  130. Kafka Stream s

  131. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG,

    env.getClientId()); props.setProperty(StreamsConfig.NUM_STREAM_THREADS_CONFIG, "10"); return props;

  132. http

  133. http topic: “games”

  134. topic: “blocks” topic: “games” http topic: “games”

  135. topic: “blocks” topic: “games” http topic: “games” topic: “games”

  136. Testy

  137. •API Blokujące - wątek zapytania

  138. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

  139. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

    •Kafka Streams

  140. x 200 x 80

  141. •API Blokujące - wątek zapytania

  142. None
  143. None

  144. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

  145. None
  146. None

  147. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

    •Kafka Streams
  148. None
  149. None
  150. None

  151. nexoblocks https://bit.ly/2oN3L2q

  152. nexoblocks https://bit.ly/2oN3L2q

  153. Documentation Reference Guide Developers Guide Kafka Reactor Kafka Kafka Streams

    Project Reactor Any Tool... +

  154. nexoblocks https://bit.ly/2oN3L2q Dziękuję! Wojciech Marusarz // nexocode