$30 off During Our Annual Pro Sale. View Details »

Kafka jako strumień i co się z tym wiąże

Avatar for Wojciech Marusarz Wojciech Marusarz
October 24, 2019
Programming
0
15

Kafka jako strumień i co się z tym wiąże

Avatar for Wojciech Marusarz

Wojciech Marusarz

October 24, 2019
Tweet

More Decks by Wojciech Marusarz

See All by Wojciech Marusarz
Kafka jako strumień i co się z tym wiąże
Avatar for Wojciech Marusarz marwoj
0
100

Other Decks in Programming

See All in Programming
生成AI時代を勝ち抜くエンジニア組織マネジメント
Avatar for coconala_engineer coconala_engineer
0
610
宅宅自以為的浪漫：跟 AI 一起為自己辦的研討會寫一個售票系統
Avatar for 高見龍 eddie
0
530
Claude Codeの「Compacting Conversation」を体感50%減！ CLAUDE.md + 8 Skills で挑むコンテキスト管理術
Avatar for Kazuki Murahama kmurahama
1
630
Patterns of Patterns
Avatar for Denys Poltorak denyspoltorak
0
280
実は歴史的なアップデートだと思う AWS Interconnect - multicloud
Avatar for maroon1st maroon1st
0
250
AI 駆動開発ライフサイクル(AI-DLC):ソフトウェアエンジニアリングの再構築 / AI-DLC Introduction
Avatar for kanamsasa kanamasa
11
3.7k
Cell-Based Architecture
Avatar for Luram Archanjo larchanjo
0
140
認証・認可の基本を学ぼう後編
Avatar for kaza kouyuume
0
250
AtCoder Conference 2025「LLM時代のAHC」
Avatar for Yuki Imajuku imjk
2
570
脳の「省エネモード」をデバッグする ～System 1（直感）と System 2（論理）の切り替え～
Avatar for HideyukiKitao panda728
PRO
0
120
堅牢なフロントエンドテスト基盤を構築するために行った取り組み
Avatar for Shogo Fukami shogo4131
8
2.5k
GISエンジニアから見たLINKSデータ
Avatar for nokonoko1203 nokonoko1203
0
180

Featured

See All Featured
Gemini Prompt Engineering: Practical Techniques for Tangible AI Outcomes
Avatar for Mfonobong Umondia mfonobong
2
230
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1371
200k
職位にかかわらず全員がリーダーシップを発揮するチーム作り / Building a team where everyone can demonstrate leadership regardless of position
Avatar for MADOX madoxten
50
41k
WCS-LA-2024
Avatar for Leonardo Collado-Torres lcolladotor
0
390
Un-Boring Meetings
Avatar for Sebastian Deterding codingconduct
0
160
Typedesign – Prime Four
Avatar for Hannes Fritz hannesfritz
42
2.9k
Mind Mapping
Avatar for Hélio Medeiros helmedeiros
PRO
0
38
The untapped power of vector embeddings
Avatar for Frank van Dijk frankvandijk
1
1.5k
How to audit for AI Accessibility on your Front & Back End
Avatar for davetheseo davetheseo
0
120
Joys of Absence: A Defence of Solitary Play
Avatar for Sebastian Deterding codingconduct
1
260
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
31
3k
How to Get Subject Matter Experts Bought In and Actively Contributing to SEO & PR Initiatives.
Avatar for Liv Day livdayseo
0
29

Transcript

  1. Kafka jako strumień i co się z tym wiąże Wojciech

    Marusarz // nexocode

  2. Kafka jako strumień i co się z tym wiąże Wojciech

    Marusarz // nexocode
  3. None

  4. Programiści...

  5. Modyfikacja zapytań http:// { "totalTime": "1855", "result": "16" }

  6. Kodowanie wartości { "totalTime": "gXe", "result": "bM" } { "totalTime":

    "1855", "result": "16" }

  7. Obfuskacja

  8. var _0xaed1 = ["\x6C\x65\x6E\x67\x74\x68","\x65\x6E\x75\x6D\x65\x72\x61\x62\x6C\x65","\x 63\x6F\x6E\x66\x69\x67\x75\x72\x61\x62\x6C\x65","\x76\x61\x6C\x75\x65","\ x77\x72\x69\x74\x61\x62\x6C\x65","\x6B\x65\x79","\x64\x65\x66\x69\x6E\x65 \x50\x72\x6F\x70\x65\x72\x74\x79", ...

  9. Uparci programiści...

  10. Przesyłanie modelu gry { "totalTime": "gXe", "result": "bM", "levels": [

    { "rectangle": "dpPTVXT3XI7p" }, { "rectangle": "4ajhYLh23FdQ" } ] }

  11. Walidacja modelu gry na serwerze { "totalTime": "750", "result": "2",

    "levels": [ { "rectangle": "[10,10,80,80]" } { "rectangle": "[0,0,100,100]" }, ] }

  12. Brace yourself, gra nadchodzi...

  13. None

  14. { "totalTime": "12478", "result": "2", "levels": [ { "rectangle": "[0,20,90,80]"

    }, { "rectangle": "[0,0,100,100]" } ] }

  15. Debug mode - ON !!! { "totalTime": "12478", "result": "3",

    "levels": [ { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,0,100,100]" } ] }

  16. Debug mode - ON !!! { "totalTime": "12478", "result": "4",

    "levels": [ { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,0,100,100]" } ] }

  17. Challenge accepted…

  18. •Przesyłanie klocków pojedynczo

  19. •Przesyłanie klocków pojedynczo •Przechowywanie klocków

  20. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry

  21. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry •Walidacja &

    wykrywanie nadużyć

  22. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry •Walidacja &

    wykrywanie nadużyć •Zapis w bazie danych

  23. Request/Response HTTP

  24. Request/Response HTTP Progress

  25. Request/Response Progress

  26. Przechowywanie klocków

  27. Przechowywanie klocków Przetwarzanie modelu gry

  28. Przechowywanie klocków Przetwarzanie modelu gry Przechowywanie wyników

  29. Czy takie rozwiązanie ma sens?

  30. Po co Apache Kafka ?

  31. Szyna danych z warstwą persystencji

  32. Szyna danych z warstwą persystencji cechująca się wysoką dostępnością oraz

    szybkością działania 99.999%

  33. Szyna danych z warstwą persystencji cechująca się wysoką dostępnością oraz

    szybkością działania
  34. None
  35. None
  36. None
  37. None
  38. None
  39. None

  40. 12 000 /s

  41. 500 /s

  42. Technikalia

  43. #1 Problem - Architektura

  44. Producent Konsument Topic

  45. Producent Konsument “blocks”

  46. Broker A Producent Konsument Topic

  47. Broker A Topic A B C D E ... A

    B C ... A B C D ... Partition_0 Partition_1 Partition_2 Producent Konsument

  48. Topic Broker C Broker B Broker A A B C

    D E ... A B C ... Partition_0 Partition_1 A B C D E ... A B C D ... Partition_0 Partition_2 Producent Konsument A B C D ... Partition_2 A B C D E ... Partition_0

  49. Topic Broker C Broker B Broker A A B C

    D E ... A B C D Partition_0 Partition_1 A B C D E ... Partition_0 Producent Konsument A B C D E ... Partition_0

  50. Broker A A B C D E ... A B

    C ... Partition_0 Partition_1 Broker B Broker C A B C ... A B C D ... Partition_1 Partition_2 A C D E ... A B C D ... Partition_0 Partition_2 B

  51. Broker A A B C D E ... A B

    C ... Partition_0 Partition_1 Broker B Broker C A B C ... A B C D ... Partition_1 Partition_2 A C D E ... A B C D ... Partition_0 Partition_2 B

  52. Broker B IP: 192.168.0.2 Broker C IP: 192.168.0.3 Broker B

    IP: 192.168.0.2 Broker A IP: 192.168.0.1 Zookeeper Broker A IP: 192.168.0.1 Broker C IP: 192.168.0.3

  53. IP: 192.168.0.1 IP: 192.168.0.3 Broker C IP: 192.168.0.3 Broker B

    IP: 192.168.0.2 Broker A IP: 192.168.0.1 Zookeeper IP: 192.168.0.2 IP: 192.168.0.3 IP: 192.168.0.1 IP: 192.168.0.2

  54. Broker A Producent Konsument TOPIC-PARTITION-ADDRESS 192.168.0.1 - blocks-partition_0 192.168.0.2 -

    blocks-partition_2 192.168.0.3 - blocks-partition_1

  55. A B C D E Partition_0 Partition_1 B C D

    A E

  56. B A E C D Partition_0 Partition_1 B C D

    A E

  57. Partition_0 Partition_1 1 2 3 4 5 1 3 5

    2 4

  58. Partition_0 Partition_1 1 3 5 2 4 1 3 5

    2 4

  59. KEY-2 KEY-1 Partition_0 Partition_1 C A E B A E

    C D B A E C D Partition_0 Partition_1 A E KEY-1 B KEY-1 D D

  60. A E Partition_0 Partition_1 E E A E E A

    Partition_0 Partition_1 A KEY-2

  61. Partition_0 Partition_1 Klucz Modulo 2 1 2 3 4 5

    1 3 5 2 4

  62. Consumer Partition_0 Partition_1 1 3 5 2 4

  63. Consumer Partition_0 Partition_1 1 3 5 2 4 1 2

    3 4 5

  64. Partition_0 Partition_1 Consumer Group Consumer Consumer 1 2 3 4

    5 1 3 5 2 4

  65. Partition_0 Partition_1 Consumer Group Consumer Consumer 1 2 3 4

    5 2 4 1 3 5
  66. None
  67. None

  68. #1 Problem - Architektura

  69. Po co Kafka Streams ?

  70. App App App Consumers Producers Stream Processors Kafka Cluster

  71. •Stream processing

  72. •Stream processing •Request/Response •Batch processing

  73. •Stream processing •Request/Response •Batch processing

  74. •Stream processing •Request/Response •Batch processing

  75. •Stream processing •Request/Response •Batch processing

  76. #2 Problem - Stream Processing

  77. Single event-processing

  78. Single event-processing Filter Map

  79. Single event-processing Nie przechowuje stanu

  80. Processing with local state

  81. Local-state Aggregate

  82. Local-state Aggregate

  83. None

  84. Local-state Aggregate

  85. Local-state Aggregate Restart

  86. topic-partition-from_begining Offset: blocks-Partition_0-3 Partition_0

  87. Local-state Aggregate

  88. None

  89. Stream Wyliczony stan Aggregate

  90. Stream Wyliczony stan Aggregate

  91. Duality of Streams and Tables

  92. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1)

  93. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1

  94. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 [email protected]

    1 [email protected] 1

  95. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 [email protected]

    1 [email protected] 1 [email protected] 2 [email protected] 1

  96. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 (”[email protected]”,1)

    (”[email protected]”,1) [email protected] 1 [email protected] 1 [email protected] 2 [email protected] 1

  97. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 (”[email protected]”,1)

    (”[email protected]”,2) (”[email protected]”,1) [email protected] 1 [email protected] 1 [email protected] 2 [email protected] 1

  98. Source code

  99. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG,

    env.getClientId()); return props;

  100. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG,

    env.getClientId()); return props; “localhost:9021,localhost:9022,localhost:9023”

  101. Przechowywanie bloków

  102. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(),

    block) blocksProducer.send(record);

  103. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(),

    block) blocksProducer.send(record);

  104. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(),

    block) blocksProducer.send(record);

  105. Topologia: przetwarzanie bloków

  106. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  107. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  108. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  109. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  110. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  111. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  112. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  113. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();

  114. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();

  115. Topologia: przetwarzanie gry

  116. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);

  117. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);

  118. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);

  119. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();

  120. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();
  121. None

  122. #2 Problem - Stream Processing

  123. System reaktywny

  124. None
  125. None
  126. None
  127. None

  128. Kafka Stream s

  129. #3 Problem - nadmiar danych

  130. Kafka Stream s

  131. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG,

    env.getClientId()); props.setProperty(StreamsConfig.NUM_STREAM_THREADS_CONFIG, "10"); return props;

  132. http

  133. http topic: “games”

  134. topic: “blocks” topic: “games” http topic: “games”

  135. topic: “blocks” topic: “games” http topic: “games” topic: “games”

  136. Testy

  137. •API Blokujące - wątek zapytania

  138. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

  139. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

    •Kafka Streams

  140. x 200 x 80

  141. •API Blokujące - wątek zapytania

  142. None
  143. None

  144. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

  145. None
  146. None

  147. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

    •Kafka Streams
  148. None
  149. None
  150. None

  151. nexoblocks https://bit.ly/2oN3L2q

  152. nexoblocks https://bit.ly/2oN3L2q

  153. Documentation Reference Guide Developers Guide Kafka Reactor Kafka Kafka Streams

    Project Reactor Any Tool... +

  154. nexoblocks https://bit.ly/2oN3L2q Dziękuję! Wojciech Marusarz // nexocode