Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kafka jako strumień i co się z tym wiąże

Avatar for Wojciech Marusarz Wojciech Marusarz
October 24, 2019
Programming
0
15

Kafka jako strumień i co się z tym wiąże

Avatar for Wojciech Marusarz

Wojciech Marusarz

October 24, 2019
Tweet

More Decks by Wojciech Marusarz

See All by Wojciech Marusarz
Kafka jako strumień i co się z tym wiąże
Avatar for Wojciech Marusarz marwoj
0
100

Other Decks in Programming

See All in Programming
AI によるインシデント初動調査の自動化を行う AI インシデントコマンダーを作った話
Avatar for azukiazusa azukiazusa1
1
750
今こそ知るべき耐量子計算機暗号(PQC)入門 / PQC: What You Need to Know Now
Avatar for mackey0225 mackey0225
3
390
生成AIを使ったコードレビューで定性的に品質カバー
Avatar for Chiaki Okamoto chiilog
1
280
AI & Enginnering
Avatar for codelynx codelynx
0
120
CSC307 Lecture 03
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
1
490
CSC307 Lecture 07
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
1
560
コマンドとリード間の連携に対する脅威分析フレームワーク
Avatar for Aja9tochin pandayumi
1
470
Vibe Coding - AI 驅動的軟體開發
Avatar for Micky Li mickyp100
0
180
QAフローを最適化し、品質水準を満たしながらリリースまでの期間を最短化する #RSGT2026
Avatar for shibayu36 shibayu36
2
4.4k
登壇資料を作る時に意識していること #登壇資料_findy
Avatar for konifar konifar
4
1.7k
dchart: charts from deck markup
Avatar for Anthony Starks ajstarks
3
1k
疑似コードによるプロンプト記述、どのくらい正確に実行される？
Avatar for kokuyouwind kokuyouwind
0
390

Featured

See All Featured
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
Avatar for Eileen M. Uchitelle eileencodes
141
34k
Save Time (by Creating Custom Rails Generators)
Avatar for Garrett Dimon garrettdimon
PRO
32
2.1k
StorybookのUI Testing Handbookを読んだ
Avatar for Shingo Yamazaki zakiyama
31
6.6k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
Avatar for Tammy Everts tammyeverts
11
830
The Psychology of Web Performance [Beyond Tellerrand 2023]
Avatar for Tammy Everts tammyeverts
49
3.3k
Ruling the World: When Life Gets Gamed
Avatar for Sebastian Deterding codingconduct
0
150
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
Avatar for mongodb mongodb
49
9.9k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
38
2.7k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
287
14k
GraphQLとの向き合い方2022年版
Avatar for Yosuke Kurami quramy
50
14k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
Avatar for Addy Osmani addyosmani
10
1.1k
<Decoding/> the Language of Devs - We Love SEO 2024
Avatar for Nikki Halliwell nikkihalliwell
1
130

Transcript

  1. Kafka jako strumień i co się z tym wiąże Wojciech

    Marusarz // nexocode

  2. Kafka jako strumień i co się z tym wiąże Wojciech

    Marusarz // nexocode
  3. None

  4. Programiści...

  5. Modyfikacja zapytań http:// { "totalTime": "1855", "result": "16" }

  6. Kodowanie wartości { "totalTime": "gXe", "result": "bM" } { "totalTime":

    "1855", "result": "16" }

  7. Obfuskacja

  8. var _0xaed1 = ["\x6C\x65\x6E\x67\x74\x68","\x65\x6E\x75\x6D\x65\x72\x61\x62\x6C\x65","\x 63\x6F\x6E\x66\x69\x67\x75\x72\x61\x62\x6C\x65","\x76\x61\x6C\x75\x65","\ x77\x72\x69\x74\x61\x62\x6C\x65","\x6B\x65\x79","\x64\x65\x66\x69\x6E\x65 \x50\x72\x6F\x70\x65\x72\x74\x79", ...

  9. Uparci programiści...

  10. Przesyłanie modelu gry { "totalTime": "gXe", "result": "bM", "levels": [

    { "rectangle": "dpPTVXT3XI7p" }, { "rectangle": "4ajhYLh23FdQ" } ] }

  11. Walidacja modelu gry na serwerze { "totalTime": "750", "result": "2",

    "levels": [ { "rectangle": "[10,10,80,80]" } { "rectangle": "[0,0,100,100]" }, ] }

  12. Brace yourself, gra nadchodzi...

  13. None

  14. { "totalTime": "12478", "result": "2", "levels": [ { "rectangle": "[0,20,90,80]"

    }, { "rectangle": "[0,0,100,100]" } ] }

  15. Debug mode - ON !!! { "totalTime": "12478", "result": "3",

    "levels": [ { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,0,100,100]" } ] }

  16. Debug mode - ON !!! { "totalTime": "12478", "result": "4",

    "levels": [ { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,0,100,100]" } ] }

  17. Challenge accepted…

  18. •Przesyłanie klocków pojedynczo

  19. •Przesyłanie klocków pojedynczo •Przechowywanie klocków

  20. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry

  21. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry •Walidacja &

    wykrywanie nadużyć

  22. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry •Walidacja &

    wykrywanie nadużyć •Zapis w bazie danych

  23. Request/Response HTTP

  24. Request/Response HTTP Progress

  25. Request/Response Progress

  26. Przechowywanie klocków

  27. Przechowywanie klocków Przetwarzanie modelu gry

  28. Przechowywanie klocków Przetwarzanie modelu gry Przechowywanie wyników

  29. Czy takie rozwiązanie ma sens?

  30. Po co Apache Kafka ?

  31. Szyna danych z warstwą persystencji

  32. Szyna danych z warstwą persystencji cechująca się wysoką dostępnością oraz

    szybkością działania 99.999%

  33. Szyna danych z warstwą persystencji cechująca się wysoką dostępnością oraz

    szybkością działania
  34. None
  35. None
  36. None
  37. None
  38. None
  39. None

  40. 12 000 /s

  41. 500 /s

  42. Technikalia

  43. #1 Problem - Architektura

  44. Producent Konsument Topic

  45. Producent Konsument “blocks”

  46. Broker A Producent Konsument Topic

  47. Broker A Topic A B C D E ... A

    B C ... A B C D ... Partition_0 Partition_1 Partition_2 Producent Konsument

  48. Topic Broker C Broker B Broker A A B C

    D E ... A B C ... Partition_0 Partition_1 A B C D E ... A B C D ... Partition_0 Partition_2 Producent Konsument A B C D ... Partition_2 A B C D E ... Partition_0

  49. Topic Broker C Broker B Broker A A B C

    D E ... A B C D Partition_0 Partition_1 A B C D E ... Partition_0 Producent Konsument A B C D E ... Partition_0

  50. Broker A A B C D E ... A B

    C ... Partition_0 Partition_1 Broker B Broker C A B C ... A B C D ... Partition_1 Partition_2 A C D E ... A B C D ... Partition_0 Partition_2 B

  51. Broker A A B C D E ... A B

    C ... Partition_0 Partition_1 Broker B Broker C A B C ... A B C D ... Partition_1 Partition_2 A C D E ... A B C D ... Partition_0 Partition_2 B

  52. Broker B IP: 192.168.0.2 Broker C IP: 192.168.0.3 Broker B

    IP: 192.168.0.2 Broker A IP: 192.168.0.1 Zookeeper Broker A IP: 192.168.0.1 Broker C IP: 192.168.0.3

  53. IP: 192.168.0.1 IP: 192.168.0.3 Broker C IP: 192.168.0.3 Broker B

    IP: 192.168.0.2 Broker A IP: 192.168.0.1 Zookeeper IP: 192.168.0.2 IP: 192.168.0.3 IP: 192.168.0.1 IP: 192.168.0.2

  54. Broker A Producent Konsument TOPIC-PARTITION-ADDRESS 192.168.0.1 - blocks-partition_0 192.168.0.2 -

    blocks-partition_2 192.168.0.3 - blocks-partition_1

  55. A B C D E Partition_0 Partition_1 B C D

    A E

  56. B A E C D Partition_0 Partition_1 B C D

    A E

  57. Partition_0 Partition_1 1 2 3 4 5 1 3 5

    2 4

  58. Partition_0 Partition_1 1 3 5 2 4 1 3 5

    2 4

  59. KEY-2 KEY-1 Partition_0 Partition_1 C A E B A E

    C D B A E C D Partition_0 Partition_1 A E KEY-1 B KEY-1 D D

  60. A E Partition_0 Partition_1 E E A E E A

    Partition_0 Partition_1 A KEY-2

  61. Partition_0 Partition_1 Klucz Modulo 2 1 2 3 4 5

    1 3 5 2 4

  62. Consumer Partition_0 Partition_1 1 3 5 2 4

  63. Consumer Partition_0 Partition_1 1 3 5 2 4 1 2

    3 4 5

  64. Partition_0 Partition_1 Consumer Group Consumer Consumer 1 2 3 4

    5 1 3 5 2 4

  65. Partition_0 Partition_1 Consumer Group Consumer Consumer 1 2 3 4

    5 2 4 1 3 5
  66. None
  67. None

  68. #1 Problem - Architektura

  69. Po co Kafka Streams ?

  70. App App App Consumers Producers Stream Processors Kafka Cluster

  71. •Stream processing

  72. •Stream processing •Request/Response •Batch processing

  73. •Stream processing •Request/Response •Batch processing

  74. •Stream processing •Request/Response •Batch processing

  75. •Stream processing •Request/Response •Batch processing

  76. #2 Problem - Stream Processing

  77. Single event-processing

  78. Single event-processing Filter Map

  79. Single event-processing Nie przechowuje stanu

  80. Processing with local state

  81. Local-state Aggregate

  82. Local-state Aggregate

  83. None

  84. Local-state Aggregate

  85. Local-state Aggregate Restart

  86. topic-partition-from_begining Offset: blocks-Partition_0-3 Partition_0

  87. Local-state Aggregate

  88. None

  89. Stream Wyliczony stan Aggregate

  90. Stream Wyliczony stan Aggregate

  91. Duality of Streams and Tables

  92. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1)

  93. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1

  94. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 [email protected]

    1 [email protected] 1

  95. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 [email protected]

    1 [email protected] 1 [email protected] 2 [email protected] 1

  96. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 (”[email protected]”,1)

    (”[email protected]”,1) [email protected] 1 [email protected] 1 [email protected] 2 [email protected] 1

  97. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 (”[email protected]”,1)

    (”[email protected]”,2) (”[email protected]”,1) [email protected] 1 [email protected] 1 [email protected] 2 [email protected] 1

  98. Source code

  99. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG,

    env.getClientId()); return props;

  100. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG,

    env.getClientId()); return props; “localhost:9021,localhost:9022,localhost:9023”

  101. Przechowywanie bloków

  102. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(),

    block) blocksProducer.send(record);

  103. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(),

    block) blocksProducer.send(record);

  104. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(),

    block) blocksProducer.send(record);

  105. Topologia: przetwarzanie bloków

  106. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  107. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  108. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  109. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  110. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  111. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  112. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  113. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();

  114. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();

  115. Topologia: przetwarzanie gry

  116. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);

  117. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);

  118. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);

  119. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();

  120. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();
  121. None

  122. #2 Problem - Stream Processing

  123. System reaktywny

  124. None
  125. None
  126. None
  127. None

  128. Kafka Stream s

  129. #3 Problem - nadmiar danych

  130. Kafka Stream s

  131. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG,

    env.getClientId()); props.setProperty(StreamsConfig.NUM_STREAM_THREADS_CONFIG, "10"); return props;

  132. http

  133. http topic: “games”

  134. topic: “blocks” topic: “games” http topic: “games”

  135. topic: “blocks” topic: “games” http topic: “games” topic: “games”

  136. Testy

  137. •API Blokujące - wątek zapytania

  138. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

  139. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

    •Kafka Streams

  140. x 200 x 80

  141. •API Blokujące - wątek zapytania

  142. None
  143. None

  144. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

  145. None
  146. None

  147. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

    •Kafka Streams
  148. None
  149. None
  150. None

  151. nexoblocks https://bit.ly/2oN3L2q

  152. nexoblocks https://bit.ly/2oN3L2q

  153. Documentation Reference Guide Developers Guide Kafka Reactor Kafka Kafka Streams

    Project Reactor Any Tool... +

  154. nexoblocks https://bit.ly/2oN3L2q Dziękuję! Wojciech Marusarz // nexocode