$30 off During Our Annual Pro Sale. View Details »

Kafka jako strumień i co się z tym wiąże

Avatar for Wojciech Marusarz Wojciech Marusarz
October 24, 2019
Programming
0
15

Kafka jako strumień i co się z tym wiąże

Avatar for Wojciech Marusarz

Wojciech Marusarz

October 24, 2019
Tweet

More Decks by Wojciech Marusarz

See All by Wojciech Marusarz
Kafka jako strumień i co się z tym wiąże
Avatar for Wojciech Marusarz marwoj
0
100

Other Decks in Programming

See All in Programming
ゲームの物理 剛体編
Avatar for Fadis fadis
0
370
AIコーディングエージェント（NotebookLM）
Avatar for kondai kondai24
0
220
GISエンジニアから見たLINKSデータ
Avatar for nokonoko1203 nokonoko1203
0
180
マスタデータ問題、マイクロサービスでどう解くか
Avatar for Kato Keisuke kts
0
120
Giselleで作るAI QAアシスタント 〜 Pull Requestレビューに継続的QAを
Avatar for Tadashi Shigeoka codenote
0
290
re:Invent 2025 のイケてるサービスを紹介する
Avatar for maroon1st maroon1st
0
150
チームをチームにするEM
Avatar for hitode909 hitode909
0
370
Graviton と Nitro と私
Avatar for maroon1st maroon1st
0
130
公共交通オープンデータ × モバイルUX 複雑な運行情報を 『直感』に変換する技術
Avatar for Tsubasa SEKIGUCHI tinykitten
PRO
0
160
AI 駆動開発ライフサイクル(AI-DLC):ソフトウェアエンジニアリングの再構築 / AI-DLC Introduction
Avatar for kanamsasa kanamasa
11
3.7k
Canon EOS R50 V と R5 Mark II 購入でみえてきた最近のデジイチ VR180 事情、そして VR180 静止画に活路を見出すまで
Avatar for kazuhiro hara karad
0
130
AIエージェントを活かすPM術 AI駆動開発の現場から
Avatar for gyuta gyuta
0
470

Featured

See All Featured
How to audit for AI Accessibility on your Front & Back End
Avatar for davetheseo davetheseo
0
120
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
Avatar for Eileen M. Uchitelle eileencodes
141
34k
My Coaching Mixtape
Avatar for mlcsv mlcsv
0
13
Keith and Marios Guide to Fast Websites
Avatar for Keith Pitt keithpitt
413
23k
KATA
Avatar for Megan Lloyd mclloyd
PRO
33
15k
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
333
24k
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
58
6.2k
Design in an AI World
Avatar for tapps tapps
0
98
Building a Modern Day 
E-commerce SEO Strategy
Avatar for Aleyda Solis aleyda
45
8.4k
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
54
7.9k
Technical Leadership for Architectural Decision Making
Avatar for Kenny Baas-Schwegler baasie
0
180
A Soul's Torment
Avatar for Nat Ma seathinner
1
2k

Transcript

  1. Kafka jako strumień i co się z tym wiąże Wojciech

    Marusarz // nexocode

  2. Kafka jako strumień i co się z tym wiąże Wojciech

    Marusarz // nexocode
  3. None

  4. Programiści...

  5. Modyfikacja zapytań http:// { "totalTime": "1855", "result": "16" }

  6. Kodowanie wartości { "totalTime": "gXe", "result": "bM" } { "totalTime":

    "1855", "result": "16" }

  7. Obfuskacja

  8. var _0xaed1 = ["\x6C\x65\x6E\x67\x74\x68","\x65\x6E\x75\x6D\x65\x72\x61\x62\x6C\x65","\x 63\x6F\x6E\x66\x69\x67\x75\x72\x61\x62\x6C\x65","\x76\x61\x6C\x75\x65","\ x77\x72\x69\x74\x61\x62\x6C\x65","\x6B\x65\x79","\x64\x65\x66\x69\x6E\x65 \x50\x72\x6F\x70\x65\x72\x74\x79", ...

  9. Uparci programiści...

  10. Przesyłanie modelu gry { "totalTime": "gXe", "result": "bM", "levels": [

    { "rectangle": "dpPTVXT3XI7p" }, { "rectangle": "4ajhYLh23FdQ" } ] }

  11. Walidacja modelu gry na serwerze { "totalTime": "750", "result": "2",

    "levels": [ { "rectangle": "[10,10,80,80]" } { "rectangle": "[0,0,100,100]" }, ] }

  12. Brace yourself, gra nadchodzi...

  13. None

  14. { "totalTime": "12478", "result": "2", "levels": [ { "rectangle": "[0,20,90,80]"

    }, { "rectangle": "[0,0,100,100]" } ] }

  15. Debug mode - ON !!! { "totalTime": "12478", "result": "3",

    "levels": [ { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,0,100,100]" } ] }

  16. Debug mode - ON !!! { "totalTime": "12478", "result": "4",

    "levels": [ { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,0,100,100]" } ] }

  17. Challenge accepted…

  18. •Przesyłanie klocków pojedynczo

  19. •Przesyłanie klocków pojedynczo •Przechowywanie klocków

  20. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry

  21. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry •Walidacja &

    wykrywanie nadużyć

  22. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry •Walidacja &

    wykrywanie nadużyć •Zapis w bazie danych

  23. Request/Response HTTP

  24. Request/Response HTTP Progress

  25. Request/Response Progress

  26. Przechowywanie klocków

  27. Przechowywanie klocków Przetwarzanie modelu gry

  28. Przechowywanie klocków Przetwarzanie modelu gry Przechowywanie wyników

  29. Czy takie rozwiązanie ma sens?

  30. Po co Apache Kafka ?

  31. Szyna danych z warstwą persystencji

  32. Szyna danych z warstwą persystencji cechująca się wysoką dostępnością oraz

    szybkością działania 99.999%

  33. Szyna danych z warstwą persystencji cechująca się wysoką dostępnością oraz

    szybkością działania
  34. None
  35. None
  36. None
  37. None
  38. None
  39. None

  40. 12 000 /s

  41. 500 /s

  42. Technikalia

  43. #1 Problem - Architektura

  44. Producent Konsument Topic

  45. Producent Konsument “blocks”

  46. Broker A Producent Konsument Topic

  47. Broker A Topic A B C D E ... A

    B C ... A B C D ... Partition_0 Partition_1 Partition_2 Producent Konsument

  48. Topic Broker C Broker B Broker A A B C

    D E ... A B C ... Partition_0 Partition_1 A B C D E ... A B C D ... Partition_0 Partition_2 Producent Konsument A B C D ... Partition_2 A B C D E ... Partition_0

  49. Topic Broker C Broker B Broker A A B C

    D E ... A B C D Partition_0 Partition_1 A B C D E ... Partition_0 Producent Konsument A B C D E ... Partition_0

  50. Broker A A B C D E ... A B

    C ... Partition_0 Partition_1 Broker B Broker C A B C ... A B C D ... Partition_1 Partition_2 A C D E ... A B C D ... Partition_0 Partition_2 B

  51. Broker A A B C D E ... A B

    C ... Partition_0 Partition_1 Broker B Broker C A B C ... A B C D ... Partition_1 Partition_2 A C D E ... A B C D ... Partition_0 Partition_2 B

  52. Broker B IP: 192.168.0.2 Broker C IP: 192.168.0.3 Broker B

    IP: 192.168.0.2 Broker A IP: 192.168.0.1 Zookeeper Broker A IP: 192.168.0.1 Broker C IP: 192.168.0.3

  53. IP: 192.168.0.1 IP: 192.168.0.3 Broker C IP: 192.168.0.3 Broker B

    IP: 192.168.0.2 Broker A IP: 192.168.0.1 Zookeeper IP: 192.168.0.2 IP: 192.168.0.3 IP: 192.168.0.1 IP: 192.168.0.2

  54. Broker A Producent Konsument TOPIC-PARTITION-ADDRESS 192.168.0.1 - blocks-partition_0 192.168.0.2 -

    blocks-partition_2 192.168.0.3 - blocks-partition_1

  55. A B C D E Partition_0 Partition_1 B C D

    A E

  56. B A E C D Partition_0 Partition_1 B C D

    A E

  57. Partition_0 Partition_1 1 2 3 4 5 1 3 5

    2 4

  58. Partition_0 Partition_1 1 3 5 2 4 1 3 5

    2 4

  59. KEY-2 KEY-1 Partition_0 Partition_1 C A E B A E

    C D B A E C D Partition_0 Partition_1 A E KEY-1 B KEY-1 D D

  60. A E Partition_0 Partition_1 E E A E E A

    Partition_0 Partition_1 A KEY-2

  61. Partition_0 Partition_1 Klucz Modulo 2 1 2 3 4 5

    1 3 5 2 4

  62. Consumer Partition_0 Partition_1 1 3 5 2 4

  63. Consumer Partition_0 Partition_1 1 3 5 2 4 1 2

    3 4 5

  64. Partition_0 Partition_1 Consumer Group Consumer Consumer 1 2 3 4

    5 1 3 5 2 4

  65. Partition_0 Partition_1 Consumer Group Consumer Consumer 1 2 3 4

    5 2 4 1 3 5
  66. None
  67. None

  68. #1 Problem - Architektura

  69. Po co Kafka Streams ?

  70. App App App Consumers Producers Stream Processors Kafka Cluster

  71. •Stream processing

  72. •Stream processing •Request/Response •Batch processing

  73. •Stream processing •Request/Response •Batch processing

  74. •Stream processing •Request/Response •Batch processing

  75. •Stream processing •Request/Response •Batch processing

  76. #2 Problem - Stream Processing

  77. Single event-processing

  78. Single event-processing Filter Map

  79. Single event-processing Nie przechowuje stanu

  80. Processing with local state

  81. Local-state Aggregate

  82. Local-state Aggregate

  83. None

  84. Local-state Aggregate

  85. Local-state Aggregate Restart

  86. topic-partition-from_begining Offset: blocks-Partition_0-3 Partition_0

  87. Local-state Aggregate

  88. None

  89. Stream Wyliczony stan Aggregate

  90. Stream Wyliczony stan Aggregate

  91. Duality of Streams and Tables

  92. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1)

  93. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1

  94. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 [email protected]

    1 [email protected] 1

  95. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 [email protected]

    1 [email protected] 1 [email protected] 2 [email protected] 1

  96. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 (”[email protected]”,1)

    (”[email protected]”,1) [email protected] 1 [email protected] 1 [email protected] 2 [email protected] 1

  97. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 (”[email protected]”,1)

    (”[email protected]”,2) (”[email protected]”,1) [email protected] 1 [email protected] 1 [email protected] 2 [email protected] 1

  98. Source code

  99. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG,

    env.getClientId()); return props;

  100. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG,

    env.getClientId()); return props; “localhost:9021,localhost:9022,localhost:9023”

  101. Przechowywanie bloków

  102. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(),

    block) blocksProducer.send(record);

  103. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(),

    block) blocksProducer.send(record);

  104. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(),

    block) blocksProducer.send(record);

  105. Topologia: przetwarzanie bloków

  106. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  107. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  108. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  109. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  110. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  111. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  112. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  113. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();

  114. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();

  115. Topologia: przetwarzanie gry

  116. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);

  117. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);

  118. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);

  119. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();

  120. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();
  121. None

  122. #2 Problem - Stream Processing

  123. System reaktywny

  124. None
  125. None
  126. None
  127. None

  128. Kafka Stream s

  129. #3 Problem - nadmiar danych

  130. Kafka Stream s

  131. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG,

    env.getClientId()); props.setProperty(StreamsConfig.NUM_STREAM_THREADS_CONFIG, "10"); return props;

  132. http

  133. http topic: “games”

  134. topic: “blocks” topic: “games” http topic: “games”

  135. topic: “blocks” topic: “games” http topic: “games” topic: “games”

  136. Testy

  137. •API Blokujące - wątek zapytania

  138. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

  139. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

    •Kafka Streams

  140. x 200 x 80

  141. •API Blokujące - wątek zapytania

  142. None
  143. None

  144. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

  145. None
  146. None

  147. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

    •Kafka Streams
  148. None
  149. None
  150. None

  151. nexoblocks https://bit.ly/2oN3L2q

  152. nexoblocks https://bit.ly/2oN3L2q

  153. Documentation Reference Guide Developers Guide Kafka Reactor Kafka Kafka Streams

    Project Reactor Any Tool... +

  154. nexoblocks https://bit.ly/2oN3L2q Dziękuję! Wojciech Marusarz // nexocode