Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kafka jako strumień i co się z tym wiąże

Avatar for Wojciech Marusarz Wojciech Marusarz
October 24, 2019
Programming
0
14

Kafka jako strumień i co się z tym wiąże

Avatar for Wojciech Marusarz

Wojciech Marusarz

October 24, 2019
Tweet

More Decks by Wojciech Marusarz

See All by Wojciech Marusarz
Kafka jako strumień i co się z tym wiąże
Avatar for Wojciech Marusarz marwoj
0
98

Other Decks in Programming

See All in Programming
今だからこそ入門する Server-Sent Events (SSE)
Avatar for NearMeの技術発表資料です nearme_tech
PRO
3
260
私の後悔をAWS DMSで解決した話
Avatar for 平目 hiramax
4
210
Amazon RDS 向けに提供されている MCP Server と仕組みを調べてみた/jawsug-okayama-2025-aurora-mcp
Avatar for Takahashi Ikki takahashiikki
1
120
デザイナーが Androidエンジニアに 挑戦してみた
Avatar for Kanon Fujita 874wokiite
0
550
チームのテスト力を鍛える
Avatar for Hiroki Iseri goyoki
3
930
Putting The Genie in the Bottle - A Crash Course on running LLMs on Android
Avatar for Iury Souza iurysza
0
140
複雑なフォームに立ち向かう Next.js の技術選定
Avatar for Takashi Machinaga macchiitaka
2
240
はじめてのMaterial3 Expressive
Avatar for yume ym223
2
900
Reading Rails 1.0 Source Code
Avatar for Masafumi Okura okuramasafumi
0
250
2025 年のコーディングエージェントの現在地とエンジニアの仕事の変化について
Avatar for azukiazusa azukiazusa1
24
12k
print("Hello, World")
Avatar for 高見龍 eddie
2
530
機能追加とリーダー業務の類似性
Avatar for Rinchoku rinchoku
2
1.3k

Featured

See All Featured
Fantastic passwords and where to find them - at NoRuKo
Avatar for Phil Nash philnash
52
3.4k
Unsuck your backbone
Avatar for Amy Palamountain ammeep
671
58k
Building Applications with DynamoDB
Avatar for Matt Wood mza
96
6.6k
The MySQL Ecosystem @ GitHub 2015
Avatar for Sam Lambert samlambert
251
13k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
Avatar for Chris Coyier chriscoyier
507
140k
How to Ace a Technical Interview
Avatar for Jacob Kaplan-Moss jacobian
279
23k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1370
200k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
64
7.9k
Building a Scalable Design System with Sketch
Avatar for Laura Van Doore lauravandoore
462
33k
How STYLIGHT went responsive
Avatar for nonsquared nonsquared
100
5.8k
Put a Button on it: Removing Barriers to Going Fast.
Avatar for kastner kastner
60
4k
A Modern Web Designer's Workflow
Avatar for Chris Coyier chriscoyier
696
190k

Transcript

  1. Kafka jako strumień i co się z tym wiąże Wojciech

    Marusarz // nexocode

  2. Kafka jako strumień i co się z tym wiąże Wojciech

    Marusarz // nexocode
  3. None

  4. Programiści...

  5. Modyfikacja zapytań http:// { "totalTime": "1855", "result": "16" }

  6. Kodowanie wartości { "totalTime": "gXe", "result": "bM" } { "totalTime":

    "1855", "result": "16" }

  7. Obfuskacja

  8. var _0xaed1 = ["\x6C\x65\x6E\x67\x74\x68","\x65\x6E\x75\x6D\x65\x72\x61\x62\x6C\x65","\x 63\x6F\x6E\x66\x69\x67\x75\x72\x61\x62\x6C\x65","\x76\x61\x6C\x75\x65","\ x77\x72\x69\x74\x61\x62\x6C\x65","\x6B\x65\x79","\x64\x65\x66\x69\x6E\x65 \x50\x72\x6F\x70\x65\x72\x74\x79", ...

  9. Uparci programiści...

  10. Przesyłanie modelu gry { "totalTime": "gXe", "result": "bM", "levels": [

    { "rectangle": "dpPTVXT3XI7p" }, { "rectangle": "4ajhYLh23FdQ" } ] }

  11. Walidacja modelu gry na serwerze { "totalTime": "750", "result": "2",

    "levels": [ { "rectangle": "[10,10,80,80]" } { "rectangle": "[0,0,100,100]" }, ] }

  12. Brace yourself, gra nadchodzi...

  13. None

  14. { "totalTime": "12478", "result": "2", "levels": [ { "rectangle": "[0,20,90,80]"

    }, { "rectangle": "[0,0,100,100]" } ] }

  15. Debug mode - ON !!! { "totalTime": "12478", "result": "3",

    "levels": [ { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,0,100,100]" } ] }

  16. Debug mode - ON !!! { "totalTime": "12478", "result": "4",

    "levels": [ { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,20,90,80]" }, { "rectangle": "[0,0,100,100]" } ] }

  17. Challenge accepted…

  18. •Przesyłanie klocków pojedynczo

  19. •Przesyłanie klocków pojedynczo •Przechowywanie klocków

  20. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry

  21. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry •Walidacja &

    wykrywanie nadużyć

  22. •Przesyłanie klocków pojedynczo •Przechowywanie klocków •Budowa modelu gry •Walidacja &

    wykrywanie nadużyć •Zapis w bazie danych

  23. Request/Response HTTP

  24. Request/Response HTTP Progress

  25. Request/Response Progress

  26. Przechowywanie klocków

  27. Przechowywanie klocków Przetwarzanie modelu gry

  28. Przechowywanie klocków Przetwarzanie modelu gry Przechowywanie wyników

  29. Czy takie rozwiązanie ma sens?

  30. Po co Apache Kafka ?

  31. Szyna danych z warstwą persystencji

  32. Szyna danych z warstwą persystencji cechująca się wysoką dostępnością oraz

    szybkością działania 99.999%

  33. Szyna danych z warstwą persystencji cechująca się wysoką dostępnością oraz

    szybkością działania
  34. None
  35. None
  36. None
  37. None
  38. None
  39. None

  40. 12 000 /s

  41. 500 /s

  42. Technikalia

  43. #1 Problem - Architektura

  44. Producent Konsument Topic

  45. Producent Konsument “blocks”

  46. Broker A Producent Konsument Topic

  47. Broker A Topic A B C D E ... A

    B C ... A B C D ... Partition_0 Partition_1 Partition_2 Producent Konsument

  48. Topic Broker C Broker B Broker A A B C

    D E ... A B C ... Partition_0 Partition_1 A B C D E ... A B C D ... Partition_0 Partition_2 Producent Konsument A B C D ... Partition_2 A B C D E ... Partition_0

  49. Topic Broker C Broker B Broker A A B C

    D E ... A B C D Partition_0 Partition_1 A B C D E ... Partition_0 Producent Konsument A B C D E ... Partition_0

  50. Broker A A B C D E ... A B

    C ... Partition_0 Partition_1 Broker B Broker C A B C ... A B C D ... Partition_1 Partition_2 A C D E ... A B C D ... Partition_0 Partition_2 B

  51. Broker A A B C D E ... A B

    C ... Partition_0 Partition_1 Broker B Broker C A B C ... A B C D ... Partition_1 Partition_2 A C D E ... A B C D ... Partition_0 Partition_2 B

  52. Broker B IP: 192.168.0.2 Broker C IP: 192.168.0.3 Broker B

    IP: 192.168.0.2 Broker A IP: 192.168.0.1 Zookeeper Broker A IP: 192.168.0.1 Broker C IP: 192.168.0.3

  53. IP: 192.168.0.1 IP: 192.168.0.3 Broker C IP: 192.168.0.3 Broker B

    IP: 192.168.0.2 Broker A IP: 192.168.0.1 Zookeeper IP: 192.168.0.2 IP: 192.168.0.3 IP: 192.168.0.1 IP: 192.168.0.2

  54. Broker A Producent Konsument TOPIC-PARTITION-ADDRESS 192.168.0.1 - blocks-partition_0 192.168.0.2 -

    blocks-partition_2 192.168.0.3 - blocks-partition_1

  55. A B C D E Partition_0 Partition_1 B C D

    A E

  56. B A E C D Partition_0 Partition_1 B C D

    A E

  57. Partition_0 Partition_1 1 2 3 4 5 1 3 5

    2 4

  58. Partition_0 Partition_1 1 3 5 2 4 1 3 5

    2 4

  59. KEY-2 KEY-1 Partition_0 Partition_1 C A E B A E

    C D B A E C D Partition_0 Partition_1 A E KEY-1 B KEY-1 D D

  60. A E Partition_0 Partition_1 E E A E E A

    Partition_0 Partition_1 A KEY-2

  61. Partition_0 Partition_1 Klucz Modulo 2 1 2 3 4 5

    1 3 5 2 4

  62. Consumer Partition_0 Partition_1 1 3 5 2 4

  63. Consumer Partition_0 Partition_1 1 3 5 2 4 1 2

    3 4 5

  64. Partition_0 Partition_1 Consumer Group Consumer Consumer 1 2 3 4

    5 1 3 5 2 4

  65. Partition_0 Partition_1 Consumer Group Consumer Consumer 1 2 3 4

    5 2 4 1 3 5
  66. None
  67. None

  68. #1 Problem - Architektura

  69. Po co Kafka Streams ?

  70. App App App Consumers Producers Stream Processors Kafka Cluster

  71. •Stream processing

  72. •Stream processing •Request/Response •Batch processing

  73. •Stream processing •Request/Response •Batch processing

  74. •Stream processing •Request/Response •Batch processing

  75. •Stream processing •Request/Response •Batch processing

  76. #2 Problem - Stream Processing

  77. Single event-processing

  78. Single event-processing Filter Map

  79. Single event-processing Nie przechowuje stanu

  80. Processing with local state

  81. Local-state Aggregate

  82. Local-state Aggregate

  83. None

  84. Local-state Aggregate

  85. Local-state Aggregate Restart

  86. topic-partition-from_begining Offset: blocks-Partition_0-3 Partition_0

  87. Local-state Aggregate

  88. None

  89. Stream Wyliczony stan Aggregate

  90. Stream Wyliczony stan Aggregate

  91. Duality of Streams and Tables

  92. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1)

  93. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1

  94. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 [email protected]

    1 [email protected] 1

  95. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 [email protected]

    1 [email protected] 1 [email protected] 2 [email protected] 1

  96. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 (”[email protected]”,1)

    (”[email protected]”,1) [email protected] 1 [email protected] 1 [email protected] 2 [email protected] 1

  97. Stream, (as record Stream) (”[email protected]”,1) (”[email protected]”,1) (”[email protected]”,1) [email protected] 1 (”[email protected]”,1)

    (”[email protected]”,2) (”[email protected]”,1) [email protected] 1 [email protected] 1 [email protected] 2 [email protected] 1

  98. Source code

  99. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG,

    env.getClientId()); return props;

  100. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG,

    env.getClientId()); return props; “localhost:9021,localhost:9022,localhost:9023”

  101. Przechowywanie bloków

  102. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(),

    block) blocksProducer.send(record);

  103. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(),

    block) blocksProducer.send(record);

  104. KafkaProducer blocksProducer = new KafkaProducer<>(props) ProducerRecord record = new ProducerRecord<>(”blocks”,block.getGameGuid(),

    block) blocksProducer.send(record);

  105. Topologia: przetwarzanie bloków

  106. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  107. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  108. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  109. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  110. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  111. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  112. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”blocks”, Consumed.with(Serdes.String(), GSerdes.BlockModel())) .groupByKey() .aggregate(GameAggregate::new,

    (gameGuid, newBlock, gameAggregate) -> { gameAggregate.add(newBlock); return gameAggregate; }, Materialized.with(Serdes.String(), CustomSerdes.GameAggregate())) .filter((gameGuid, gameAggregate) -> gameAggregate.isComplete()) .toStream() .to(”games”, Produced.with(Serdes.String(), Serdes.GameAggregate()));

  113. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();

  114. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();

  115. Topologia: przetwarzanie gry

  116. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);

  117. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);

  118. StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.stream(”games”,Consumed.with(Serdes.String(),GSerdes.GameAggregate())) .peek(gameAggregateHandler::handle);

  119. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();

  120. Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, props);

    kafkaStreams.start();
  121. None

  122. #2 Problem - Stream Processing

  123. System reaktywny

  124. None
  125. None
  126. None
  127. None

  128. Kafka Stream s

  129. #3 Problem - nadmiar danych

  130. Kafka Stream s

  131. Properties props = new Properties(); props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, env.getKafkaBrokers()); props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG, env.getAppId()); props.setProperty(StreamsConfig.CLIENT_ID_CONFIG,

    env.getClientId()); props.setProperty(StreamsConfig.NUM_STREAM_THREADS_CONFIG, "10"); return props;

  132. http

  133. http topic: “games”

  134. topic: “blocks” topic: “games” http topic: “games”

  135. topic: “blocks” topic: “games” http topic: “games” topic: “games”

  136. Testy

  137. •API Blokujące - wątek zapytania

  138. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

  139. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

    •Kafka Streams

  140. x 200 x 80

  141. •API Blokujące - wątek zapytania

  142. None
  143. None

  144. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

  145. None
  146. None

  147. •API Blokujące - wątek zapytania •API Nieblokujące - pula wątków

    •Kafka Streams
  148. None
  149. None
  150. None

  151. nexoblocks https://bit.ly/2oN3L2q

  152. nexoblocks https://bit.ly/2oN3L2q

  153. Documentation Reference Guide Developers Guide Kafka Reactor Kafka Kafka Streams

    Project Reactor Any Tool... +

  154. nexoblocks https://bit.ly/2oN3L2q Dziękuję! Wojciech Marusarz // nexocode