Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Event-Driven Applications: Apache Kafka and Python
Search
FTisiot
March 24, 2021
Technology
0
950
Event-Driven Applications: Apache Kafka and Python
How to create Event-Driven Applications with Apache Kafka and Python
FTisiot
March 24, 2021
Tweet
Share
More Decks by FTisiot
See All by FTisiot
Apache Kafka JDBC Source Connector: What could go wrong?
ftisiot
0
98
Solve the Knapsack problem with recursive queries and PostgreSQL
ftisiot
0
120
Build an Open Source Data Pipeline
ftisiot
0
290
Get to know Apache Kafka with Jupyter Notebooks
ftisiot
0
700
Fix Your Strings in PostgreSQL
ftisiot
0
260
Practical Tips & Tricks for Apache Kafka Messages Integration
ftisiot
0
210
Track Database Changes with Apache Kafka
ftisiot
0
270
Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL
ftisiot
0
940
Kickstart your Kafka with Faker Data
ftisiot
0
400
Other Decks in Technology
See All in Technology
スプリントレトロスペクティブはチーム観察の宝庫? 〜チームの衝突レベルに合わせたアプローチ仮説!〜
electricsatie
1
150
MCPで変わる Amebaデザインシステム「Spindle」の開発
spindle
PRO
2
2k
Vault meets Kubernetes
mochizuki875
0
150
Snowflakeの生成AI機能を活用したデータ分析アプリの作成 〜Cortex AnalystとCortex Searchの活用とStreamlitアプリでの利用〜
nayuts
0
140
PRDの正しい使い方 ~AI時代にも効く思考・対話・成長ツールとして~
techtekt
PRO
0
310
【初心者向け】ローカルLLMの色々な動かし方まとめ
aratako
4
2.1k
【5分でわかる】セーフィー エンジニア向け会社紹介
safie_recruit
0
30k
AWSで推進するデータマネジメント
kawanago
0
800
衝突して強くなる! BLUE GIANTと アジャイルチームの共通点とは ― いきいきと活気に満ちたグルーヴあるチームを作るコツ ― / BLUE GIANT and Agile Teams
naitosatoshi
0
290
実践アプリケーション設計 ③ドメイン駆動設計
recruitengineers
PRO
13
4k
JavaScript 研修
recruitengineers
PRO
6
1.4k
Kubernetes における cgroup v2 でのOut-Of-Memory 問題の解決
pfn
PRO
0
440
Featured
See All Featured
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
44
2.5k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
Site-Speed That Sticks
csswizardry
10
800
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Imperfection Machines: The Place of Print at Facebook
scottboms
268
13k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
910
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
Docker and Python
trallard
45
3.5k
Six Lessons from altMBA
skipperchong
28
4k
Writing Fast Ruby
sferik
628
62k
YesSQL, Process and Tooling at Scale
rocio
173
14k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Transcript
Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer
Advocate @ftisiot
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Event Driven Applications
@ftisiot What is an Event?
@ftisiot Why Event-driven?
@ftisiot What is Kafka? Topic A Topic B 0 1
2 3 4 0 1 2 3
@ftisiot Distributed Brokers Replication Factor 3 2
@ftisiot Distributed Replication Factor 2
@ftisiot What is a Kafka Event? <K, V> <"temp_max", 35.3>
<{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >
@ftisiot Write to Kafka Producer Topic A Hostname + Port
Authentication Encoding
@ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from
Kafka Consumer Offset 0 1 2 3
@ftisiot Pizza Demo!
@ftisiot Log Size
@ftisiot Topic Partitions
@ftisiot Distributed
@ftisiot Topic Retention Policies Time Size
@ftisiot Selecting a Partition <K, V> Same Key Same Partition
@ftisiot Ordering 2 1 3 2 1 3 Partition 0
Partition 1 Partition 1 Guaranteed per Partition
@ftisiot Scaling Out P1 P2 P3 C1 C2
@ftisiot Partitions Demo!
@ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer
Group 2
@ftisiot Consumer Groups Demo!
@ftisiot Kafka Connect Kafka Connect Kafka Connect
@ftisiot Evolve Existing Applications Kafka Connect CDC
@ftisiot Distribute Events Config File Only!
@ftisiot Kafka Connect Demo!
@ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks
@ftisiot Ask me Anything