Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Event-Driven Applications: Apache Kafka and Python
Search
FTisiot
March 24, 2021
Technology
0
950
Event-Driven Applications: Apache Kafka and Python
How to create Event-Driven Applications with Apache Kafka and Python
FTisiot
March 24, 2021
Tweet
Share
More Decks by FTisiot
See All by FTisiot
Apache Kafka JDBC Source Connector: What could go wrong?
ftisiot
0
98
Solve the Knapsack problem with recursive queries and PostgreSQL
ftisiot
0
120
Build an Open Source Data Pipeline
ftisiot
0
290
Get to know Apache Kafka with Jupyter Notebooks
ftisiot
0
710
Fix Your Strings in PostgreSQL
ftisiot
0
260
Practical Tips & Tricks for Apache Kafka Messages Integration
ftisiot
0
210
Track Database Changes with Apache Kafka
ftisiot
0
270
Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL
ftisiot
0
940
Kickstart your Kafka with Faker Data
ftisiot
0
400
Other Decks in Technology
See All in Technology
新規プロダクトでプロトタイプから正式リリースまでNext.jsで開発したリアル
kawanoriku0
1
160
複数サービスを支えるマルチテナント型Batch MLプラットフォーム
lycorptech_jp
PRO
1
820
AI開発ツールCreateがAnythingになったよ
tendasato
0
130
250905 大吉祥寺.pm 2025 前夜祭 「プログラミングに出会って20年、『今』が1番楽しい」
msykd
PRO
1
980
実践!カスタムインストラクション&スラッシュコマンド
puku0x
0
480
いま注目のAIエージェントを作ってみよう
supermarimobros
0
340
スマートファクトリーの第一歩 〜AWSマネージドサービスで 実現する予知保全と生成AI活用まで
ganota
2
280
大「個人開発サービス」時代に僕たちはどう生きるか
sotarok
20
10k
Platform開発が先行する Platform Engineeringの違和感
kintotechdev
4
580
slog.Handlerのよくある実装ミス
sakiengineer
4
420
要件定義・デザインフェーズでもAIを活用して、コミュニケーションの密度を高める
kazukihayase
0
120
未経験者・初心者に贈る!40分でわかるAndroidアプリ開発の今と大事なポイント
operando
5
730
Featured
See All Featured
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3k
Embracing the Ebb and Flow
colly
87
4.8k
GitHub's CSS Performance
jonrohan
1032
460k
Building Adaptive Systems
keathley
43
2.7k
jQuery: Nuts, Bolts and Bling
dougneiner
64
7.9k
Imperfection Machines: The Place of Print at Facebook
scottboms
268
13k
How to Ace a Technical Interview
jacobian
279
23k
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.9k
Building an army of robots
kneath
306
46k
Thoughts on Productivity
jonyablonski
70
4.8k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
920
Transcript
Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer
Advocate @ftisiot
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Event Driven Applications
@ftisiot What is an Event?
@ftisiot Why Event-driven?
@ftisiot What is Kafka? Topic A Topic B 0 1
2 3 4 0 1 2 3
@ftisiot Distributed Brokers Replication Factor 3 2
@ftisiot Distributed Replication Factor 2
@ftisiot What is a Kafka Event? <K, V> <"temp_max", 35.3>
<{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >
@ftisiot Write to Kafka Producer Topic A Hostname + Port
Authentication Encoding
@ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from
Kafka Consumer Offset 0 1 2 3
@ftisiot Pizza Demo!
@ftisiot Log Size
@ftisiot Topic Partitions
@ftisiot Distributed
@ftisiot Topic Retention Policies Time Size
@ftisiot Selecting a Partition <K, V> Same Key Same Partition
@ftisiot Ordering 2 1 3 2 1 3 Partition 0
Partition 1 Partition 1 Guaranteed per Partition
@ftisiot Scaling Out P1 P2 P3 C1 C2
@ftisiot Partitions Demo!
@ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer
Group 2
@ftisiot Consumer Groups Demo!
@ftisiot Kafka Connect Kafka Connect Kafka Connect
@ftisiot Evolve Existing Applications Kafka Connect CDC
@ftisiot Distribute Events Config File Only!
@ftisiot Kafka Connect Demo!
@ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks
@ftisiot Ask me Anything