Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Event-Driven Applications: Apache Kafka and Python
Search
FTisiot
March 24, 2021
Technology
0
950
Event-Driven Applications: Apache Kafka and Python
How to create Event-Driven Applications with Apache Kafka and Python
FTisiot
March 24, 2021
Tweet
Share
More Decks by FTisiot
See All by FTisiot
Apache Kafka JDBC Source Connector: What could go wrong?
ftisiot
0
100
Solve the Knapsack problem with recursive queries and PostgreSQL
ftisiot
0
130
Build an Open Source Data Pipeline
ftisiot
0
290
Get to know Apache Kafka with Jupyter Notebooks
ftisiot
0
720
Fix Your Strings in PostgreSQL
ftisiot
0
260
Practical Tips & Tricks for Apache Kafka Messages Integration
ftisiot
0
220
Track Database Changes with Apache Kafka
ftisiot
0
270
Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL
ftisiot
0
950
Kickstart your Kafka with Faker Data
ftisiot
0
410
Other Decks in Technology
See All in Technology
SRE × マネジメントレイヤーが挑戦した組織・会社のオブザーバビリティ改革 ― ビジネス価値と信頼性を両立するリアルな挑戦
coconala_engineer
0
150
Introdução a Service Mesh usando o Istio
aeciopires
1
280
Linux カーネルが支えるコンテナの仕組み / LF Japan Community Days 2025 Osaka
tenforward
1
120
QA業務を変える(!?)AIを併用した不具合分析の実践
ma2ri
0
140
ゼロコード計装導入後のカスタム計装でさらに可観測性を高めよう
sansantech
PRO
1
280
オブザーバビリティが育むシステム理解と好奇心
maruloop
1
960
SCONE - 動画配信の帯域を最適化する新プロトコル
kazuho
1
360
SQLAlchemy の select(User).where(User.id =="123") を理解してみる/sqlalchemy deep dive
3l4l5
3
340
入院医療費算定業務をAIで支援する:包括医療費支払い制度とDPCコーディング (公開版)
hagino3000
0
110
デザインとエンジニアリングの架け橋を目指す OPTiMのデザインシステム「nucleus」の軌跡と広げ方
optim
0
110
ストレージエンジニアの仕事と、近年の計算機について / 第58回 情報科学若手の会
pfn
PRO
3
750
ハノーファーメッセ2025で見た生成AI活用ユースケース.pdf
hamadakoji
1
450
Featured
See All Featured
The Cult of Friendly URLs
andyhume
79
6.6k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
37
2.6k
Designing Experiences People Love
moore
142
24k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
190
55k
The Language of Interfaces
destraynor
162
25k
Optimising Largest Contentful Paint
csswizardry
37
3.5k
BBQ
matthewcrist
89
9.9k
The Cost Of JavaScript in 2023
addyosmani
55
9.1k
The Power of CSS Pseudo Elements
geoffreycrofte
80
6k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
27k
Designing for humans not robots
tammielis
254
26k
Making Projects Easy
brettharned
120
6.4k
Transcript
Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer
Advocate @ftisiot
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Event Driven Applications
@ftisiot What is an Event?
@ftisiot Why Event-driven?
@ftisiot What is Kafka? Topic A Topic B 0 1
2 3 4 0 1 2 3
@ftisiot Distributed Brokers Replication Factor 3 2
@ftisiot Distributed Replication Factor 2
@ftisiot What is a Kafka Event? <K, V> <"temp_max", 35.3>
<{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >
@ftisiot Write to Kafka Producer Topic A Hostname + Port
Authentication Encoding
@ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from
Kafka Consumer Offset 0 1 2 3
@ftisiot Pizza Demo!
@ftisiot Log Size
@ftisiot Topic Partitions
@ftisiot Distributed
@ftisiot Topic Retention Policies Time Size
@ftisiot Selecting a Partition <K, V> Same Key Same Partition
@ftisiot Ordering 2 1 3 2 1 3 Partition 0
Partition 1 Partition 1 Guaranteed per Partition
@ftisiot Scaling Out P1 P2 P3 C1 C2
@ftisiot Partitions Demo!
@ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer
Group 2
@ftisiot Consumer Groups Demo!
@ftisiot Kafka Connect Kafka Connect Kafka Connect
@ftisiot Evolve Existing Applications Kafka Connect CDC
@ftisiot Distribute Events Config File Only!
@ftisiot Kafka Connect Demo!
@ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks
@ftisiot Ask me Anything