Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Event-Driven Applications: Apache Kafka and Python
Search
FTisiot
March 24, 2021
Technology
0
830
Event-Driven Applications: Apache Kafka and Python
How to create Event-Driven Applications with Apache Kafka and Python
FTisiot
March 24, 2021
Tweet
Share
More Decks by FTisiot
See All by FTisiot
Apache Kafka JDBC Source Connector: What could go wrong?
ftisiot
0
49
Solve the Knapsack problem with recursive queries and PostgreSQL
ftisiot
0
88
Build an Open Source Data Pipeline
ftisiot
0
250
Get to know Apache Kafka with Jupyter Notebooks
ftisiot
0
590
Fix Your Strings in PostgreSQL
ftisiot
0
220
Practical Tips & Tricks for Apache Kafka Messages Integration
ftisiot
0
160
Track Database Changes with Apache Kafka
ftisiot
0
230
Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL
ftisiot
0
840
Kickstart your Kafka with Faker Data
ftisiot
0
300
Other Decks in Technology
See All in Technology
AWSサービスメニュー開発をしていてAWSを好きだ!と感じた瞬間
toru_kubota
0
130
初中級者用如何使用backlog -VALE TUDOEDITION-
in0u
0
140
DDDにおける認可の扱いとKotlinにおける実装パターン / authorization-for-ddd-and-kotlin-implement-pattern
urmot
4
390
JBUG岡山 #6 WordCamp男木島の チームビルディング
takeshifurusato
0
150
スレットハンティングについて知っておきたいこと
hacket
0
130
クラウド利用者の「責任」をどう果たす?AWSセキュリティ対策のススメ #AWSSummit
hiashisan
0
280
頼られるのが大好きな 皆さんへ - 支援相手との期待の合わせ方、突き放し方 -/For_people_who_like_to_be_relied_on
naitosatoshi
1
290
Matterport を使ってクラスメソッド各拠点のバーチャルオフィスツアーを作成してみた
wakatsuki
0
160
「単なる OAuth 2.0 を認証に使うと、車が通れるほどのどでかいセキュリティー・ホールができる」のか検証してみた
terara
0
380
DevIO2024_レガシー運用からの脱却 -クラウド活用の実践事例とベストプラクティス-
jun2882
0
210
コンテナ・K8s研修 - 前半 コンテナ基礎・ハンズオン【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
170
OSSコミットしてZennの課題を解決した話
dyoshikawa1993
0
150
Featured
See All Featured
No one is an island. Learnings from fostering a developers community.
thoeni
17
2.8k
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.4k
Scaling GitHub
holman
458
140k
BBQ
matthewcrist
82
9k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
129
32k
Robots, Beer and Maslow
schacon
PRO
157
8.1k
How GitHub (no longer) Works
holman
305
140k
A designer walks into a library…
pauljervisheath
201
24k
The Language of Interfaces
destraynor
151
23k
Infographics Made Easy
chrislema
238
18k
Teambox: Starting and Learning
jrom
130
8.6k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
325
21k
Transcript
Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer
Advocate @ftisiot
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Event Driven Applications
@ftisiot What is an Event?
@ftisiot Why Event-driven?
@ftisiot What is Kafka? Topic A Topic B 0 1
2 3 4 0 1 2 3
@ftisiot Distributed Brokers Replication Factor 3 2
@ftisiot Distributed Replication Factor 2
@ftisiot What is a Kafka Event? <K, V> <"temp_max", 35.3>
<{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >
@ftisiot Write to Kafka Producer Topic A Hostname + Port
Authentication Encoding
@ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from
Kafka Consumer Offset 0 1 2 3
@ftisiot Pizza Demo!
@ftisiot Log Size
@ftisiot Topic Partitions
@ftisiot Distributed
@ftisiot Topic Retention Policies Time Size
@ftisiot Selecting a Partition <K, V> Same Key Same Partition
@ftisiot Ordering 2 1 3 2 1 3 Partition 0
Partition 1 Partition 1 Guaranteed per Partition
@ftisiot Scaling Out P1 P2 P3 C1 C2
@ftisiot Partitions Demo!
@ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer
Group 2
@ftisiot Consumer Groups Demo!
@ftisiot Kafka Connect Kafka Connect Kafka Connect
@ftisiot Evolve Existing Applications Kafka Connect CDC
@ftisiot Distribute Events Config File Only!
@ftisiot Kafka Connect Demo!
@ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks
@ftisiot Ask me Anything