Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Event-Driven Applications: Apache Kafka and Python
Search
FTisiot
March 24, 2021
Technology
0
800
Event-Driven Applications: Apache Kafka and Python
How to create Event-Driven Applications with Apache Kafka and Python
FTisiot
March 24, 2021
Tweet
Share
More Decks by FTisiot
See All by FTisiot
Apache Kafka JDBC Source Connector: What could go wrong?
ftisiot
0
44
Solve the Knapsack problem with recursive queries and PostgreSQL
ftisiot
0
77
Build an Open Source Data Pipeline
ftisiot
0
240
Get to know Apache Kafka with Jupyter Notebooks
ftisiot
0
550
Fix Your Strings in PostgreSQL
ftisiot
0
220
Practical Tips & Tricks for Apache Kafka Messages Integration
ftisiot
0
160
Track Database Changes with Apache Kafka
ftisiot
0
220
Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL
ftisiot
0
820
Kickstart your Kafka with Faker Data
ftisiot
0
270
Other Decks in Technology
See All in Technology
Next'24 事例セッションの紹介とクラウド資格を活用したキャリア形成について語りMuscle
yasumuusan
1
320
元インフラエンジニアに成る / Human Resources to Human Relations
bobtani
3
760
The CloudCompare project by Dr. Daniel Girardeau-Montaut
kentaitakura
0
500
TransitGatewayの基礎
toru_kubota
0
230
Janus
bkuhlmann
1
490
0→1開発における技術選定において一番大切なこと
bicstone
1
320
開発生産性向上サービスを作るFindyが自分たちで開発生産性を爆上げした組織づくりの歩み / Findy's path to boosting its own development productivity 2024-04-17
ma3tk
3
310
小さな開発会社がWebサービスを作る理由
polidog
PRO
1
150
AWS を使う上で知っておきたいオンプレミス知識/aws-on-premise-essentials
emiki
1
4.2k
ここが嬉しいABAC ここが辛いよABAC #再解説+補足編
masahirokawahara
0
210
オーナーシップを持つ領域を明確にする
konifar
11
2.2k
Databricks:『生成AI World Cup』のご案内
databricksjapan
2
140
Featured
See All Featured
Code Reviewing Like a Champion
maltzj
513
39k
BBQ
matthewcrist
80
8.7k
A designer walks into a library…
pauljervisheath
199
23k
jQuery: Nuts, Bolts and Bling
dougneiner
59
7.1k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
226
51k
Infographics Made Easy
chrislema
237
18k
Making Projects Easy
brettharned
108
5.5k
5 minutes of I Can Smell Your CMS
philhawksworth
199
19k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
9
8.3k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
5
1.5k
From Idea to $5000 a Month in 5 Months
shpigford
377
45k
Fantastic passwords and where to find them - at NoRuKo
philnash
36
2.5k
Transcript
Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer
Advocate @ftisiot
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Event Driven Applications
@ftisiot What is an Event?
@ftisiot Why Event-driven?
@ftisiot What is Kafka? Topic A Topic B 0 1
2 3 4 0 1 2 3
@ftisiot Distributed Brokers Replication Factor 3 2
@ftisiot Distributed Replication Factor 2
@ftisiot What is a Kafka Event? <K, V> <"temp_max", 35.3>
<{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >
@ftisiot Write to Kafka Producer Topic A Hostname + Port
Authentication Encoding
@ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from
Kafka Consumer Offset 0 1 2 3
@ftisiot Pizza Demo!
@ftisiot Log Size
@ftisiot Topic Partitions
@ftisiot Distributed
@ftisiot Topic Retention Policies Time Size
@ftisiot Selecting a Partition <K, V> Same Key Same Partition
@ftisiot Ordering 2 1 3 2 1 3 Partition 0
Partition 1 Partition 1 Guaranteed per Partition
@ftisiot Scaling Out P1 P2 P3 C1 C2
@ftisiot Partitions Demo!
@ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer
Group 2
@ftisiot Consumer Groups Demo!
@ftisiot Kafka Connect Kafka Connect Kafka Connect
@ftisiot Evolve Existing Applications Kafka Connect CDC
@ftisiot Distribute Events Config File Only!
@ftisiot Kafka Connect Demo!
@ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks
@ftisiot Ask me Anything