Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Event-Driven Applications: Apache Kafka and Python
Search
FTisiot
March 24, 2021
Technology
0
900
Event-Driven Applications: Apache Kafka and Python
How to create Event-Driven Applications with Apache Kafka and Python
FTisiot
March 24, 2021
Tweet
Share
More Decks by FTisiot
See All by FTisiot
Apache Kafka JDBC Source Connector: What could go wrong?
ftisiot
0
73
Solve the Knapsack problem with recursive queries and PostgreSQL
ftisiot
0
110
Build an Open Source Data Pipeline
ftisiot
0
270
Get to know Apache Kafka with Jupyter Notebooks
ftisiot
0
670
Fix Your Strings in PostgreSQL
ftisiot
0
240
Practical Tips & Tricks for Apache Kafka Messages Integration
ftisiot
0
180
Track Database Changes with Apache Kafka
ftisiot
0
250
Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL
ftisiot
0
910
Kickstart your Kafka with Faker Data
ftisiot
0
360
Other Decks in Technology
See All in Technology
データベースの負荷を紐解く/untangle-the-database-load
emiki
2
540
自分だけの仮想クラスタを高速かつ効率的に作る kubefork
donkomura
0
110
OPENLOGI Company Profile
hr01
0
60k
開発者のための FinOps/FinOps for Engineers
oracle4engineer
PRO
2
210
Exadata Database Service on Cloud@Customer セキュリティ、ネットワーク、および管理について
oracle4engineer
PRO
2
1.5k
AI Agent時代なのでAWSのLLMs.txtが欲しい!
watany
3
340
30→150人のエンジニア組織拡大に伴うアジャイル文化を醸成する役割と取り組みの変化
nagata03
0
200
[OpsJAWS Meetup33 AIOps] Amazon Bedrockガードレールで守る安全なAI運用
akiratameto
1
120
Global Databaseで実現するマルチリージョン自動切替とBlue/Greenデプロイ
j2yano
0
140
JavaにおけるNull非許容性
skrb
2
2.7k
目標と時間軸 〜ベイビーステップでケイパビリティを高めよう〜
kakehashi
PRO
8
860
入門 PEAK Threat Hunting @SECCON
odorusatoshi
0
170
Featured
See All Featured
Bash Introduction
62gerente
611
210k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
30
2.2k
The Cost Of JavaScript in 2023
addyosmani
47
7.4k
Fashionably flexible responsive web design (full day workshop)
malarkey
406
66k
How STYLIGHT went responsive
nonsquared
99
5.4k
The Power of CSS Pseudo Elements
geoffreycrofte
75
5.5k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
227
22k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
GraphQLの誤解/rethinking-graphql
sonatard
69
10k
The Art of Programming - Codeland 2020
erikaheidi
53
13k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
29
1k
Transcript
Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer
Advocate @ftisiot
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Event Driven Applications
@ftisiot What is an Event?
@ftisiot Why Event-driven?
@ftisiot What is Kafka? Topic A Topic B 0 1
2 3 4 0 1 2 3
@ftisiot Distributed Brokers Replication Factor 3 2
@ftisiot Distributed Replication Factor 2
@ftisiot What is a Kafka Event? <K, V> <"temp_max", 35.3>
<{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >
@ftisiot Write to Kafka Producer Topic A Hostname + Port
Authentication Encoding
@ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from
Kafka Consumer Offset 0 1 2 3
@ftisiot Pizza Demo!
@ftisiot Log Size
@ftisiot Topic Partitions
@ftisiot Distributed
@ftisiot Topic Retention Policies Time Size
@ftisiot Selecting a Partition <K, V> Same Key Same Partition
@ftisiot Ordering 2 1 3 2 1 3 Partition 0
Partition 1 Partition 1 Guaranteed per Partition
@ftisiot Scaling Out P1 P2 P3 C1 C2
@ftisiot Partitions Demo!
@ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer
Group 2
@ftisiot Consumer Groups Demo!
@ftisiot Kafka Connect Kafka Connect Kafka Connect
@ftisiot Evolve Existing Applications Kafka Connect CDC
@ftisiot Distribute Events Config File Only!
@ftisiot Kafka Connect Demo!
@ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks
@ftisiot Ask me Anything