Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Event-Driven Applications: Apache Kafka and Python
FTisiot
March 24, 2021
Technology
0
680
Event-Driven Applications: Apache Kafka and Python
How to create Event-Driven Applications with Apache Kafka and Python
FTisiot
March 24, 2021
Tweet
Share
More Decks by FTisiot
See All by FTisiot
Apache Kafka JDBC Source Connector: What could go wrong?
ftisiot
0
26
Solve the Knapsack problem with recursive queries and PostgreSQL
ftisiot
0
42
Build an Open Source Data Pipeline
ftisiot
0
170
Get to know Apache Kafka with Jupyter Notebooks
ftisiot
0
410
Fix Your Strings in PostgreSQL
ftisiot
0
200
Practical Tips & Tricks for Apache Kafka Messages Integration
ftisiot
0
140
Track Database Changes with Apache Kafka
ftisiot
0
200
Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL
ftisiot
0
720
Kickstart your Kafka with Faker Data
ftisiot
0
200
Other Decks in Technology
See All in Technology
OCI DevOps 概要 / OCI DevOps overview
oracle4engineer
PRO
0
500
Google Cloud Workflows: API automation, patterns and best practices
glaforge
0
100
NGINXENG JP#2 - 4-NGINX-エンジニアリング勉強会
hiropo20
0
120
02_プロトタイピングの進め方
kouzoukaikaku
0
530
OpenShiftのリリースノートを整理してみた
loftkun
2
390
日経電子版だけじゃない! 日経の新規Webメディアの開発 - NIKKEI Tech Talk #3
sztm
0
320
CSS Variable をもっと活用する / Kyoto.js 18
spring_raining
2
870
WebLogic Server for OCI 概要
oracle4engineer
PRO
3
880
Hatena Engineer Seminar #23 「チームとプロダクトを育てる Mackerel 開発合宿」
arthur1
0
520
OCI技術資料 : ロード・バランサー 詳細 / Load Balancer 200
ocise
2
7.2k
FlexScan HD2452Wの 後継を探して
tring
0
6.4k
Stripe / Okta Customer Identity Cloud(旧Auth0) の採用に至った理由 〜モリサワの SaaS 戦略〜
tomuro
0
130
Featured
See All Featured
Designing with Data
zakiwarfel
91
4.2k
Embracing the Ebb and Flow
colly
75
3.6k
Faster Mobile Websites
deanohume
295
29k
Web development in the modern age
philhawksworth
197
9.6k
Keith and Marios Guide to Fast Websites
keithpitt
407
21k
Build your cross-platform service in a week with App Engine
jlugia
221
17k
Fontdeck: Realign not Redesign
paulrobertlloyd
74
4.3k
Designing the Hi-DPI Web
ddemaree
273
32k
Raft: Consensus for Rubyists
vanstee
130
5.7k
Fireside Chat
paigeccino
16
1.8k
GitHub's CSS Performance
jonrohan
1020
430k
The Invisible Customer
myddelton
113
12k
Transcript
Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer
Advocate @ftisiot
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Event Driven Applications
@ftisiot What is an Event?
@ftisiot Why Event-driven?
@ftisiot What is Kafka? Topic A Topic B 0 1
2 3 4 0 1 2 3
@ftisiot Distributed Brokers Replication Factor 3 2
@ftisiot Distributed Replication Factor 2
@ftisiot What is a Kafka Event? <K, V> <"temp_max", 35.3>
<{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >
@ftisiot Write to Kafka Producer Topic A Hostname + Port
Authentication Encoding
@ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from
Kafka Consumer Offset 0 1 2 3
@ftisiot Pizza Demo!
@ftisiot Log Size
@ftisiot Topic Partitions
@ftisiot Distributed
@ftisiot Topic Retention Policies Time Size
@ftisiot Selecting a Partition <K, V> Same Key Same Partition
@ftisiot Ordering 2 1 3 2 1 3 Partition 0
Partition 1 Partition 1 Guaranteed per Partition
@ftisiot Scaling Out P1 P2 P3 C1 C2
@ftisiot Partitions Demo!
@ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer
Group 2
@ftisiot Consumer Groups Demo!
@ftisiot Kafka Connect Kafka Connect Kafka Connect
@ftisiot Evolve Existing Applications Kafka Connect CDC
@ftisiot Distribute Events Config File Only!
@ftisiot Kafka Connect Demo!
@ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks
@ftisiot Ask me Anything