Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Event-Driven Applications: Apache Kafka and Python
Search
FTisiot
March 24, 2021
Technology
0
900
Event-Driven Applications: Apache Kafka and Python
How to create Event-Driven Applications with Apache Kafka and Python
FTisiot
March 24, 2021
Tweet
Share
More Decks by FTisiot
See All by FTisiot
Apache Kafka JDBC Source Connector: What could go wrong?
ftisiot
0
68
Solve the Knapsack problem with recursive queries and PostgreSQL
ftisiot
0
100
Build an Open Source Data Pipeline
ftisiot
0
270
Get to know Apache Kafka with Jupyter Notebooks
ftisiot
0
670
Fix Your Strings in PostgreSQL
ftisiot
0
240
Practical Tips & Tricks for Apache Kafka Messages Integration
ftisiot
0
180
Track Database Changes with Apache Kafka
ftisiot
0
250
Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL
ftisiot
0
900
Kickstart your Kafka with Faker Data
ftisiot
0
360
Other Decks in Technology
See All in Technology
なぜ私は自分が使わないサービスを作るのか? / Why would I create a service that I would not use?
aiandrox
0
700
2/18/25: Java meets AI: Build LLM-Powered Apps with LangChain4j
edeandrea
PRO
0
100
RECRUIT TECH CONFERENCE 2025 プレイベント【高橋】
recruitengineers
PRO
0
150
TAMとre:Capセキュリティ編 〜拡張脅威検出デモを添えて〜
fujiihda
2
230
Larkご案内資料
customercloud
PRO
0
650
君も受託系GISエンジニアにならないか
sudataka
2
420
白金鉱業Meetup Vol.17_あるデータサイエンティストのデータマネジメントとの向き合い方
brainpadpr
5
600
開発組織のための セキュアコーディング研修の始め方
flatt_security
3
2k
速くて安いWebサイトを作る
nishiharatsubasa
10
12k
運用しているアプリケーションのDBのリプレイスをやってみた
miura55
1
680
The Future of SEO: The Impact of AI on Search
badams
0
180
リーダブルテストコード 〜メンテナンスしやすい テストコードを作成する方法を考える〜 #DevSumi #DevSumiB / Readable test code
nihonbuson
11
7.1k
Featured
See All Featured
Building Flexible Design Systems
yeseniaperezcruz
328
38k
The Invisible Side of Design
smashingmag
299
50k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
12
960
What's in a price? How to price your products and services
michaelherold
244
12k
The World Runs on Bad Software
bkeepers
PRO
67
11k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
160
15k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
40
2k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.4k
Raft: Consensus for Rubyists
vanstee
137
6.8k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Building an army of robots
kneath
303
45k
Producing Creativity
orderedlist
PRO
344
39k
Transcript
Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer
Advocate @ftisiot
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Event Driven Applications
@ftisiot What is an Event?
@ftisiot Why Event-driven?
@ftisiot What is Kafka? Topic A Topic B 0 1
2 3 4 0 1 2 3
@ftisiot Distributed Brokers Replication Factor 3 2
@ftisiot Distributed Replication Factor 2
@ftisiot What is a Kafka Event? <K, V> <"temp_max", 35.3>
<{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >
@ftisiot Write to Kafka Producer Topic A Hostname + Port
Authentication Encoding
@ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from
Kafka Consumer Offset 0 1 2 3
@ftisiot Pizza Demo!
@ftisiot Log Size
@ftisiot Topic Partitions
@ftisiot Distributed
@ftisiot Topic Retention Policies Time Size
@ftisiot Selecting a Partition <K, V> Same Key Same Partition
@ftisiot Ordering 2 1 3 2 1 3 Partition 0
Partition 1 Partition 1 Guaranteed per Partition
@ftisiot Scaling Out P1 P2 P3 C1 C2
@ftisiot Partitions Demo!
@ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer
Group 2
@ftisiot Consumer Groups Demo!
@ftisiot Kafka Connect Kafka Connect Kafka Connect
@ftisiot Evolve Existing Applications Kafka Connect CDC
@ftisiot Distribute Events Config File Only!
@ftisiot Kafka Connect Demo!
@ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks
@ftisiot Ask me Anything