Event-Driven Applications: Apache Kafka and Python

How to create Event-Driven Applications with Apache Kafka and Python

FTisiot

March 24, 2021

More Decks by FTisiot

See All by FTisiot

Apache Kafka JDBC Source Connector: What could go wrong?

0

150

Solve the Knapsack problem with recursive queries and PostgreSQL

0

180

Build an Open Source Data Pipeline

0

350

Get to know Apache Kafka with Jupyter Notebooks

0

790

Fix Your Strings in PostgreSQL

0

300

Practical Tips & Tricks for Apache Kafka Messages Integration

0

270

Track Database Changes with Apache Kafka

0

320

Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL

0

1k

Kickstart your Kafka with Faker Data

0

480

Other Decks in Technology

See All in Technology

Making AI Agents Safe and Fast- Jev, Obsidian, and the Meta-Harness

PRO

0

120

作って終わりじゃないサーバーレス〜9年運用する大規模EC物流API基盤の設計・運用のリアル〜

PRO

0

460

ほんとうの信頼性はヒーローが死んでからはじまる / True reliability begins after the hero dies

0

100

バイブコーディング時代のWebアプリ開発入門~Cloud Runで学ぶセキュアなビルドとデプロイ

1

140

人間はどの意思決定を手放せるのか

15

8k

30座EKS, 180次升級淬煉的EKS Upgrade Skill 的歷程

0

210

2026-09-18 gotanda.sre Terraformで複数環境作ったり、複数Stateに分割したりそれとTerragrunt / Terraform multi envs and multi states

4

700

白金鉱業Meetup Vol.25 アウトカムが二値のデータに対するCausal Impact

0

290

AIで社員の自主発信に広報目線を組み込む

0

140

AIを活用するために決めた "やらないこと" - 価値に注目する / Not betting on AI

PRO

3

590

おい、エージェントを使って終わらせろ

3

870

Oracle Base Database Service 技術詳細

oracle4engineer

PRO

16

120k

Featured

See All Featured

Building Experiences: Design Systems, User Experience, and Full Site Editing

1

610

Designing for Performance

611

70k

Ten Tips & Tricks for a 🌱 transition

1

240

RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub

141

35k

Agile Leadership in an Agile Organization

PRO

0

240

The Pragmatic Product Professional

37

7.4k

Building Better People: How to give real-time feedback that sticks.

370

20k

No one is an island. Learnings from fostering a developers community.

21

3.8k

Let's Do A Bunch of Simple Stuff to Make Websites Faster

508

140k

Beyond borders and beyond the search box: How to win the global "messy middle" with AI-driven SEO

3

260

30

1.6k

Bridging the Design Gap: How Collaborative Modelling removes blockers to flow between stakeholders and teams @FastFlow conf

0

700

Transcript

Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer
Advocate @ftisiot
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Event Driven Applications
@ftisiot What is an Event?
@ftisiot Why Event-driven?
@ftisiot What is Kafka? Topic A Topic B 0 1
2 3 4 0 1 2 3
@ftisiot Distributed Brokers Replication Factor 3 2
@ftisiot Distributed Replication Factor 2
@ftisiot What is a Kafka Event? <K, V> <"temp_max", 35.3>
<{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >
@ftisiot Write to Kafka Producer Topic A Hostname + Port
Authentication Encoding
@ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from
Kafka Consumer Offset 0 1 2 3
@ftisiot Pizza Demo!
@ftisiot Log Size
@ftisiot Topic Partitions
@ftisiot Distributed
@ftisiot Topic Retention Policies Time Size
@ftisiot Selecting a Partition <K, V> Same Key Same Partition
@ftisiot Ordering 2 1 3 2 1 3 Partition 0
Partition 1 Partition 1 Guaranteed per Partition
@ftisiot Scaling Out P1 P2 P3 C1 C2
@ftisiot Partitions Demo!
@ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer
Group 2
@ftisiot Consumer Groups Demo!
@ftisiot Kafka Connect Kafka Connect Kafka Connect
@ftisiot Evolve Existing Applications Kafka Connect CDC
@ftisiot Distribute Events Config File Only!
@ftisiot Kafka Connect Demo!
@ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks
@ftisiot Ask me Anything