Event-Driven Applications: Apache Kafka and Python

Slide 1

Slide 1 text

Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer Advocate @ftisiot

Slide 2

Slide 2 text

@ftisiot Why Kafka?

Slide 3

Slide 3 text

@ftisiot Why Kafka?

Slide 4

Slide 4 text

@ftisiot Why Kafka?

Slide 5

Slide 5 text

@ftisiot Event Driven Applications

Slide 6

Slide 6 text

@ftisiot What is an Event?

Slide 7

Slide 7 text

@ftisiot Why Event-driven?

Slide 8

Slide 8 text

@ftisiot What is Kafka? Topic A Topic B 0 1 2 3 4 0 1 2 3

Slide 9

Slide 9 text

@ftisiot Distributed Brokers Replication Factor 3 2

Slide 10

Slide 10 text

@ftisiot Distributed Replication Factor 2

Slide 11

Slide 11 text

@ftisiot What is a Kafka Event? <"temp_max", 35.3> <{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >

Slide 12

Slide 12 text

@ftisiot Write to Kafka Producer Topic A Hostname + Port Authentication Encoding

Slide 13

Slide 13 text

@ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from Kafka Consumer Offset 0 1 2 3

Slide 14

Slide 14 text

@ftisiot Pizza Demo!

Slide 15

Slide 15 text

@ftisiot Log Size

Slide 16

Slide 16 text

@ftisiot Topic Partitions

Slide 17

Slide 17 text

@ftisiot Distributed

Slide 18

Slide 18 text

@ftisiot Topic Retention Policies Time Size

Slide 19

Slide 19 text

@ftisiot Selecting a Partition Same Key Same Partition

Slide 20

Slide 20 text

@ftisiot Ordering 2 1 3 2 1 3 Partition 0 Partition 1 Partition 1 Guaranteed per Partition

Slide 21

Slide 21 text

@ftisiot Scaling Out P1 P2 P3 C1 C2

Slide 22

Slide 22 text

@ftisiot Partitions Demo!

Slide 23

Slide 23 text

@ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer Group 2

Slide 24

Slide 24 text

@ftisiot Consumer Groups Demo!

Slide 25

Slide 25 text

@ftisiot Kafka Connect Kafka Connect Kafka Connect

Slide 26

Slide 26 text

@ftisiot Evolve Existing Applications Kafka Connect CDC

Slide 27

Slide 27 text

@ftisiot Distribute Events Config File Only!

Slide 28

Slide 28 text

@ftisiot Kafka Connect Demo!

Slide 29

Slide 29 text

@ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks

Slide 30

Slide 30 text

@ftisiot Ask me Anything