Event-Driven Applications: Apache Kafka and Python

Transcript

1. Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer

   Advocate @ftisiot

2. @ftisiot Why Kafka?

3. @ftisiot Why Kafka?

4. @ftisiot Why Kafka?

5. @ftisiot Event Driven Applications

6. @ftisiot What is an Event?

7. @ftisiot Why Event-driven?

8. @ftisiot What is Kafka? Topic A Topic B 0 1

   2 3 4 0 1 2 3

9. @ftisiot Distributed Brokers Replication Factor 3 2

10. @ftisiot Distributed Replication Factor 2

11. @ftisiot What is a Kafka Event? <K, V> <"temp_max", 35.3>

    <{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >

12. @ftisiot Write to Kafka Producer Topic A Hostname + Port

    Authentication Encoding

13. @ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from

    Kafka Consumer Offset 0 1 2 3

14. @ftisiot Pizza Demo!

15. @ftisiot Log Size

16. @ftisiot Topic Partitions

17. @ftisiot Distributed

18. @ftisiot Topic Retention Policies Time Size

19. @ftisiot Selecting a Partition <K, V> Same Key Same Partition

20. @ftisiot Ordering 2 1 3 2 1 3 Partition 0

    Partition 1 Partition 1 Guaranteed per Partition

21. @ftisiot Scaling Out P1 P2 P3 C1 C2

22. @ftisiot Partitions Demo!

23. @ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer

    Group 2

24. @ftisiot Consumer Groups Demo!

25. @ftisiot Kafka Connect Kafka Connect Kafka Connect

26. @ftisiot Evolve Existing Applications Kafka Connect CDC

27. @ftisiot Distribute Events Config File Only!

28. @ftisiot Kafka Connect Demo!

29. @ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks

30. @ftisiot Ask me Anything