Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Get to know Apache Kafka with Jupyter Notebooks

A23789f299ed06fe7d9f1c6940440bfa?s=47 FTisiot
October 28, 2021

Get to know Apache Kafka with Jupyter Notebooks

Apache Kafka is recognised as the best data streaming platform around, but it can be difficult to observe what is happening when you are just getting started with this excellent technology platform. In this session, you will get a tour of key Kafka features using the delightful web UI of Jupyter notebooks.

Use the notebooks to see Kafka in action, producing and consuming records. We’ll also cover how to get the best from your application by making good use of topic partitioning and consumer groups. From high-level concepts to nitty-gritty details, this session will equip you with the skills you need to build applications with Kafka. Recommended for anyone curious about Apache Kafka or using Jupyter to work with or learn new data platforms.

A23789f299ed06fe7d9f1c6940440bfa?s=128

FTisiot

October 28, 2021
Tweet

Transcript

  1. Francesco Tisiot - Developer Advocate @ftisiot Get to Know Apache

    Kafka with Jupyter Notebooks
  2. @ftisiot | @aiven_io Why Jupyter Notebooks

  3. @ftisiot | @aiven_io Title Text Markdown Text Images Executable Code

  4. @ftisiot | @aiven_io Title Text https:/ /aiven.io/blog/teach-yourself-apache-kafka-and-python-with-a-jupyter-notebook

  5. @ftisiot | @aiven_io Why Apache Kafka

  6. @ftisiot | @aiven_io

  7. @ftisiot | @aiven_io

  8. @ftisiot | @aiven_io Event-Driven Applications

  9. @ftisiot | @aiven_io What is an Event

  10. @ftisiot | @aiven_io Why Event Driven?

  11. @ftisiot | @aiven_io What is Kafka Topic A Topic B

    0 1 2 3 4 0 1 2 3
  12. @ftisiot | @aiven_io Distributed Brokers Replication Factor 3 2

  13. @ftisiot | @aiven_io What’s a Kafka Event? <K, V> <"temp_max",

    35.3> <{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >
  14. @ftisiot | @aiven_io Write to Kafka Producer Topic A Hostname

    + Port Authentication Encoding
  15. @ftisiot | @aiven_io Read from Kafka Hostname + Port Authentication

    Decoding Topic Name(s) Consumer Offset 0 1 2 3
  16. @ftisiot | @aiven_io Pizza Demo!

  17. @ftisiot | @aiven_io Log Size

  18. @ftisiot | @aiven_io Topic Partitions

  19. @ftisiot | @aiven_io

  20. @ftisiot | @aiven_io Selecting the Partition <K, V> Same Key

    Same Partition
  21. @ftisiot | @aiven_io Topic Partitions 2 1 3 2 1

    3 Partition 0 Partition 1 Guaranteed per Partition
  22. @ftisiot | @aiven_io Scaling Out P1 P2 P3 C1 C2

  23. @ftisiot | @aiven_io Partitions Demo!

  24. @ftisiot | @aiven_io Multiple Applications C1 C2 Consumer Group 1

    CX Consumer Group 2
  25. @ftisiot | @aiven_io Kafka Connect Kafka Connect Kafka Connect

  26. @ftisiot | @aiven_io References https:/ /github.com/aiven/kafka-python-fake-data-producer https:/ /aiven.io @ftisiot https:/

    /github.com/aiven/kafka-python-notebooks
  27. @ftisiot | @aiven_io Title Text