Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Get to know Apache Kafka with Jupyter Notebooks

FTisiot
October 28, 2021

Get to know Apache Kafka with Jupyter Notebooks

Apache Kafka is recognised as the best data streaming platform around, but it can be difficult to observe what is happening when you are just getting started with this excellent technology platform. In this session, you will get a tour of key Kafka features using the delightful web UI of Jupyter notebooks.

Use the notebooks to see Kafka in action, producing and consuming records. We’ll also cover how to get the best from your application by making good use of topic partitioning and consumer groups. From high-level concepts to nitty-gritty details, this session will equip you with the skills you need to build applications with Kafka. Recommended for anyone curious about Apache Kafka or using Jupyter to work with or learn new data platforms.

FTisiot

October 28, 2021
Tweet

More Decks by FTisiot

Other Decks in Technology

Transcript

  1. Francesco Tisiot - Developer Advocate @ftisiot
    Get to Know


    Apache Kafka


    with


    Jupyter Notebooks

    View full-size slide

  2. @ftisiot | @aiven_io
    Why
    Jupyter
    Notebooks

    View full-size slide

  3. @ftisiot | @aiven_io
    Title Text
    Markdown Text
    Images
    Executable Code

    View full-size slide

  4. @ftisiot | @aiven_io
    Title Text
    https:/
    /aiven.io/blog/teach-yourself-apache-kafka-and-python-with-a-jupyter-notebook

    View full-size slide

  5. @ftisiot | @aiven_io
    Why
    Apache
    Kafka

    View full-size slide

  6. @ftisiot | @aiven_io

    View full-size slide

  7. @ftisiot | @aiven_io

    View full-size slide

  8. @ftisiot | @aiven_io
    Event-Driven Applications

    View full-size slide

  9. @ftisiot | @aiven_io
    What is an Event

    View full-size slide

  10. @ftisiot | @aiven_io
    Why Event Driven?

    View full-size slide

  11. @ftisiot | @aiven_io
    What is Kafka
    Topic A
    Topic B
    0
    1
    2
    3
    4
    0
    1
    2
    3

    View full-size slide

  12. @ftisiot | @aiven_io
    Distributed
    Brokers
    Replication Factor
    3 2

    View full-size slide

  13. @ftisiot | @aiven_io
    What’s a Kafka Event?
    <"temp_max", 35.3>
    <{


    shop: "Mario's Pizza",


    phone_line: 3


    }
    {


    orderId: 123456


    name: "Andrea Pirlo",


    pizzas: [


    {


    pizza: "Margherita",


    additionalToppings: ["Tuna","Onions","Ham"]


    },


    ...


    ]


    }
    , >

    View full-size slide

  14. @ftisiot | @aiven_io
    Write to Kafka
    Producer
    Topic A
    Hostname + Port
    Authentication
    Encoding

    View full-size slide

  15. @ftisiot | @aiven_io
    Read from Kafka
    Hostname + Port
    Authentication
    Decoding
    Topic Name(s)
    Consumer
    Offset
    0
    1
    2
    3

    View full-size slide

  16. @ftisiot | @aiven_io
    Pizza
    Demo!

    View full-size slide

  17. @ftisiot | @aiven_io
    Log Size

    View full-size slide

  18. @ftisiot | @aiven_io
    Topic Partitions

    View full-size slide

  19. @ftisiot | @aiven_io

    View full-size slide

  20. @ftisiot | @aiven_io
    Selecting the Partition

    Same Key Same Partition

    View full-size slide

  21. @ftisiot | @aiven_io
    Topic Partitions
    2 1
    3 2 1
    3
    Partition 0
    Partition 1
    Guaranteed per Partition

    View full-size slide

  22. @ftisiot | @aiven_io
    Scaling Out
    P1
    P2
    P3
    C1
    C2

    View full-size slide

  23. @ftisiot | @aiven_io
    Partitions
    Demo!

    View full-size slide

  24. @ftisiot | @aiven_io
    Multiple Applications
    C1
    C2
    Consumer


    Group 1
    CX
    Consumer


    Group 2

    View full-size slide

  25. @ftisiot | @aiven_io
    Kafka Connect
    Kafka Connect Kafka Connect

    View full-size slide

  26. @ftisiot | @aiven_io
    References
    https:/
    /github.com/aiven/kafka-python-fake-data-producer
    https:/
    /aiven.io
    @ftisiot
    https:/
    /github.com/aiven/kafka-python-notebooks

    View full-size slide

  27. @ftisiot | @aiven_io
    Title Text

    View full-size slide