Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kafka Meetup from March 2020

Kafka Meetup from March 2020

Meetup: https://www.meetup.com/Raleigh-Apache-Kafka-Meetup-by-Confluent/events/269215507

This meetup was originally posted on Raleigh's meetup user groups, but it is also available for: Charlotte, Richmond, Montreal, Boston, Atlanta, Toronto, Ottawa, Quebec, Waterloo, Philadelphia, Pittsburgh, Columbus, Cleveland, Cincinnati, Detroit, Western New York, Columbia, Charleston, Jacksonville, Orlando, Miami, Tampa, Louisville, Indianapolis, Nashville, Richmond, Ft. Wayne, Stamford, Hartford, Providence, Norfolk, and Washington, DC.

Ricardo Ferreira

March 31, 2020
Tweet

More Decks by Ricardo Ferreira

Other Decks in Programming

Transcript

  1. The talk will start shortly - before then, have a chat with your fellow attendees!
    Questions at the end
    This is unless the speaker says
    otherwise. But generally,
    because of internet delays and
    lags it can be a bit disruptive
    during an Online talk.
    Online Meetup Etiquette
    Mute during talk
    But remember to unmute when
    you want to ask a question at
    the end (or chat with fellow
    attendees before/After)
    Use chat to engage
    Use Zoom chat and/or create a
    thread in the #events channel
    on our Community Slack space
    so that your discussion can
    continue afterwards with
    fellow Kafkateers.
    Be on Camera &
    React!
    Speakers during these events
    are talking to blank screens.
    And it can be exhausting! Use
    Zoom reactions at the bottom of
    the page to give them something
    to work off of!
    1.
    2.
    Continue learning and collaborating
    WELCOME!
    Thank you for joining us in these unique circumstances. We hope you’re safe and well.
    3. If you haven’t already joined, get on the our slack
    workspace and continue the conversation with
    other Kafkateers. If you’re already a member,
    start a thread for this event in #events.
    A single online source of everything
    you’ll need to learn Kafka. Plus it’s
    totally free and ungated…
    Confluent Developer
    developer.confluent.io
    https://cnfl.io/slack
    Confluent Community Slack
    You are being recorded and this footage may be added to public channels

    View Slide

  2. About me
    @riferrei | #kafkameetup | @CONFLUENTINC
    • RICARDO FERREIRA
    • Works for confluent
    • Developer advocate
    [email protected]
    • HTTPS://RIFERREI.NET

    View Slide

  3. Building event
    streaming apps
    with pac-man
    @riferrei | #kafkameetup | @CONFLUENTINC

    View Slide

  4. @riferrei | #kafkameetup | @CONFLUENTINC
    Current state
    SQL
    SQL
    SQL
    application
    application
    application
    database
    LOG

    View Slide

  5. @riferrei | #kafkameetup | @CONFLUENTINC
    Why?

    View Slide

  6. @riferrei | #kafkameetup | @CONFLUENTINC
    I don’t know
    “my app only understands sql”
    “the source only understands sql”
    “I can only process atomic data”
    “that is what current books says”
    “it just feels right doing like this”

    View Slide

  7. @riferrei | #kafkameetup | @CONFLUENTINC
    But what if…
    database
    1000x more volume
    Non-transactional events
    Transactional events
    LOG

    View Slide

  8. what?
    Are you
    kidding me?

    View Slide

  9. @riferrei | #kafkameetup | @CONFLUENTINC
    This is fine

    View Slide

  10. @riferrei | #kafkameetup | @CONFLUENTINC
    Solving the
    problem

    View Slide

  11. @riferrei | #kafkameetup | @CONFLUENTINC
    Using the log
    database
    LOG
    0 1 2 3 4 5 6 7 8
    LOG
    reads
    New records
    application a
    (time = 1)
    application b
    (time = 3)

    View Slide

  12. @riferrei | #kafkameetup | @CONFLUENTINC
    But log programming hurts
    public void readTransactions() {
    while (true) {
    ConsumerRecords records =
    consumer.poll(Duration.ofMillis(100));
    for (Record record : records) {
    if (record.offset() != lastSavedOffset) {...}
    }
    }
    }

    View Slide

  13. @riferrei | #kafkameetup | @CONFLUENTINC
    What then?

    View Slide

  14. @riferrei | #kafkameetup | @CONFLUENTINC
    Event
    streaming
    appS

    View Slide

  15. @riferrei | @confluentinc | @itau

    View Slide

  16. @riferrei | @confluentinc | @itau

    View Slide

  17. @riferrei | #kafkameetup | @CONFLUENTINC
    PAC-MAN

    View Slide

  18. @riferrei | #kafkameetup | @CONFLUENTINC
    https://github.com/confluentinc/demo-scene
    <>
    Getting started

    View Slide

  19. @riferrei | #kafkameetup | @CONFLUENTINC
    IMPLEMENTATION
    Api
    gateway
    Lambda
    function
    Kafka
    topic
    Ksqldb
    apps
    Kafka
    topic
    scoreboard

    View Slide

  20. @riferrei | #kafkameetup | @CONFLUENTINC
    IMPLEMENTATION
    Kafka
    topic
    Ksqldb
    apps
    Kafka
    topic

    View Slide

  21. @riferrei | #kafkameetup | @CONFLUENTINC
    Let’s create
    the streams

    View Slide

  22. @riferrei | #kafkameetup | @CONFLUENTINC
    instructions
    1. Get the game 2. Name yourself

    View Slide

  23. @riferrei | #kafkameetup | @CONFLUENTINC
    BUILDING the
    scoreboard

    View Slide

  24. @riferrei | #kafkameetup | @CONFLUENTINC
    Introduction to ksqldb
    https://ksqldb.io

    View Slide

  25. @riferrei | #kafkameetup | @CONFLUENTINC
    Source: USER_GAME TOPIC

    View Slide

  26. @riferrei | #kafkameetup | @CONFLUENTINC
    Creating User_game stream

    View Slide

  27. @riferrei | #kafkameetup | @CONFLUENTINC
    Querying USER_GAME STREAM

    View Slide

  28. @riferrei | #kafkameetup | @CONFLUENTINC
    Creating Stats_per_user table

    View Slide

  29. @riferrei | #kafkameetup | @CONFLUENTINC
    Querying STATS_PER_USER TABLE

    View Slide

  30. @riferrei | #kafkameetup | @CONFLUENTINC
    Low latency Pull queries

    View Slide

  31. @riferrei | #kafkameetup | @CONFLUENTINC
    Source: User_losses topic

    View Slide

  32. @riferrei | #kafkameetup | @CONFLUENTINC
    Creating USER_LOSSES STREAM

    View Slide

  33. @riferrei | #kafkameetup | @CONFLUENTINC
    querying USER_LOSSES STREAM

    View Slide

  34. @riferrei | #kafkameetup | @CONFLUENTINC
    Creating LOSSES_PER_USER TABLE

    View Slide

  35. @riferrei | #kafkameetup | @CONFLUENTINC
    Querying LOSSES_PER_USER TABLE

    View Slide

  36. @riferrei | #kafkameetup | @CONFLUENTINC
    Creating SCOREBOARD TABLE

    View Slide

  37. @riferrei | #kafkameetup | @CONFLUENTINC
    Querying SCOREBOARD TABLE

    View Slide

  38. @riferrei | #kafkameetup | @CONFLUENTINC
    Playing with
    The data

    View Slide

  39. @riferrei | #kafkameetup | @CONFLUENTINC
    Read from kafka
    Api
    gateway
    Lambda
    function
    scoreboard
    Redis
    cache
    push

    View Slide

  40. @riferrei | #kafkameetup | @CONFLUENTINC
    Read from kafka
    Amazon
    alexa
    Lambda
    function
    scoreboard
    Redis
    cache
    push

    View Slide

  41. @riferrei | #kafkameetup | @CONFLUENTINC
    Read from ksqldb
    Your
    code
    Ksqldb
    table
    pull
    Kafka
    topic

    View Slide

  42. @riferrei | #kafkameetup | @CONFLUENTINC
    Creating highest_score TABLE

    View Slide

  43. @riferrei | #kafkameetup | @CONFLUENTINC
    Low latency Pull queries
    69920

    View Slide

  44. @riferrei | #kafkameetup | @CONFLUENTINC
    how can I
    learn more?

    View Slide

  45. @riferrei | #kafkameetup | @CONFLUENTINC
    Get kafka: confluent cloud
    Try free:
    https://cnfl.io/confluent-cloud

    View Slide

  46. @riferrei | #kafkameetup | @CONFLUENTINC
    https://cnfl.io/tutorials
    Get examples: kafka tutorials

    View Slide

  47. @riferrei | #kafkameetup | @CONFLUENTINC
    https://cnfl.io/books
    Get books: o’reilly bundle

    View Slide

  48. @riferrei | #kafkameetup | @CONFLUENTINC
    https://kafka-summit.org/events/kafka-summit-austin-2020
    join kafka summit
    https://myeventi.events/kafka20/aus
    Use 25% discount code: KSa20Meetup

    View Slide

  49. @riferrei | #kafkameetup | @CONFLUENTINC
    Thank you

    View Slide