Kafka Meetup from March 2020

Meetup: https://www.meetup.com/Raleigh-Apache-Kafka-Meetup-by-Confluent/events/269215507

This meetup was originally posted on Raleigh's meetup user groups, but it is also available for: Charlotte, Richmond, Montreal, Boston, Atlanta, Toronto, Ottawa, Quebec, Waterloo, Philadelphia, Pittsburgh, Columbus, Cleveland, Cincinnati, Detroit, Western New York, Columbia, Charleston, Jacksonville, Orlando, Miami, Tampa, Louisville, Indianapolis, Nashville, Richmond, Ft. Wayne, Stamford, Hartford, Providence, Norfolk, and Washington, DC.

Transcript

1. The talk will start shortly - before then, have a chat with your fellow attendees!
   Questions at the end
   This is unless the speaker says
   otherwise. But generally,
   because of internet delays and
   lags it can be a bit disruptive
   during an Online talk.
   Online Meetup Etiquette
   Mute during talk
   But remember to unmute when
   you want to ask a question at
   the end (or chat with fellow
   attendees before/After)
   Use chat to engage
   Use Zoom chat and/or create a
   thread in the #events channel
   on our Community Slack space
   so that your discussion can
   continue afterwards with
   fellow Kafkateers.
   Be on Camera &
   React!
   Speakers during these events
   are talking to blank screens.
   And it can be exhausting! Use
   Zoom reactions at the bottom of
   the page to give them something
   to work off of!
   1.
   2.
   Continue learning and collaborating
   WELCOME!
   Thank you for joining us in these unique circumstances. We hope you’re safe and well.
   3. If you haven’t already joined, get on the our slack
   workspace and continue the conversation with
   other Kafkateers. If you’re already a member,
   start a thread for this event in #events.
   A single online source of everything
   you’ll need to learn Kafka. Plus it’s
   totally free and ungated…
   Confluent Developer
   developer.confluent.io
   https://cnfl.io/slack
   Confluent Community Slack
   You are being recorded and this footage may be added to public channels
   

   View Slide

2. About me
   @riferrei | #kafkameetup | @CONFLUENTINC
   • RICARDO FERREIRA
   • Works for confluent
   • Developer advocate
   • Ricardo@confluent.iO
   • HTTPS://RIFERREI.NET
   

   View Slide

3. Building event
   streaming apps
   with pac-man
   @riferrei | #kafkameetup | @CONFLUENTINC
   

   View Slide

4. @riferrei | #kafkameetup | @CONFLUENTINC
   Current state
   SQL
   SQL
   SQL
   application
   application
   application
   database
   LOG
   

   View Slide

5. @riferrei | #kafkameetup | @CONFLUENTINC
   Why?
   

   View Slide

6. @riferrei | #kafkameetup | @CONFLUENTINC
   I don’t know
   “my app only understands sql”
   “the source only understands sql”
   “I can only process atomic data”
   “that is what current books says”
   “it just feels right doing like this”
   

   View Slide

7. @riferrei | #kafkameetup | @CONFLUENTINC
   But what if…
   database
   1000x more volume
   Non-transactional events
   Transactional events
   LOG
   

   View Slide

8. what?
   Are you
   kidding me?
   

   View Slide

9. @riferrei | #kafkameetup | @CONFLUENTINC
   This is fine
   

   View Slide

10. @riferrei | #kafkameetup | @CONFLUENTINC
    Solving the
    problem
    

    View Slide

11. @riferrei | #kafkameetup | @CONFLUENTINC
    Using the log
    database
    LOG
    0 1 2 3 4 5 6 7 8
    LOG
    reads
    New records
    application a
    (time = 1)
    application b
    (time = 3)
    

    View Slide

12. @riferrei | #kafkameetup | @CONFLUENTINC
    But log programming hurts
    public void readTransactions() {
    while (true) {
    ConsumerRecords records =
    consumer.poll(Duration.ofMillis(100));
    for (Record record : records) {
    if (record.offset() != lastSavedOffset) {...}
    }
    }
    }
    

    View Slide

13. @riferrei | #kafkameetup | @CONFLUENTINC
    What then?
    

    View Slide

14. @riferrei | #kafkameetup | @CONFLUENTINC
    Event
    streaming
    appS
    

    View Slide

15. @riferrei | @confluentinc | @itau
    

    View Slide

16. @riferrei | @confluentinc | @itau
    

    View Slide

17. @riferrei | #kafkameetup | @CONFLUENTINC
    PAC-MAN
    

    View Slide

18. @riferrei | #kafkameetup | @CONFLUENTINC
    https://github.com/confluentinc/demo-scene
    <>
    Getting started
    

    View Slide

19. @riferrei | #kafkameetup | @CONFLUENTINC
    IMPLEMENTATION
    Api
    gateway
    Lambda
    function
    Kafka
    topic
    Ksqldb
    apps
    Kafka
    topic
    scoreboard
    

    View Slide

20. @riferrei | #kafkameetup | @CONFLUENTINC
    IMPLEMENTATION
    Kafka
    topic
    Ksqldb
    apps
    Kafka
    topic
    

    View Slide

21. @riferrei | #kafkameetup | @CONFLUENTINC
    Let’s create
    the streams
    

    View Slide

22. @riferrei | #kafkameetup | @CONFLUENTINC
    instructions
    1. Get the game 2. Name yourself
    

    View Slide

23. @riferrei | #kafkameetup | @CONFLUENTINC
    BUILDING the
    scoreboard
    

    View Slide

24. @riferrei | #kafkameetup | @CONFLUENTINC
    Introduction to ksqldb
    https://ksqldb.io
    

    View Slide

25. @riferrei | #kafkameetup | @CONFLUENTINC
    Source: USER_GAME TOPIC
    

    View Slide

26. @riferrei | #kafkameetup | @CONFLUENTINC
    Creating User_game stream
    

    View Slide

27. @riferrei | #kafkameetup | @CONFLUENTINC
    Querying USER_GAME STREAM
    

    View Slide

28. @riferrei | #kafkameetup | @CONFLUENTINC
    Creating Stats_per_user table
    

    View Slide

29. @riferrei | #kafkameetup | @CONFLUENTINC
    Querying STATS_PER_USER TABLE
    

    View Slide

30. @riferrei | #kafkameetup | @CONFLUENTINC
    Low latency Pull queries
    

    View Slide

31. @riferrei | #kafkameetup | @CONFLUENTINC
    Source: User_losses topic
    

    View Slide

32. @riferrei | #kafkameetup | @CONFLUENTINC
    Creating USER_LOSSES STREAM
    

    View Slide

33. @riferrei | #kafkameetup | @CONFLUENTINC
    querying USER_LOSSES STREAM
    

    View Slide

34. @riferrei | #kafkameetup | @CONFLUENTINC
    Creating LOSSES_PER_USER TABLE
    

    View Slide

35. @riferrei | #kafkameetup | @CONFLUENTINC
    Querying LOSSES_PER_USER TABLE
    

    View Slide

36. @riferrei | #kafkameetup | @CONFLUENTINC
    Creating SCOREBOARD TABLE
    

    View Slide

37. @riferrei | #kafkameetup | @CONFLUENTINC
    Querying SCOREBOARD TABLE
    

    View Slide

38. @riferrei | #kafkameetup | @CONFLUENTINC
    Playing with
    The data
    

    View Slide

39. @riferrei | #kafkameetup | @CONFLUENTINC
    Read from kafka
    Api
    gateway
    Lambda
    function
    scoreboard
    Redis
    cache
    push
    

    View Slide

40. @riferrei | #kafkameetup | @CONFLUENTINC
    Read from kafka
    Amazon
    alexa
    Lambda
    function
    scoreboard
    Redis
    cache
    push
    

    View Slide

41. @riferrei | #kafkameetup | @CONFLUENTINC
    Read from ksqldb
    Your
    code
    Ksqldb
    table
    pull
    Kafka
    topic
    

    View Slide

42. @riferrei | #kafkameetup | @CONFLUENTINC
    Creating highest_score TABLE
    

    View Slide

43. @riferrei | #kafkameetup | @CONFLUENTINC
    Low latency Pull queries
    69920
    

    View Slide

44. @riferrei | #kafkameetup | @CONFLUENTINC
    how can I
    learn more?
    

    View Slide

45. @riferrei | #kafkameetup | @CONFLUENTINC
    Get kafka: confluent cloud
    Try free:
    https://cnfl.io/confluent-cloud
    

    View Slide

46. @riferrei | #kafkameetup | @CONFLUENTINC
    https://cnfl.io/tutorials
    Get examples: kafka tutorials
    

    View Slide

47. @riferrei | #kafkameetup | @CONFLUENTINC
    https://cnfl.io/books
    Get books: o’reilly bundle
    

    View Slide

48. @riferrei | #kafkameetup | @CONFLUENTINC
    https://kafka-summit.org/events/kafka-summit-austin-2020
    join kafka summit
    https://myeventi.events/kafka20/aus
    Use 25% discount code: KSa20Meetup
    

    View Slide

49. @riferrei | #kafkameetup | @CONFLUENTINC
    Thank you
    

    View Slide