Kafka Meetup from March 2020

Kafka Meetup from March 2020

Meetup: https://www.meetup.com/Raleigh-Apache-Kafka-Meetup-by-Confluent/events/269215507

This meetup was originally posted on Raleigh's meetup user groups, but it is also available for: Charlotte, Richmond, Montreal, Boston, Atlanta, Toronto, Ottawa, Quebec, Waterloo, Philadelphia, Pittsburgh, Columbus, Cleveland, Cincinnati, Detroit, Western New York, Columbia, Charleston, Jacksonville, Orlando, Miami, Tampa, Louisville, Indianapolis, Nashville, Richmond, Ft. Wayne, Stamford, Hartford, Providence, Norfolk, and Washington, DC.

02ff2dde723b6e26f4ef03ee6b3f6eb9?s=128

Ricardo Ferreira

March 31, 2020
Tweet

Transcript

  1. The talk will start shortly - before then, have a

    chat with your fellow attendees! Questions at the end This is unless the speaker says otherwise. But generally, because of internet delays and lags it can be a bit disruptive during an Online talk. Online Meetup Etiquette Mute during talk But remember to unmute when you want to ask a question at the end (or chat with fellow attendees before/After) Use chat to engage Use Zoom chat and/or create a thread in the #events channel on our Community Slack space so that your discussion can continue afterwards with fellow Kafkateers. Be on Camera & React! Speakers during these events are talking to blank screens. And it can be exhausting! Use Zoom reactions at the bottom of the page to give them something to work off of! 1. 2. Continue learning and collaborating WELCOME! Thank you for joining us in these unique circumstances. We hope you’re safe and well. 3. If you haven’t already joined, get on the our slack workspace and continue the conversation with other Kafkateers. If you’re already a member, start a thread for this event in #events. A single online source of everything you’ll need to learn Kafka. Plus it’s totally free and ungated… Confluent Developer developer.confluent.io https://cnfl.io/slack Confluent Community Slack You are being recorded and this footage may be added to public channels
  2. About me @riferrei | #kafkameetup | @CONFLUENTINC • RICARDO FERREIRA

    • Works for confluent • Developer advocate • Ricardo@confluent.iO • HTTPS://RIFERREI.NET
  3. Building event streaming apps with pac-man @riferrei | #kafkameetup |

    @CONFLUENTINC
  4. @riferrei | #kafkameetup | @CONFLUENTINC Current state SQL SQL SQL

    application application application database LOG
  5. @riferrei | #kafkameetup | @CONFLUENTINC Why?

  6. @riferrei | #kafkameetup | @CONFLUENTINC I don’t know “my app

    only understands sql” “the source only understands sql” “I can only process atomic data” “that is what current books says” “it just feels right doing like this”
  7. @riferrei | #kafkameetup | @CONFLUENTINC But what if… database 1000x

    more volume Non-transactional events Transactional events LOG
  8. what? Are you kidding me?

  9. @riferrei | #kafkameetup | @CONFLUENTINC This is fine

  10. @riferrei | #kafkameetup | @CONFLUENTINC Solving the problem

  11. @riferrei | #kafkameetup | @CONFLUENTINC Using the log database LOG

    0 1 2 3 4 5 6 7 8 LOG reads New records application a (time = 1) application b (time = 3)
  12. @riferrei | #kafkameetup | @CONFLUENTINC But log programming hurts public

    void readTransactions() { while (true) { ConsumerRecords<String, Type> records = consumer.poll(Duration.ofMillis(100)); for (Record<String, Type> record : records) { if (record.offset() != lastSavedOffset) {...} } } }
  13. @riferrei | #kafkameetup | @CONFLUENTINC What then?

  14. @riferrei | #kafkameetup | @CONFLUENTINC Event streaming appS

  15. @riferrei | @confluentinc | @itau

  16. @riferrei | @confluentinc | @itau

  17. @riferrei | #kafkameetup | @CONFLUENTINC PAC-MAN

  18. @riferrei | #kafkameetup | @CONFLUENTINC https://github.com/confluentinc/demo-scene <<Pacman-ccloud>> Getting started

  19. @riferrei | #kafkameetup | @CONFLUENTINC IMPLEMENTATION Api gateway Lambda function

    Kafka topic Ksqldb apps Kafka topic scoreboard
  20. @riferrei | #kafkameetup | @CONFLUENTINC IMPLEMENTATION Kafka topic Ksqldb apps

    Kafka topic
  21. @riferrei | #kafkameetup | @CONFLUENTINC Let’s create the streams

  22. @riferrei | #kafkameetup | @CONFLUENTINC instructions 1. Get the game

    2. Name yourself
  23. @riferrei | #kafkameetup | @CONFLUENTINC BUILDING the scoreboard

  24. @riferrei | #kafkameetup | @CONFLUENTINC Introduction to ksqldb https://ksqldb.io

  25. @riferrei | #kafkameetup | @CONFLUENTINC Source: USER_GAME TOPIC

  26. @riferrei | #kafkameetup | @CONFLUENTINC Creating User_game stream

  27. @riferrei | #kafkameetup | @CONFLUENTINC Querying USER_GAME STREAM

  28. @riferrei | #kafkameetup | @CONFLUENTINC Creating Stats_per_user table

  29. @riferrei | #kafkameetup | @CONFLUENTINC Querying STATS_PER_USER TABLE

  30. @riferrei | #kafkameetup | @CONFLUENTINC Low latency Pull queries

  31. @riferrei | #kafkameetup | @CONFLUENTINC Source: User_losses topic

  32. @riferrei | #kafkameetup | @CONFLUENTINC Creating USER_LOSSES STREAM

  33. @riferrei | #kafkameetup | @CONFLUENTINC querying USER_LOSSES STREAM

  34. @riferrei | #kafkameetup | @CONFLUENTINC Creating LOSSES_PER_USER TABLE

  35. @riferrei | #kafkameetup | @CONFLUENTINC Querying LOSSES_PER_USER TABLE

  36. @riferrei | #kafkameetup | @CONFLUENTINC Creating SCOREBOARD TABLE

  37. @riferrei | #kafkameetup | @CONFLUENTINC Querying SCOREBOARD TABLE

  38. @riferrei | #kafkameetup | @CONFLUENTINC Playing with The data

  39. @riferrei | #kafkameetup | @CONFLUENTINC Read from kafka Api gateway

    Lambda function scoreboard Redis cache push
  40. @riferrei | #kafkameetup | @CONFLUENTINC Read from kafka Amazon alexa

    Lambda function scoreboard Redis cache push
  41. @riferrei | #kafkameetup | @CONFLUENTINC Read from ksqldb Your code

    Ksqldb table pull Kafka topic
  42. @riferrei | #kafkameetup | @CONFLUENTINC Creating highest_score TABLE

  43. @riferrei | #kafkameetup | @CONFLUENTINC Low latency Pull queries 69920

  44. @riferrei | #kafkameetup | @CONFLUENTINC how can I learn more?

  45. @riferrei | #kafkameetup | @CONFLUENTINC Get kafka: confluent cloud Try

    free: https://cnfl.io/confluent-cloud
  46. @riferrei | #kafkameetup | @CONFLUENTINC https://cnfl.io/tutorials Get examples: kafka tutorials

  47. @riferrei | #kafkameetup | @CONFLUENTINC https://cnfl.io/books Get books: o’reilly bundle

  48. @riferrei | #kafkameetup | @CONFLUENTINC https://kafka-summit.org/events/kafka-summit-austin-2020 join kafka summit https://myeventi.events/kafka20/aus

    Use 25% discount code: KSa20Meetup
  49. @riferrei | #kafkameetup | @CONFLUENTINC Thank you