Everything You Wanted to Know About Apache Kafka But You Were Too Afraid to Ask!

Everything You Wanted to Know About Apache Kafka But You Were Too Afraid to Ask!

02ff2dde723b6e26f4ef03ee6b3f6eb9?s=128

Ricardo Ferreira

June 04, 2019
Tweet

Transcript

  1. 1 Everything you Wanted to Know about Apache Kafka But

    You Were Too Afraid to Ask Ricardo Ferreira Developer Advocate, Confluent
  2. 2 Wakanda? Forever! Hulk? Smash! Apache Kafka? Its Like Messaging?

  3. About Us: • Ricardo Ferreira ❑ Developer Advocate @ Confluent

    ❑ Ex-Oracle, Red Hat, IONA Tech ❑ Currently ~70% Dev, ~30 Ops ❑ https://riferrei.net • Echo Dot (Alexa) ❑ The voice behind Amazon ❑ Ex-Raspberry Pi, Arduino @riferrei @alexa99
  4. @riferrei | @JNationConf | @confluentinc Question: "What is a Distributed

    Streaming Platform?"
  5. @riferrei | @JNationConf | @confluentinc ? ? ?

  6. Let's do some time travel Shaw we?

  7. @riferrei | @JNationConf | @confluentinc SQL DBs 25 years ago…

    SQL DBs Today Dude, you're embarrassing me in front of the wizards…
  8. @riferrei | @JNationConf | @confluentinc ETL/Batch

  9. @riferrei | @JNationConf | @confluentinc

  10. @riferrei | @JNationConf | @confluentinc What did it cost to

    extract data from the transactional DB?
  11. Latency

  12. Data Plumbing

  13. @riferrei | @JNationConf | @confluentinc

  14. @riferrei | @JNationConf | @confluentinc Solution for "Combining" Processing and

    Data: NoSQL Solution for Large Amounts of Data: Big Data!
  15. @riferrei | @JNationConf | @confluentinc How about Messaging?

  16. None
  17. None
  18. @riferrei | @JNationConf | @confluentinc

  19. 19 Event-Driven App (Location Tracking) Only Real-Time Events Messaging Queues

    and Event Streaming Platforms can do this Contextual Event-Driven App (ETA) Real-Time combined with stored data Only Event Streaming Platforms can do this Where is my driver? When will my driver get here? Where is my driver? When will my driver get here? Why Combine Real-time With Historical Context? 2 min
  20. 20 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to

    Scale No Persistence After Consumption No Replay Highly Scalable Durable Persistent Ordered Fast (Low Latency) What is happening in the world What happened in the world
  21. 21 Highly Scalable Durable Persistent Maintains Order ETL/Data Integration Messaging

    ETL/Data Integration Messaging Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay Fast (Low Latency) What happened in the world What is happening in the world Highly Scalable Durable Persistent Ordered Fast (Low Latency) Event Streaming Thinking
  22. @riferrei | @JNationConf | @confluentinc

  23. @riferrei | @JNationConf | @confluentinc

  24. @riferrei | @JNationConf | @confluentinc http://the-song-is-riferrei.s3- website-us-east-1.amazonaws.com/

  25. @riferrei | @JNationConf | @confluentinc https://github.com/riferrei/the-song-is

  26. @riferrei | @JNationConf | @confluentinc Have you Ever Heard About

    the #AskConfluent Initiative?
  27. There was an idea…

  28. To bring together a group of remarkable people…

  29. That could answer the questions…

  30. That we never could…

  31. @riferrei | @JNationConf | @confluentinc @tlberglund @gwenshap

  32. @riferrei | @JNationConf | @confluentinc https://www.youtube.com/playlist?list= PLa7VYi0yPIH0snucuYWkuUXwasMr-HR7Y Kafka is so

    Cool!
  33. @riferrei | @JNationConf | @confluentinc Steps: 1. Make your question

    on Twitter using the hashtag #AskConfluent 2. Wait for the next episode
  34. @riferrei | @JNationConf | @confluentinc Question: "Is There Such a

    Thing as Oversubscribing to Kafka?"
  35. @riferrei | @JNationConf | @confluentinc Question: "Is it More Costly

    to Up Convert or Down Convert Message Formats for Records Sent to Apache Kafka?"
  36. @riferrei | @JNationConf | @confluentinc Question: "How Many Partitions a

    Topic Can Have?"
  37. @riferrei | @JNationConf | @confluentinc Question: "What are the Pros

    and Cons About Exactly-Once Feature?"
  38. @riferrei | @JNationConf | @confluentinc Question: "'auto.offset.reset=earliest' was set but

    after restarting the app (it was 4 days down) it started to process from the beginning of the topic…"
  39. @riferrei | @JNationConf | @confluentinc Question: "Not Using Keys to

    Produce a Record is Considered a Bad Practice?"
  40. @riferrei | @JNationConf | @confluentinc Free Books

  41. @riferrei | @JNationConf | @confluentinc

  42. 42