Everything You Wanted to Know About Apache Kafka But You Were Too Afraid to Ask!

Transcript

1. 1 Everything you Wanted to Know about Apache Kafka But You Were Too Afraid to Ask Ricardo Ferreira Developer Advocate, Confluent
2. 2 Wakanda? Forever! Hulk? Smash! Apache Kafka? Its Like Messaging?
3. About Us: • Ricardo Ferreira ❑ Developer Advocate @ Confluent ❑ Ex-Oracle, Red Hat, IONA Tech ❑ Currently ~70% Dev, ~30 Ops ❑ https://riferrei.net • Echo Dot (Alexa) ❑ The voice behind Amazon ❑ Ex-Raspberry Pi, Arduino @riferrei @alexa99
4. @riferrei | @JNationConf | @confluentinc Question: "What is a Distributed Streaming Platform?"
5. @riferrei | @JNationConf | @confluentinc ? ? ?
6. Let's do some time travel Shaw we?
7. @riferrei | @JNationConf | @confluentinc SQL DBs 25 years ago… SQL DBs Today Dude, you're embarrassing me in front of the wizards…
8. @riferrei | @JNationConf | @confluentinc ETL/Batch
9. @riferrei | @JNationConf | @confluentinc
10. @riferrei | @JNationConf | @confluentinc What did it cost to extract data from the transactional DB?
11. Latency
12. Data Plumbing
13. @riferrei | @JNationConf | @confluentinc
14. @riferrei | @JNationConf | @confluentinc Solution for "Combining" Processing and Data: NoSQL Solution for Large Amounts of Data: Big Data!
15. @riferrei | @JNationConf | @confluentinc How about Messaging?
18. @riferrei | @JNationConf | @confluentinc
19. 19 Event-Driven App (Location Tracking) Only Real-Time Events Messaging Queues and Event Streaming Platforms can do this Contextual Event-Driven App (ETA) Real-Time combined with stored data Only Event Streaming Platforms can do this Where is my driver? When will my driver get here? Where is my driver? When will my driver get here? Why Combine Real-time With Historical Context? 2 min
20. 20 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence After Consumption No Replay Highly Scalable Durable Persistent Ordered Fast (Low Latency) What is happening in the world What happened in the world
21. 21 Highly Scalable Durable Persistent Maintains Order ETL/Data Integration Messaging ETL/Data Integration Messaging Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay Fast (Low Latency) What happened in the world What is happening in the world Highly Scalable Durable Persistent Ordered Fast (Low Latency) Event Streaming Thinking
22. @riferrei | @JNationConf | @confluentinc
23. @riferrei | @JNationConf | @confluentinc
24. @riferrei | @JNationConf | @confluentinc http://the-song-is-riferrei.s3- website-us-east-1.amazonaws.com/
25. @riferrei | @JNationConf | @confluentinc https://github.com/riferrei/the-song-is
26. @riferrei | @JNationConf | @confluentinc Have you Ever Heard About the #AskConfluent Initiative?
27. There was an idea…
28. To bring together a group of remarkable people…
29. That could answer the questions…
30. That we never could…
31. @riferrei | @JNationConf | @confluentinc @tlberglund @gwenshap
32. @riferrei | @JNationConf | @confluentinc https://www.youtube.com/playlist?list= PLa7VYi0yPIH0snucuYWkuUXwasMr-HR7Y Kafka is so Cool!
33. @riferrei | @JNationConf | @confluentinc Steps: 1. Make your question on Twitter using the hashtag #AskConfluent 2. Wait for the next episode
34. @riferrei | @JNationConf | @confluentinc Question: "Is There Such a Thing as Oversubscribing to Kafka?"
35. @riferrei | @JNationConf | @confluentinc Question: "Is it More Costly to Up Convert or Down Convert Message Formats for Records Sent to Apache Kafka?"
36. @riferrei | @JNationConf | @confluentinc Question: "How Many Partitions a Topic Can Have?"
37. @riferrei | @JNationConf | @confluentinc Question: "What are the Pros and Cons About Exactly-Once Feature?"
38. @riferrei | @JNationConf | @confluentinc Question: "'auto.offset.reset=earliest' was set but after restarting the app (it was 4 days down) it started to process from the beginning of the topic…"
39. @riferrei | @JNationConf | @confluentinc Question: "Not Using Keys to Produce a Record is Considered a Bad Practice?"
40. @riferrei | @JNationConf | @confluentinc Free Books
41. @riferrei | @JNationConf | @confluentinc