Everything You Wanted to Know About Apache Kafka But Were Too Afraid to Ask!

Presented in the Data Science Meetup in San Antonio, TX.

Transcript

1. Join the Confluent Community Slack Subscribe to the Confluent blog cnfl.io/slack cnfl.io/read Welcome to the San Antonio Apache Kafka® Meetup! 6:00pm Doors open 6:00pm - 6:30pm Pizza, Drinks and Networking 6:30pm - 7:30pm Ricardo Ferreira, Confluent 7:30pm - 7:45pm Additional Q&A & Networking Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the Apache Software Foundation. The Apache Software Foundation has no affiliation with and does not endorse the materials provided at this event.
2. @riferrei | #kafkameetup | @CONFLUENTINC Everything you wanted to know about kafka But you were too afraid to ask! @riferrei | #kafkameetup | @CONFLUENTINC
3. @riferrei | #kafkameetup | @CONFLUENTINC Wakanda? Forever! Hulk? Smash! Apache Kafka? Its Like Messaging?
4. About Us: • Ricardo Ferreira ❑ Developer Advocate @ Confluent ❑ Ex-Oracle, Red Hat, IONA Tech ❑ Ricardo@confluent.io ❑ https://riferrei.net • Alexa (echo dot) ❑ The voice behind Amazon ❑ Ex-Raspberry Pi, Arduino ❑ She is a female in character! @riferrei @alexa99
5. @riferrei | #kafkameetup | @CONFLUENTINC Question: What is a distributed streaming platform?
6. @riferrei | #kafkameetup | @CONFLUENTINC ? ? ?
7. @riferrei | #kafkameetup | @CONFLUENTINC Let's do some time travel Shaw we?
8. @riferrei | #kafkameetup | @CONFLUENTINC SQL DBs 25 years ago… SQL DBs Today Dude, you're embarrassing me in front of the wizards…
9. @riferrei | #kafkameetup | @CONFLUENTINC ETL / Batch
10. @riferrei | #kafkameetup | @CONFLUENTINC
11. @riferrei | #kafkameetup | @CONFLUENTINC What does it cost extracting data from the transactional DB?
12. @riferrei | #kafkameetup | @CONFLUENTINC latency
13. @riferrei | #kafkameetup | @CONFLUENTINC plumbing
14. @riferrei | #kafkameetup | @CONFLUENTINC Mgmt complexity
15. @riferrei | #kafkameetup | @CONFLUENTINC 15 Industry solutions
16. @riferrei | #kafkameetup | @CONFLUENTINC Solution for "Combining" Processing and Data: NoSQL Solution for Large Amounts of Data: big data
17. @riferrei | #kafkameetup | @CONFLUENTINC How about Messaging?
18. @riferrei | #kafkameetup | @CONFLUENTINC
19. @riferrei | #kafkameetup | @CONFLUENTINC Let's do some time travel Shaw we?
20. @riferrei | #kafkameetup | @CONFLUENTINC
21. @riferrei | #kafkameetup | @CONFLUENTINC
22. @riferrei | #kafkameetup | @CONFLUENTINC “The truth is the log. The database is a cache of a subset of the log.” — pat helland Immutability changes everything http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
23. 23 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence After Consumption No Replay Highly Scalable Durable Persistent Ordered Fast (Low Latency) What is happening in the world What happened in the world
24. 24 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence After Consumption No Replay Highly Scalable Durable Persistent Ordered Fast (Low Latency) What is happening in the world What happened in the world Highly Scalable Durable Persistent Ordered Fast (Low Latency) Event Streaming Thinking
25. @riferrei | #kafkameetup | @CONFLUENTINC Question: What is a distributed streaming platform?
26. @riferrei | #kafkameetup | @CONFLUENTINC 01 Messaging done right 02 Scalable data pipelines 03 Stream processing @riferrei | #kafkameetup | @CONFLUENTINC
27. 27 Event-Driven App (Location Tracking) Only Real-Time Events Messaging Queues and pub/sub Platforms can do this Contextual Event-Driven App (ETA) Real-Time combined with stored data Only Streaming data Platforms can do this Where is my driver? When will my driver get here? Where is my driver? When will my driver arrive? Why Combine Real-time With Historical Context? 2 min
28. @riferrei | #kafkameetup | @CONFLUENTINC 28 demo
29. @riferrei | #kafkameetup | @CONFLUENTINC
30. Join The fun
31. @riferrei | #kafkameetup | @CONFLUENTINC https://github.com/riferrei/the-song-is
32. @riferrei | #kafkameetup | @CONFLUENTINC Have you EVER HEARD ABOUT THE #ASKCONFLUENT INITIATIVE?
33. @riferrei | #kafkameetup | @CONFLUENTINC There was an idea…
34. @riferrei | #kafkameetup | @CONFLUENTINC TO BRING TOGETHER A GROUP OF REMARKABLE PEOPLE
35. @riferrei | #kafkameetup | @CONFLUENTINC THAT COULD ANSWER THE QUESTIONS…
36. @riferrei | #kafkameetup | @CONFLUENTINC THAT they NEVER COULD...
37. @riferrei | #kafkameetup | @CONFLUENTINC @tlberglund @gwenshap
38. @riferrei | #kafkameetup | @CONFLUENTINC https://www.youtube.com/playlist?list= PLa7VYi0yPIH0snucuYWkuUXwasMr-HR7Y Kafka is so Cool!
39. @riferrei | #kafkameetup | @CONFLUENTINC Question: IS THERE SUCH A THING OF OVERSUBSCRIPING KAFKA?
40. @riferrei | #kafkameetup | @CONFLUENTINC Question: WHAT IS MOST COSTLY... UP CONVERT OR DOWN CONVERT?
41. @riferrei | #kafkameetup | @CONFLUENTINC Question: HOW MANY PARTITIONS A TOPIC CAN HAVE?
42. @riferrei | #kafkameetup | @CONFLUENTINC Question: WHAT ARE THE PROS AND CONS OF THE EXACTLY-ONCE FEATURE?
43. @riferrei | #kafkameetup | @CONFLUENTINC Question: WHAT IS THE IMPACT OF CONSUMER APPS BEING DOWN FOR 4 DAYS?
44. @riferrei | #kafkameetup | @CONFLUENTINC Question: NOT USING KEYS WHILE PRODUCING RECORDS IS A BAD PRACTICE?
45. @riferrei | #kafkameetup | @CONFLUENTINC 45 announcements
46. NOMINATE YOURSELF OR A PEER AT CONFLUENT.IO/NOMINATE
47. KS19Meetup. CONFLUENT COMMUNITY DISCOUNT CODE 25% OFF* *Standard Priced Conference pass
48. @riferrei | #kafkameetup | @CONFLUENTINC