Save 37% off PRO during our Black Friday Sale! »

Everything you Wanted to Know about Apache Kafka But you Were Too Afraid to Ask!

Everything you Wanted to Know about Apache Kafka But you Were Too Afraid to Ask!

02ff2dde723b6e26f4ef03ee6b3f6eb9?s=128

Ricardo Ferreira

July 22, 2019
Tweet

Transcript

  1. Join the Confluent Community Slack Subscribe to the Confluent blog

    cnfl.io/slack cnfl.io/read Welcome to the DURHAM, NC Apache Kafka® Meetup! 6:00pm Doors open 6:00pm - 6:30pm Pizza, Drinks and Networking 6:30pm - 7:00pm John turner, mediamath 7:00pm - 8:00pm Ricardo ferreira, confluent Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the Apache Software Foundation. The Apache Software Foundation has no affiliation with and does not endorse the materials provided at this event.
  2. @riferrei | #kafkameetup | @CONFLUENTINC 2 Before we begin

  3. NOMINATE YOURSELF OR A PEER AT CONFLUENT.IO/NOMINATE

  4. KS19Meetup. CONFLUENT COMMUNITY DISCOUNT CODE 25% OFF* *Standard Priced Conference

    pass
  5. @riferrei | #kafkameetup | @CONFLUENTINC Everything you wanted to know

    about kafka But you were too afraid to ask! @riferrei | #kafkameetup | @CONFLUENTINC
  6. @riferrei | #kafkameetup | @CONFLUENTINC Wakanda? Forever! Hulk? Smash! Apache

    Kafka? Is it Like Messaging?
  7. About Us: • Ricardo Ferreira ❑ Developer Advocate @ Confluent

    ❑ Ex-Oracle, Red Hat, IONA Tech ❑ Ricardo@confluent.io ❑ https://riferrei.net • Alexa (amazon echo) ❑ The voice behind Amazon ❑ Ex-Raspberry Pi, Arduino ❑ She is a female in character! @riferrei @alexa99
  8. @riferrei | #kafkameetup | @CONFLUENTINC Question: What is a distributed

    streaming platform?
  9. @riferrei | #kafkameetup | @CONFLUENTINC ? ? ?

  10. @riferrei | #kafkameetup | @CONFLUENTINC Let’s go back in time.

    Shaw we?
  11. @riferrei | #kafkameetup | @CONFLUENTINC databases 30 years ago… Dude,

    you're embarrassing me in front of the wizards… Databases these days…
  12. @riferrei | #kafkameetup | @CONFLUENTINC 12 Database problems

  13. @riferrei | #kafkameetup | @CONFLUENTINC It is so limited Makes

    mess
  14. @riferrei | #kafkameetup | @CONFLUENTINC Etl is the result of

    databases being limited!
  15. @riferrei | #kafkameetup | @CONFLUENTINC What does it cost to

    extract data from the transactional?
  16. @riferrei | #kafkameetup | @CONFLUENTINC Silos = mess

  17. @riferrei | #kafkameetup | @CONFLUENTINC How easy is to change

    a etl code because the data model changed?
  18. @riferrei | #kafkameetup | @CONFLUENTINC 18 workarounds

  19. @riferrei | #kafkameetup | @CONFLUENTINC Another day... Another doug

  20. @riferrei | #kafkameetup | @CONFLUENTINC Databases to eliminate the need

    to process data! Database for large amounts of data!
  21. @riferrei | #kafkameetup | @CONFLUENTINC And lots of messaging!

  22. @riferrei | #kafkameetup | @CONFLUENTINC

  23. @riferrei | #kafkameetup | @CONFLUENTINC Let’s go back in time.

    Shaw we?
  24. @riferrei | #kafkameetup | @CONFLUENTINC

  25. @riferrei | #kafkameetup | @CONFLUENTINC

  26. 26 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to

    Scale No Persistence After Consumption No Replay Highly Scalable Durable Persistent Ordered Fast (Low Latency) What is happening in the world What happened in the world
  27. 27 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to

    Scale No Persistence After Consumption No Replay Highly Scalable Durable Persistent Ordered Fast (Low Latency) What is happening in the world What happened in the world Highly Scalable Durable Persistent Ordered Fast (Low Latency) Event Streaming Thinking
  28. @riferrei | #kafkameetup | @CONFLUENTINC Question: What is a distributed

    streaming platform?
  29. @riferrei | #kafkameetup | @CONFLUENTINC “The truth is the log.

    The database is a cache of a subset of the log.” — pat helland Immutability changes everything http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
  30. @riferrei | #kafkameetup | @CONFLUENTINC 01 Messaging done right 02

    Scalable data pipelines 03 Stream processing @riferrei | #kafkameetup | @CONFLUENTINC
  31. @riferrei | #kafkameetup | @CONFLUENTINC Messaging done right @riferrei |

    #kafkameetup | @CONFLUENTINC
  32. @riferrei | #kafkameetup | @CONFLUENTINC @riferrei | #kafkameetup | @CONFLUENTINC

    Scalable data pipelines
  33. @riferrei | #kafkameetup | @CONFLUENTINC @riferrei | #kafkameetup | @CONFLUENTINC

    Stream processing
  34. @riferrei | #kafkameetup | @CONFLUENTINC 34 demo

  35. @riferrei | #kafkameetup | @CONFLUENTINC

  36. Join The fun

  37. @riferrei | #kafkameetup | @CONFLUENTINC Question: is apache kafka a

    database?
  38. @riferrei | #kafkameetup | @CONFLUENTINC It can be!

  39. @riferrei | #kafkameetup | @CONFLUENTINC Have you EVER HEARD ABOUT

    THE #ASKCONFLUENT INITIATIVE?
  40. @riferrei | #kafkameetup | @CONFLUENTINC There was an idea…

  41. @riferrei | #kafkameetup | @CONFLUENTINC of Bringing TOGETHER A GROUP

    OF REMARKABLE PEOPLE
  42. @riferrei | #kafkameetup | @CONFLUENTINC THAT COULD ANSWER THE QUESTIONS…

  43. @riferrei | #kafkameetup | @CONFLUENTINC THAT we NEVER COULD...

  44. @riferrei | #kafkameetup | @CONFLUENTINC @tlberglund @gwenshap

  45. @riferrei | #kafkameetup | @CONFLUENTINC https://www.youtube.com/playlist?list= PLa7VYi0yPIH0snucuYWkuUXwasMr-HR7Y Kafka is so

    Cool!
  46. @riferrei | #kafkameetup | @CONFLUENTINC Question: IS THERE SUCH A

    THING OF OVERSUBSCRIPING KAFKA?
  47. @riferrei | #kafkameetup | @CONFLUENTINC Question: WHAT IS MOST COSTLY...

    UP CONVERT OR DOWN CONVERT?
  48. @riferrei | #kafkameetup | @CONFLUENTINC Question: HOW MANY PARTITIONS A

    TOPIC CAN HAVE?
  49. @riferrei | #kafkameetup | @CONFLUENTINC Question: WHAT ARE THE PROS

    AND CONS OF THE EXACTLY-ONCE FEATURE?
  50. @riferrei | #kafkameetup | @CONFLUENTINC Question: WHAT IS THE IMPACT

    OF CONSUMER APPS BEING DOWN FOR 4 DAYS?
  51. @riferrei | #kafkameetup | @CONFLUENTINC Question: NOT USING KEYS WHILE

    PRODUCING RECORDS IS A BAD PRACTICE?
  52. @riferrei | #kafkameetup | @CONFLUENTINC

  53. None