Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Everything You Wanted to Know About Apache Kafka But You Were Too Afraid to Ask!

Everything You Wanted to Know About Apache Kafka But You Were Too Afraid to Ask!

Ricardo Ferreira

June 04, 2019
Tweet

More Decks by Ricardo Ferreira

Other Decks in Programming

Transcript

  1. 1 Everything you Wanted to Know about Apache Kafka But

    You Were Too Afraid to Ask Ricardo Ferreira Developer Advocate, Confluent
  2. About Us: • Ricardo Ferreira ❑ Developer Advocate @ Confluent

    ❑ Ex-Oracle, Red Hat, IONA Tech ❑ Currently ~70% Dev, ~30 Ops ❑ https://riferrei.net • Echo Dot (Alexa) ❑ The voice behind Amazon ❑ Ex-Raspberry Pi, Arduino @riferrei @alexa99
  3. @riferrei | @JNationConf | @confluentinc SQL DBs 25 years ago…

    SQL DBs Today Dude, you're embarrassing me in front of the wizards…
  4. @riferrei | @JNationConf | @confluentinc What did it cost to

    extract data from the transactional DB?
  5. @riferrei | @JNationConf | @confluentinc Solution for "Combining" Processing and

    Data: NoSQL Solution for Large Amounts of Data: Big Data!
  6. 19 Event-Driven App (Location Tracking) Only Real-Time Events Messaging Queues

    and Event Streaming Platforms can do this Contextual Event-Driven App (ETA) Real-Time combined with stored data Only Event Streaming Platforms can do this Where is my driver? When will my driver get here? Where is my driver? When will my driver get here? Why Combine Real-time With Historical Context? 2 min
  7. 20 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to

    Scale No Persistence After Consumption No Replay Highly Scalable Durable Persistent Ordered Fast (Low Latency) What is happening in the world What happened in the world
  8. 21 Highly Scalable Durable Persistent Maintains Order ETL/Data Integration Messaging

    ETL/Data Integration Messaging Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay Fast (Low Latency) What happened in the world What is happening in the world Highly Scalable Durable Persistent Ordered Fast (Low Latency) Event Streaming Thinking
  9. @riferrei | @JNationConf | @confluentinc Steps: 1. Make your question

    on Twitter using the hashtag #AskConfluent 2. Wait for the next episode
  10. @riferrei | @JNationConf | @confluentinc Question: "Is it More Costly

    to Up Convert or Down Convert Message Formats for Records Sent to Apache Kafka?"
  11. @riferrei | @JNationConf | @confluentinc Question: "'auto.offset.reset=earliest' was set but

    after restarting the app (it was 4 days down) it started to process from the beginning of the topic…"
  12. @riferrei | @JNationConf | @confluentinc Question: "Not Using Keys to

    Produce a Record is Considered a Bad Practice?"
  13. 42