Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Track Database Changes with Apache Kafka

A23789f299ed06fe7d9f1c6940440bfa?s=47 FTisiot
September 11, 2021

Track Database Changes with Apache Kafka

Tired of the spaghetti architecture created by hundreds applications and ETL flows dependencies making every database table change a matter of 100 people meeting? Understand how you can decouple and integrate better your systems with Apache Kafka and Kafka Connect!

A23789f299ed06fe7d9f1c6940440bfa?s=128

FTisiot

September 11, 2021
Tweet

Transcript

  1. I don’t Want to Miss a Thing Track Database Changes

    with Apache Kafka Francesco Tisiot - Developer Advocate @ftisiot
  2. @ftisiot

  3. @ftisiot

  4. @ftisiot

  5. @ftisiot

  6. @ftisiot What is Apache Kafka?

  7. @ftisiot Topic A Topic B 0 1 2 3 4

    0 1 2 3 Producer Consumer Producer Consumer Consumer
  8. @ftisiot Events in a Database?

  9. @ftisiot Table Log vs The Fridge Dilemma

  10. @ftisiot Table Log 5 3 6 +7 +3 +6 -5

    -2 +5
  11. @ftisiot Integrating Apache Kafka

  12. @ftisiot Kafka Connect Source Kafka Connect Sink

  13. @ftisiot Changes in Kafka?

  14. @ftisiot Query Based Approach

  15. @ftisiot

  16. @ftisiot Log Based Approach

  17. @ftisiot

  18. @ftisiot Debezium

  19. @ftisiot

  20. @ftisiot Query Bulk Query Timestamp Log INSERT UPDATE DELETE NOTES

  21. @ftisiot Resources https://aiven.io https://debezium.io/ https://aiven.io/blog/using-kafka-connect-jdbc-source- a-postgresql-example https://kafka.apache.org/