Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Track Database Changes with Apache Kafka

FTisiot
September 11, 2021

Track Database Changes with Apache Kafka

Tired of the spaghetti architecture created by hundreds applications and ETL flows dependencies making every database table change a matter of 100 people meeting? Understand how you can decouple and integrate better your systems with Apache Kafka and Kafka Connect!

FTisiot

September 11, 2021
Tweet

More Decks by FTisiot

Other Decks in Technology

Transcript

  1. I don’t Want to Miss a Thing Track Database Changes

    with Apache Kafka Francesco Tisiot - Developer Advocate @ftisiot
  2. @ftisiot

  3. @ftisiot

  4. @ftisiot

  5. @ftisiot

  6. @ftisiot What is Apache Kafka?

  7. @ftisiot Topic A Topic B 0 1 2 3 4

    0 1 2 3 Producer Consumer Producer Consumer Consumer
  8. @ftisiot Events in a Database?

  9. @ftisiot Table Log vs The Fridge Dilemma

  10. @ftisiot Table Log 5 3 6 +7 +3 +6 -5

    -2 +5
  11. @ftisiot Integrating Apache Kafka

  12. @ftisiot Kafka Connect Source Kafka Connect Sink

  13. @ftisiot Changes in Kafka?

  14. @ftisiot Query Based Approach

  15. @ftisiot

  16. @ftisiot Log Based Approach

  17. @ftisiot

  18. @ftisiot Debezium

  19. @ftisiot

  20. @ftisiot Query Bulk Query Timestamp Log INSERT UPDATE DELETE NOTES

  21. @ftisiot Resources https://aiven.io https://debezium.io/ https://aiven.io/blog/using-kafka-connect-jdbc-source- a-postgresql-example https://kafka.apache.org/