Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Track Database Changes with Apache Kafka

Avatar for FTisiot FTisiot
September 11, 2021
Technology
0
260

Track Database Changes with Apache Kafka

Tired of the spaghetti architecture created by hundreds applications and ETL flows dependencies making every database table change a matter of 100 people meeting? Understand how you can decouple and integrate better your systems with Apache Kafka and Kafka Connect!
Avatar for FTisiot

FTisiot

September 11, 2021
Tweet

More Decks by FTisiot

See All by FTisiot
Apache Kafka JDBC Source Connector: What could go wrong?
Avatar for FTisiot ftisiot
0
85
Solve the Knapsack problem with recursive queries and PostgreSQL
Avatar for FTisiot ftisiot
0
110
Build an Open Source Data Pipeline
Avatar for FTisiot ftisiot
0
280
Get to know Apache Kafka with Jupyter Notebooks
Avatar for FTisiot ftisiot
0
680
Fix Your Strings in PostgreSQL
Avatar for FTisiot ftisiot
0
250
Practical Tips & Tricks for Apache Kafka Messages Integration
Avatar for FTisiot ftisiot
0
190
Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL
Avatar for FTisiot ftisiot
0
920
Kickstart your Kafka with Faker Data
Avatar for FTisiot ftisiot
0
380
Event-Driven Applications: Apache Kafka and Python
Avatar for FTisiot ftisiot
0
920

Other Decks in Technology

See All in Technology
Winning at PHP in Production in 2025
Avatar for Benjamin Eberlei beberlei
1
270
日経電子版 for Android の技術的課題と取り組み(令和最新版)/android-20250423
Avatar for 日本経済新聞社 エンジニア採用事務局 nikkei_engineer_recruiting
2
640
Twelve-Factor-Appから学ぶECS設計プラクティス/ECS practice for Twelve-Factor-App
Avatar for k-ozawa ozawa
3
160
テストって楽しい！開発を加速させるテストの魅力 / Testing is Fun! The Fascinating of Testing to Accelerate Development
Avatar for END aiandrox
0
160
AndroidアプリエンジニアもMCPを触ろう
Avatar for Shinnosuke Kugimiya kgmyshin
2
610
非root化Androidスマホでも動く仮想マシンアプリを試してみた
Avatar for Sora Arakawa arkw
0
110
Part1　GitHubってなんだろう？その１
Avatar for tomokusaba tomokusaba
2
520
AOAI で AI アプリを開発する時にまず考えたいこと
Avatar for Yuji Masaoka | まっぴぃ mappie_kochi
1
410
コスト最適重視でAurora PostgreSQLのログ分析基盤を作ってみた #jawsug_tokyo
Avatar for のんピ non97
2
880
データベース04: SQL (1/3) 単純質問 & 集約演算
Avatar for Y. Yamamoto trycycle
PRO
0
720
ペアーズにおける評価ドリブンな AI Agent 開発のご紹介
Avatar for fukubaka0825 fukubaka0825
9
2.3k
本当に必要なのは「QAという技術」だった！試行錯誤から生まれた、品質とデリバリーの両取りアプローチ / Turns Out, "QA as a Discipline" Was the Key!
Avatar for ar_tama ar_tama
9
2.8k

Featured

See All Featured
The Language of Interfaces
Avatar for Des Traynor destraynor
158
25k
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
49
7.8k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
Avatar for Irina Nazarova irinanazarova
8
700
The World Runs on Bad Software
Avatar for Brandon Keepers bkeepers
PRO
68
11k
Making the Leap to Tech Lead
Avatar for Ryan Cromwell cromwellryan
133
9.3k
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
54
5.5k
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
52
7.6k
JavaScript: Past, Present, and Future - NDC Porto 2020
Avatar for David Neal reverentgeek
47
5.4k
StorybookのUI Testing Handbookを読んだ
Avatar for Shingo Yamazaki zakiyama
30
5.7k
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
48
49k
YesSQL, Process and Tooling at Scale
Avatar for Rocio Delgado rocio
172
14k
Reflections from 52 weeks, 52 projects
Avatar for Jefferson Lam jeffersonlam
349
20k

Transcript

  1. I don’t Want to Miss a Thing Track Database Changes

    with Apache Kafka Francesco Tisiot - Developer Advocate @ftisiot

  2. @ftisiot

  3. @ftisiot

  4. @ftisiot

  5. @ftisiot

  6. @ftisiot What is Apache Kafka?

  7. @ftisiot Topic A Topic B 0 1 2 3 4

    0 1 2 3 Producer Consumer Producer Consumer Consumer

  8. @ftisiot Events in a Database?

  9. @ftisiot Table Log vs The Fridge Dilemma

  10. @ftisiot Table Log 5 3 6 +7 +3 +6 -5

    -2 +5

  11. @ftisiot Integrating Apache Kafka

  12. @ftisiot Kafka Connect Source Kafka Connect Sink

  13. @ftisiot Changes in Kafka?

  14. @ftisiot Query Based Approach

  15. @ftisiot

  16. @ftisiot Log Based Approach

  17. @ftisiot

  18. @ftisiot Debezium

  19. @ftisiot

  20. @ftisiot Query Bulk Query Timestamp Log INSERT UPDATE DELETE NOTES

  21. @ftisiot Resources https://aiven.io https://debezium.io/ https://aiven.io/blog/using-kafka-connect-jdbc-source- a-postgresql-example https://kafka.apache.org/