Open-Source Change Data Capture With Debezium Gunnar Morling Software Engineer, Red Hat @gunnarmorling

#Debezium @gunnarmorling Today’s Objectives Learn About…

#Debezium @gunnarmorling ● Open source software engineer at Red Hat ○ Debezium ○ Quarkus ● Spec Lead for Bean Validation 2.0 ● kcctl, ModiTect, MapStruct ● Java Champion ● @gunnarmorling Gunnar Morling

#Debezium @gunnarmorling ● Taps into TX log to capture INSERT/UPDATE/DELETE events ● Propagated to consumers via Apache Kafka and Kafka Connect Debezium — Log-based Change Data Capture

#Debezium @gunnarmorling Change Data Capture A Giant Enabler for Your Data

#Debezium @gunnarmorling Debezium in a Nutshell ● A CDC Platform ■ Based on transaction logs ■ Snapshotting, filtering, etc. ■ Outbox support ■ Web-based UI ● Fully open-source, very active community ● Large production deployments

#Debezium @gunnarmorling Debezium: Connectors ● Stable ■ MySQL ■ Postgres ■ MongoDB ■ SQL Server ■ Db2 ■ Oracle ● Incubating ■ Vitess ■ Cassandra

#Debezium @gunnarmorling Debezium: Connectors Becoming the De-Facto CDC Standard https://debezium.io/blog/2021/09/22/deep-dive-into-a-debezium-community-connector-scylla-cdc-source-connector/

#Debezium @gunnarmorling Debezium Architecture

#Debezium @gunnarmorling Debezium: Deployment Alternatives Embedded Engine and Debezium Server

#Debezium @gunnarmorling Data Change Events ● Old and new row state ● Metadata on table, TX id, etc. ● Operation type, timestamp

#Debezium @gunnarmorling ● Old and new row state ● Metadata on table, TX id, etc. ● Operation type, timestamp Data Change Events

#Debezium @gunnarmorling ● Old and new row state ● Metadata on table, TX id, etc. ● Operation type, timestamp Data Change Events

Outbox Pattern

#Debezium @gunnarmorling ● Services need to update their database, ● send messages to other services, ● and that consistently! Challenge: Microservices Data Exchange

#Debezium @gunnarmorling “Dual writes” are prone to inconsistencies! Outbox Pattern

#Debezium @gunnarmorling Outbox Pattern

#Debezium @gunnarmorling Outbox Pattern

#Debezium @gunnarmorling Outbox Pattern

#Debezium @gunnarmorling Outbox Pattern

#Debezium @gunnarmorling Variation on Postgres pg_logical_emit_message() ● Directly writing arbitrary messages to the WAL ● No need for an outbox table

#Debezium @gunnarmorling pg_logical_emit_message() Exporting auditing metadata ● Pure CDC events lack metadata like business user, device id, etc. ● Solution: emit at TX begin, enrich events e.g. using SMT

#Debezium @gunnarmorling Directly Emitting WAL Events The Ask Provide a facility for producing raw WAL events

Challenges

#Debezium @gunnarmorling Keeping Track of Table Schemas How to Interpret Incoming Events? ● Messages typically not self-descriptive ● Challenge: incoming events may adhere to earlier schema version

#Debezium @gunnarmorling MySQL DDL Parser Solution: Parse DDL Events ● Based on Antlr parser generator

#Debezium @gunnarmorling Recovering schema after restarts ● Persisting schema change history in a Kafka topic

#Debezium @gunnarmorling Keeping Track of Table Schemas The Ask ● Efficient is good, but make it simple to consume ● Provide it when needed, as e.g. in Postgres (pgoutput) ○ At the beginning of session ○ After a table change ● Facility to query past schema versions

#Debezium @gunnarmorling On the Subject of Parsing… LogMiner Events

#Debezium @gunnarmorling Preventing Unbounded WAL Growth (I) Challenging API designs

#Debezium @gunnarmorling Preventing Unbounded WAL Growth (I) Can’t Commit Offsets Without Events

#Debezium @gunnarmorling Preventing Unbounded WAL Growth (II) High-traffic/Low-traffic Logical Databases ● Problem: ○ WAL global ○ Logical replication slots per database

#Debezium @gunnarmorling Preventing Unbounded WAL Growth The Ask Make sure interfaces work correctly also in corner cases

Snapshotting

#Debezium @gunnarmorling Snapshotting General Idea ● Need initial backfill of sink systems, but don’t have all TX logs ● Solution: scan data once before streaming

#Debezium @gunnarmorling Snapshotting The Ask Allow for consistent, lock-less snapshots

#Debezium @gunnarmorling Snapshotting Limitations of Classic Approach ● Can’t update filter list ● Long-running snapshots can’t be paused/resumed ● Can’t stream changes until snapshot completed ● Can’t re-snapshot selected tables

#Debezium @gunnarmorling Snapshotting Incremental Snapshotting ● “DBLog: A Watermark Based Change-Data-Capture Framework”, by Andreas Andreakis and Ioannis Papapanagiotou ● Key idea: interleave snapshot events and events from TX log https://arxiv.org/pdf/2010.12597v1.pdf

#Debezium @gunnarmorling Snapshotting Incremental Snapshotting

#Debezium @gunnarmorling Incremental Snapshotting Windowing via Watermarks

#Debezium @gunnarmorling Incremental Snapshotting Buffer Processing

#Debezium @gunnarmorling Incremental Snapshotting Buffer Processing

#Debezium @gunnarmorling Incremental Snapshotting Semantics ● No guarantee for snapshot (read) events for all records ● May receive update or delete without prior insert/read ● May receive read and update/delete ● What is guaranteed: complete data set after snapshot

#Debezium @gunnarmorling Incremental Snapshotting Comparison ● Can’t update filter list ✅ ● Long-running snapshots can’t be paused/resumed ✅ ● Can’t stream changes until snapshot completed ✅ ● Can’t re-snapshot selected tables ✅

Wrap-Up

#Debezium @gunnarmorling ● CDC: SELECT, INSERT, UPDATE, DELETE… STREAM? ● Debezium: open-source CDC for a variety of databases ● Outlook: incrementally updated materialized views? Takeaways

#Debezium @gunnarmorling ● Debezium https://debezium.io/ ● Incremental snapshotting https://debezium.io/blog/2021/10/07/incremental-snapshots/ ● Outbox implementation https://debezium.io/blog/2019/02/19/reliable-microservices-data -exchange-with-the-outbox-pattern/ ● Demo repo https://github.com/debezium/debezium-examples Resources

#Debezium @gunnarmorling Q & A [email protected] @gunnarmorling 📧 Thank You!

#Debezium @gunnarmorling Unsplash https://unsplash.com/license © Pablo García Saldaña https://unsplash.com/photos/lPQIndZz8Mo © David Clode https://unsplash.com/photos/T49WTav4LgU © Aaron Burden https://unsplash.com/photos/GFpxQ2ZyNc0 © Nathan Dumlao https://unsplash.com/photos/wQDysNUCKfw © mari lezhava https://unsplash.com/photos/q65bNe9fW-w © Michał Parzuchowski https://unsplash.com/photos/Bt0PM7cNJFQ © Charles Forerunner https://unsplash.com/photos/3fPXt37X6UQ Flickr Attribution 2.0 Generic https://creativecommons.org/licenses/by/2.0/ © Thomas Kamann https://flic.kr/p/coa2c CC0 1.0 Universal Public Domain Dedication https://creativecommons.org/publicdomain/zero/1.0/ © Wall Boat https://flic.kr/p/Y6zkmX Attribution-ShareAlike 2.0 Generic https://creativecommons.org/licenses/by-sa/2.0/ © Andrew Hart https://flic.kr/p/dmjkSk Attribution 2.0 Generic (CC BY 2.0) https://creativecommons.org/licenses/by/2.0/ © Ryan https://flic.kr/p/8gwtzo Image Credits In Order of Appearance