Debezium: What's new (KafkaSummit 2021)

Overview of latest changes in Debezium, from a presentation at the Red Hat booth at Kafka Summit Americas 2021.

Transcript

1. 1
   Debezium - What’s new,
   what’s coming?
   Gunnar Morling
   

   View Slide

2. #Debezium
   ● Taps into TX log to capture INSERT/UPDATE/DELETE events
   ● Propagated to consumers via Apache Kafka and Kafka Connect
   Debezium — Log-based Change Data Capture
   

   View Slide

3. #Debezium
   ● CDC for multiple databases
   ○ Based on transaction logs
   ○ Snapshotting, Filtering etc.
   ● Fully open-source, very active community
   ● Latest version: 1.6
   ● Large production deployments at
   multiple companies
   Debezium
   Streaming Changes out of the Database
   

   View Slide

4. #Debezium
   Debezium
   Release Model
   2021 Q3
   2021 Q4
   Red Hat Integration
   Debezium 1.6
   2021 Q1
   Upstream
   Debezium 1.5
   2021 Q2
   Red Hat Integration
   Debezium 1.4
   Upstream
   Debezium 1.6
   Upstream
   Debezium 1.7
   Upstream
   Debezium 1.8
   Red Hat Integration
   Debezium 1.5
   

   View Slide

5. #Debezium
   ● Distributed Tracing
   ● Incubating Vitess connector
   (community-led)
   Debezium 1.4
   What’s New?
   

   View Slide

6. #Debezium
   

   View Slide

7. #Debezium
   ● Stable LogMiner-based connector for Oracle
   ● Transaction metadata topic for MySQL
   ● Support for Redis Streams in Debezium Server
   Debezium 1.5
   What’s New?
   

   View Slide

8. #Debezium
   ● Incremental snapshots
   ○ Resumeable
   ○ Re-snapshot selected tables
   ○ Modify filter config
   ○ Begin streaming while snapshotting
   ● Support for Pravega in Debezium Server
   Debezium 1.6
   What’s New?
   

   View Slide

9. #Debezium
   ● Debezium UI
   ● Stabilization and improvements to incremental snapshotting
   ● Schema history compaction
   ● Off-heap transaction buffer for the Oracle connector
   ● Multi-database support for the SQL Server connector
   ● Performance improvements
   Debezium Roadmap
   What’s Coming?
   

   View Slide

10. #Debezium
    ● Debezium Examples
    https://github.com/debezium/debezium-examples/
    ● Saga orchestration with Debezium and CDC
    https://www.infoq.com/articles/saga-orchestration-outbox/
    ● Getting started with Debezium in Red Hat Integration
    https://access.redhat.com/documentation/en-us/red_hat_integra
    tion/2021.q3/html/getting_started_with_debezium/index
    Resources
    

    View Slide