Change Data Capture with Debezium @ Java Vienna - June 2024, Austria

Slide 1

Slide 1 text

Change Data Capture with Debezium ☕ Java Vienna 🎡 | 03. Juni 2024 Hans-Peter Grahsl @hpgrahsl Developer 🥑 Advocate Red Hat

Slide 2

Slide 2 text

debezium.io Hans-Peter Grahsl “I’m passionate about event-driven architectures, distributed stream processing systems and data engineering.” Developer 🥑 Advocate @ Red Hat based in Graz, Austria 󰎈 open-source enthusiast Confluent Community Catalyst MongoDB Champion since 2020 speaker at dev & tech conferences @hpgrahsl@mastodon.social

Slide 3

Slide 3 text

debezium.io Agenda ● What is Debezium and Change Data Capture (CDC)? ● Benefits of Log-based CDC ● Databases supported by Debezium ● Change Event Payload Structure ● Debezium deployment modes ● 🎬 Debezium in Action 🎬

Slide 4

Slide 4 text

debezium.io Debezium Introduction

Slide 5

Slide 5 text

debezium.io

Slide 6

Slide 6 text

debezium.io Debezium in a Nutshell ● fully open-source + very active community ● change data capture (CDC) platform ○ based on transaction logs ○ snapshotting, filtering, routing, flattening etc. ○ web-based UI ● large production deployments

Slide 7

Slide 7 text

debezium.io ● 10 databases ○ relational + non-relational ○ 8 production-ready ○ 2 incubating: Vitess + Informix ● + JDBC sink connector Currently supported Databases

Slide 8

Slide 8 text

debezium.io ● 3 “categories” of connectors ○ core ○ community-led ○ external ● 🎊 DB support is steadily growing 🤩 Currently supported Databases

Slide 9

Slide 9 text

debezium.io Benefits of Log-based CDC data model agnostic get additional meta-data no changes missed access to previous state low latency & little overhead captures delete operations

Slide 10

Slide 10 text

debezium.io Change Event Payload Structure ● Message Key: table’s primary key ● Message Value: ○ old & new data state ○ meta-data on table, TX id, etc. ○ operation type, timestamp ● Serialization → JSON, Avro, … ● Cloud Events spec also supported

Slide 11

Slide 11 text

Slide 12

Slide 12 text

debezium.io 3 Deployment Modes for Debezium Mode 1: Apache Kafka + Kafka Connect

Slide 13

Slide 13 text

debezium.io Mode 2: Debezium Server 3 Deployment Modes for Debezium

Slide 14

Slide 14 text

debezium.io Mode 3: Debezium Embedded Engine 3 Deployment Modes for Debezium

Slide 15

Slide 15 text

debezium.io 🎬 Debezium in Action 🎬

Slide 16

Slide 16 text

debezium.io

Slide 17

Slide 17 text

debezium.io

Slide 18

Slide 18 text

debezium.io Try it → dn.dev/debezium-tutorial

Slide 19

Slide 19 text

debezium.io Debezium enables various Use Cases 🤩 All of these and many more can be powered by change data capture and Debezium! 🤩

Slide 20

Slide 20 text

debezium.io Further CDC-related materials ● What is change data capture (CDC)? https://www.redhat.com/en/topics/integration/what-is-change-data-capture ● Using Change Data Capture for Stack Modernization https://www.solutionpatterns.io/solution-pattern-modernization-cdc/solution-pattern-mod ernization-cdc/index.html ● Use Case: Emerging Disease Detection https://validatedpatterns.io/patterns/emerging-disease-detection/

Slide 21

Slide 21 text

debezium.io Developer Resources: Books, Tutorials,... https://developers.redhat.com/eventtutorials

Slide 22

Slide 22 text

debezium.io Thanks ! Questions ?