Apache Kafka JDBC Source Connector: What could go wrong?

Slide 1

Slide 1 text

Francesco Tisiot - Developer Advocate @ftisiot - @aiven_io JDBC Source Connector What Could Go Wrong?

Slide 2

Slide 2 text

@ftisiot | @aiven_io

Slide 3

Slide 3 text

@ftisiot | @aiven_io

Slide 4

Slide 4 text

@ftisiot | @aiven_io

Slide 5

Slide 5 text

@ftisiot | @aiven_io

Slide 6

Slide 6 text

@ftisiot | @aiven_io

Slide 7

Slide 7 text

@ftisiot | @aiven_io

Slide 8

Slide 8 text

@ftisiot | @aiven_io Kafka Connect

Slide 9

Slide 9 text

@ftisiot | @aiven_io JDBC Source Connector

Slide 10

Slide 10 text

@ftisiot | @aiven_io List of Tables Polling Interval Query Mode

Slide 11

Slide 11 text

@ftisiot | @aiven_io Bulk Mode

Slide 12

Slide 12 text

@ftisiot | @aiven_io Incremental Mode WHERE ID > 4 WHERE ID > 6

Slide 13

Slide 13 text

@ftisiot | @aiven_io WHERE TS > 10.03 WHERE TS > 10.05 Timestamp Mode

Slide 14

Slide 14 text

@ftisiot | @aiven_io Query Mode WHERE COL = △

Slide 15

Slide 15 text

@ftisiot | @aiven_io Problems

Slide 16

Slide 16 text

@ftisiot | @aiven_io Which JDBC Connector ? https:/ /github.com/aiven/jdbc-connector-for-apache-kafka

Slide 17

Slide 17 text

@ftisiot | @aiven_io Common Challenges Data Types Out of Memory Errors Number Mapping numeric.mapping defaultRowFetchSize

Slide 18

Slide 18 text

@ftisiot | @aiven_io ERROR java.lang.IllegalArgumentException: Number of groups must be positive table.types

Slide 19

Slide 19 text

@ftisiot | @aiven_io Everything is Fine Not

Slide 20

Slide 20 text

@ftisiot | @aiven_io Fast Events

Slide 21

Slide 21 text

@ftisiot | @aiven_io Polling Interval State Event

Slide 22

Slide 22 text

@ftisiot | @aiven_io Ghost Events

Slide 23

Slide 23 text

@ftisiot | @aiven_io 1 2 3 Id Name 1 2 3 Incremental = No Updates!

Slide 24

Slide 24 text

@ftisiot | @aiven_io Name Change Timestamp 10:00 10:01 10:02 10:03 10:00 10:01 10:02 10:03 No Hard Deletes!

Slide 25

Slide 25 text

@ftisiot | @aiven_io Out of Order Events

Slide 26

Slide 26 text

@ftisiot | @aiven_io Name Change Timestamp 10:00 10:01 10:03 10:03 10:00 10:01 10:03 10:02

Slide 27

Slide 27 text

@ftisiot | @aiven_io Why? Device Clock Network Lag Transaction Duration Batching

Slide 28

Slide 28 text

@ftisiot | @aiven_io Polling Interval 10:01 10:02 10:02 Polling Interval 10:02 > 10:01

Slide 29

Slide 29 text

@ftisiot | @aiven_io Polling Interval timestamp.delay.interval.ms Delay

Slide 30

Slide 30 text

@ftisiot | @aiven_io JDBC Limits Polling Time Out of Order Events Load on the DB Updates/Deletions Require Extra Fields

Slide 31

Slide 31 text

@ftisiot | @aiven_io Log Based Approach

Slide 32

Slide 32 text

@ftisiot | @aiven_io Write Ahead Log - PostgreSQL binlog - MySQL oplog - MongoDB

Slide 33

Slide 33 text

@ftisiot | @aiven_io Debezium Connector

Slide 34

Slide 34 text

@ftisiot | @aiven_io Video

Slide 35

Slide 35 text

@ftisiot | @aiven_io Video

Slide 36

Slide 36 text

@ftisiot | @aiven_io JDBC Limits Polling Time Out of Order Events Load on the DB Updates/Deletions Require Extra Fields All Events Near Real Time Tracked as per Log Minimal Load No Extra Fields

Slide 37

Slide 37 text

@ftisiot | @aiven_io Additional Benefit - Enhanced Metadata!

Slide 38

Slide 38 text

@ftisiot | @aiven_io Timestamps Pre-Post status Operation Type Sequence Number

Slide 39

Slide 39 text

@ftisiot | @aiven_io

Slide 40

Slide 40 text

@ftisiot | @aiven_io JDBC Debezium

Slide 41

Slide 41 text

@ftisiot | @aiven_io https:/ /aiven.io Debezium Connector JDBC Source Connector in Action Debezium Connector in Action JDBC Connector https:/ /ftisiot.net/talks/kafka-jdbc-what-can-go-wrong/ kafka-summit-2022 500$