Link
Embed
Share
Beginning
This slide
Copy link URL
Copy link URL
Copy iframe embed code
Copy iframe embed code
Copy javascript embed code
Copy javascript embed code
Share
Tweet
Share
Tweet
Slide 1
Slide 1 text
Francesco Tisiot - Developer Advocate @ftisiot - @aiven_io JDBC Source Connector What Could Go Wrong?
Slide 2
Slide 2 text
@ftisiot | @aiven_io
Slide 3
Slide 3 text
@ftisiot | @aiven_io
Slide 4
Slide 4 text
@ftisiot | @aiven_io
Slide 5
Slide 5 text
@ftisiot | @aiven_io
Slide 6
Slide 6 text
@ftisiot | @aiven_io
Slide 7
Slide 7 text
@ftisiot | @aiven_io
Slide 8
Slide 8 text
@ftisiot | @aiven_io Kafka Connect
Slide 9
Slide 9 text
@ftisiot | @aiven_io JDBC Source Connector
Slide 10
Slide 10 text
@ftisiot | @aiven_io List of Tables Polling Interval Query Mode
Slide 11
Slide 11 text
@ftisiot | @aiven_io Bulk Mode
Slide 12
Slide 12 text
@ftisiot | @aiven_io Incremental Mode WHERE ID > 4 WHERE ID > 6
Slide 13
Slide 13 text
@ftisiot | @aiven_io WHERE TS > 10.03 WHERE TS > 10.05 Timestamp Mode
Slide 14
Slide 14 text
@ftisiot | @aiven_io Query Mode WHERE COL = △
Slide 15
Slide 15 text
@ftisiot | @aiven_io Problems
Slide 16
Slide 16 text
@ftisiot | @aiven_io Which JDBC Connector ? https:/ /github.com/aiven/jdbc-connector-for-apache-kafka
Slide 17
Slide 17 text
@ftisiot | @aiven_io Common Challenges Data Types Out of Memory Errors Number Mapping numeric.mapping defaultRowFetchSize
Slide 18
Slide 18 text
@ftisiot | @aiven_io ERROR java.lang.IllegalArgumentException: Number of groups must be positive table.types
Slide 19
Slide 19 text
@ftisiot | @aiven_io Everything is Fine Not
Slide 20
Slide 20 text
@ftisiot | @aiven_io Fast Events
Slide 21
Slide 21 text
@ftisiot | @aiven_io Polling Interval State Event
Slide 22
Slide 22 text
@ftisiot | @aiven_io Ghost Events
Slide 23
Slide 23 text
@ftisiot | @aiven_io 1 2 3 Id Name 1 2 3 Incremental = No Updates!
Slide 24
Slide 24 text
@ftisiot | @aiven_io Name Change Timestamp 10:00 10:01 10:02 10:03 10:00 10:01 10:02 10:03 No Hard Deletes!
Slide 25
Slide 25 text
@ftisiot | @aiven_io Out of Order Events
Slide 26
Slide 26 text
@ftisiot | @aiven_io Name Change Timestamp 10:00 10:01 10:03 10:03 10:00 10:01 10:03 10:02
Slide 27
Slide 27 text
@ftisiot | @aiven_io Why? Device Clock Network Lag Transaction Duration Batching
Slide 28
Slide 28 text
@ftisiot | @aiven_io Polling Interval 10:01 10:02 10:02 Polling Interval 10:02 > 10:01
Slide 29
Slide 29 text
@ftisiot | @aiven_io Polling Interval timestamp.delay.interval.ms Delay
Slide 30
Slide 30 text
@ftisiot | @aiven_io JDBC Limits Polling Time Out of Order Events Load on the DB Updates/Deletions Require Extra Fields
Slide 31
Slide 31 text
@ftisiot | @aiven_io Log Based Approach
Slide 32
Slide 32 text
@ftisiot | @aiven_io Write Ahead Log - PostgreSQL binlog - MySQL oplog - MongoDB
Slide 33
Slide 33 text
@ftisiot | @aiven_io Debezium Connector
Slide 34
Slide 34 text
@ftisiot | @aiven_io Video
Slide 35
Slide 35 text
@ftisiot | @aiven_io Video
Slide 36
Slide 36 text
@ftisiot | @aiven_io JDBC Limits Polling Time Out of Order Events Load on the DB Updates/Deletions Require Extra Fields All Events Near Real Time Tracked as per Log Minimal Load No Extra Fields
Slide 37
Slide 37 text
@ftisiot | @aiven_io Additional Benefit - Enhanced Metadata!
Slide 38
Slide 38 text
@ftisiot | @aiven_io Timestamps Pre-Post status Operation Type Sequence Number
Slide 39
Slide 39 text
@ftisiot | @aiven_io
Slide 40
Slide 40 text
@ftisiot | @aiven_io JDBC Debezium
Slide 41
Slide 41 text
@ftisiot | @aiven_io https:/ /aiven.io Debezium Connector JDBC Source Connector in Action Debezium Connector in Action JDBC Connector https:/ /ftisiot.net/talks/kafka-jdbc-what-can-go-wrong/ kafka-summit-2022 500$