SG Kafka Meetup - March 2022 : Incremental migration using CDC and KSQL

Slide 1

Slide 1 text

Incremental Migration using CDC & KSQL Zabeer Farook & Kaleeswaran Karuppusamy CREDIT AGRICOLE CIB, Singapore By Singapore Apache Kafka Meetup

Slide 2

Slide 2 text

Zabeer Farook Working as a Technical Architect in Credit Agricole CIB, Singapore. Experience working with Kafka & KSQL based data pipelines. https://sg.linkedin.com/in/zabeer-farook 2

Slide 3

Slide 3 text

Kaleeswaran Karuppusamy Working as a Technical Architect in Credit Agricole CIB, Singapore. Experience in migration & modernization of applications as well as Kafka based data pipelines. 3 https://www.linkedin.com/in/kaleeswaran-karuppusamy

Slide 4

Slide 4 text

Agenda ◎ General Migration Strategies ◎ Role of CDC in incremental migration ◎ Know more about CDC pipeline with KSQL & Kafka Streams ◎ Demo of a CDC pipeline with KSQL 4

Slide 5

Slide 5 text

5 Legacy systems are complex and migrations aren’t easy either

Slide 6

Slide 6 text

6 Big bang migrations are too risky

Slide 7

Slide 7 text

Incremental Migration a Better Option ◎More sustainable ◎Less Risky ◎Module by Module ◎Read workloads First, Write workloads Next ◎Combination of the above 7

Slide 8

Slide 8 text

Let’s review the migration strategies in general 8

Slide 9

Slide 9 text

9 6R’s Migration Strategy

Slide 10

Slide 10 text

10 Incremental Migration – Strangler Pattern

Slide 11

Slide 11 text

11 Incremental Migration – Legacy App with Mainframe

Slide 12

Slide 12 text

12 Data pipeline acts as backbone for incremental migration

Slide 13

Slide 13 text

What is CDC (Change Data Capture) ? ◎ Captures event streams for any database changes ◎ Change events typically sourced from DB Txn Logs ◎ Events further streamed/shipped to target typically via messaging infrastructure 13 SOURCE DATABASE MESSAGE BUS TARGET DATABASE EVENT STREAMING CDC RELAY SERVICE TAIL

Slide 14

Slide 14 text

Types of CDC ◎ Query/Poll based ◎ Trigger based ◎ Log based 14

Slide 15

Slide 15 text

“ CDC liberalizes data to realize endless opportunities 15 ◎ DB migration ◎ Audit/History event capturing ◎ Analytics data store ◎ Index based search ◎ Cache Update/Invalidation ◎ Update CQRS Read Models ◎ Outbox Microservice Pattern Other Use Cases of CDC

Slide 16

Slide 16 text

Some CDC Tools out there.. 16

Slide 17

Slide 17 text

17 Typical CDC Pipeline with Debezium, KSQL & Kafka Streams 17

Slide 18

Slide 18 text

18 Role of KSQL/Kafka Streams in a data pipeline Transformation Filtering Aggregation Lookup

Slide 19

Slide 19 text

19 KSQLDB Vs Kafka Streams

Slide 20

Slide 20 text

Demo Time 20 https://github.com/Zabi82/confluent-meetup-mar2022

Slide 21

Slide 21 text

21 Demo Usecase

Slide 22

Slide 22 text

Resources Confluent Kafka : https://developer.confluent.io/ Debezium : https://debezium.io/documentation/ Demo Repo: https://github.com/Zabi82/confluent-meetup-mar2022 22

Slide 23

Slide 23 text

Thanks.. Any questions? 23