Modernizing Legacy Systems with Apache Kafka & CDC Pipelines

Transcript

1. Modernizing Legacy Systems with Apache Kafka and CDC Pipelines by Jack Yu - Full Stack Engineer (@Jack Yu) 1
2. Agenda Introduction Apache Kafka, Debezium... Legacy System: Mainframe, AIX System, COBOL... Modernizing Legacy Systems through CDC Pipelines & Apache Kafka Change Data Capture Event Summary Q & A 2
3. Apache Kafka Kafka Kafka Connect Kafka Streams Apache Kafka Doc 3
4. Debezium Debezium Source Connector for PostgreSQL Debezium Source Connector for MySQL Debezium Source Connector for SQL Server ... Debezium Connector Overview 4
5. Introduction To Our Company Local Mini Broker, founded in 1988 IT System accross multiple era such as Mainframe, COBOL, C/C++, Java, php, python, javascript... Self own on premise data center, multiple operating system such as AIX, Linux, Windows... 5
6. Our Legacy System IBM Mainframe IBM AIX System COBOL Vender designed specific programs unable to modify by ourself... MySQL 5.5, SQL Server 2008... PHP 5.3, Perl 5.8, C/C++... 6
8. COBOL is Still Alive ref: Quora: Is there any demand for COBOL programming nowadays? The answer is a bit unclear. There are still billions of lines of COBOL code handling the business of the 21st century and someone needs to maintain and update them. In many cases there is still new development being done (though new languages and architectures are increasingly used for development). From that perspective, yes there is demand for COBOL. “ “ 8
9. Problems New developer have no idea about COBOL Business logic hard to understand, lot's of dependency Difficult to share data between mainframe and other systems Each team have their own tech stack, data store hard to collaborate 9
10. Architecture -- Before 10
12. Architecture Data Input -- After 12
14. Architecture Data Output -- After 14
16. What is CDC? - Change Data Capture mysql> SELECT * FROM customers; +------+------------+-----------+-----------------------+ | id | first_name | last_name | email | +------+------------+-----------+-----------------------+ | 1001 | Sally | Thomas | sally.thomas@acme.com | | 1002 | George | Bailey | gbailey@foobar.com | | 1003 | Edward | Walker | ed@walker.com | | 1004 | Anne | Kretchmar | annek@noanswer.org | +------+------------+-----------+-----------------------+ UPDATE customers SET first_name='Anne Marie' WHERE id=1004; 16
17. Example CDC message generate from debezium { "before": { "id": 1004, "first_name": "Anne", "last_name": "Kretchmar", "email": "annek@noanswer.org" }, "after": { "id": 1004, "first_name": "Anne Marie", "last_name": "Kretchmar", "email": "annek@noanswer.org" }, ... } 17
18. Change Data Capture for COBOL key value style action=R,id=1002,first_name=George,last_name=Bailey,email=gbailey@foobar.com action=U,id=1004,first_name=Anne Marie,last_name=Kretchmar,email=annek@noanswer.org Binary style 0xbb20f55a5eecf669602cf0a8380f57de012634ebe0a35b4d0d35432d4c232e8d 0x36f00e344b59ddbe23ba0d32cf808c74a80b4e08f074907c44e3f84a8a866bf7 0x508f8405eeacbbb5b311a323287f489fd877815a801b57fab2b27b3d08b7e5af 18
19. Development of CDC Pipeline 19
21. Benefit of New Architecture Each team can focus on their own business logic, tech stack, data store... Easy to share data between different systems through Kafka Easy to scale & implement the system with modern technology (both scale up & scale out) Development could be parallel and more efficient, since each team have their own responsibility 21
22. Scalability & Flexibility Once the data is out from mainframe & COBOL, it's easy to scale & implement the system with modern technology k8s..., mongodb..., postgresql..., python..., javascript..., golang..., kafka..., redis..., elasticsearch..., prometheus..., grafana..., fluentd..., kibana..., ... 22
23. Q & A 23
24. Reference Stand up to the Dinosaur: Mainframe Integration, Offloading and Replacement with Apache Kafka Debezium Tutorial A Gentle Introduction to Event-driven Change Data Capture 24