Data Contracts In Practice With Debezium and Apache Flink (Kafka Summit London)

Log-based change data capture (CDC) is an invaluable part of the data engineering toolbox: it enables a variety of use cases such as real-time analytics, full-text search, or cache invalidation by publishing data change events from your database. But when publishing change event streams across context or team boundaries, aren’t you tieing external consumers to your application’s data model, thus limiting yourself in evolving the same?

Enter data contracts—consciously designed abstractions between your internal data model and the outside world. Come and join us for this session to learn about:

- Challenges you may encounter when exposing table level change event streams and how data contracts can mitigate them
- Implementation strategies for data contracts such as the outbox pattern and stream processing
- Evolving your data model and the corresponding data contracts, without breaking any existing consumers

We’ll also touch on some advanced topics at the intersection of CDC and stream processing, such as hydrating partial change events, using the popular change stream processing duo of Debezium and Apache Flink.

Transcript

1. Image © Benjamin White https://flic.kr/p/2iGM2x1 (CC BY 2.0 DEED) Gunnar Morling @gunnarmorling Data Contracts In Practice With Debezium and Apache Flink
2. Data Contracts With Debezium + Apache Flink | @gunnarmorling Today’s Mission Learn About…
3. Data Contracts With Debezium + Apache Flink | @gunnarmorling • Software engineer at Decodable • Former project lead of Debezium • kcctl 🧸, JfrUnit, ModiTect, MapStruct • Java Champion • 1⃣ 🐝 🏎 Gunnar Morling
4. The observer pattern for your database
5. Data Contracts With Debezium + Apache Flink | @gunnarmorling Debezium Log-Based Change Data Capture
6. Data Contracts With Debezium + Apache Flink | @gunnarmorling Change Data Capture Liberation for Your Data https://www.decodable.co/blog/seven-ways-to-put-cdc-to-work
7. Data Contracts With Debezium + Apache Flink | @gunnarmorling The De-Facto CDC Standard https://cloud.google.com/blog/products/databases/stream-data-changes-from-spanner-at-scale-with-apache-kafka Debezium
8. Data Contracts With Debezium + Apache Flink | @gunnarmorling Debezium: Data Change Events
9. Data Contracts With Debezium + Apache Flink | @gunnarmorling Does Change Data Capture Break Encapsulation? Chris Riccomini (https://cnr.sh/essays/kafka-change-data-capture-breaks-database-encapsulation) 🤔
10. Data Contracts With Debezium + Apache Flink | @gunnarmorling (Potential) Concerns? Your Table Model Becomes Your API • Names and types directly exposed • Particularly problematic for legacy schemas Image © massmatt https://flic.kr/p/25eF9D3 (CC BY 2.0)
11. Data Contracts With Debezium + Apache Flink | @gunnarmorling (Potential) Concerns? Fine-grained Events • 1:1 relationship between tables and event streams • May be too fine-grained Image © Michele Dorsey Walfred https://flic.kr/p/MDCCP4 (CC BY 2.0 DEED)
12. Data Contracts With Debezium + Apache Flink | @gunnarmorling • Renaming columns • Changing types • Removing columns • Changing cardinality of associations (Potential) Concerns? Schema Changes Might Break Things Image © Insights Unspoken https://flic.kr/p/zNTwN9 (CC BY 2.0 DEED)
13. Data Contracts With Debezium + Apache Flink | @gunnarmorling • Exposing all columns… • …and rows (Potential) Concerns? Accidental Data Leaks Image © Leonid Mamchenkov https://flic.kr/p/qzBLy (CC BY 2.0 DEED)
14. Data Contracts With Debezium + Apache Flink | @gunnarmorling Houston, we have a problem! Image © Jeff Hitchcock https://flic.kr/p/2hN4RG7 (CC BY 2.0 DEED)
15. Data Contracts With Debezium + Apache Flink | @gunnarmorling Is It Really a Problem, Though?
16. Data Contracts R. D. Barry https://flic.kr/p/P5RWWR (CC BY-SA 2.0 DEED)
17. Data Contracts With Debezium + Apache Flink | @gunnarmorling Data Contracts A data contract is a document that defines the structure, format, semantics, quality, and terms of use for exchanging data between a data provider and their consumers.
18. Data Contracts With Debezium + Apache Flink | @gunnarmorling Data Contracts Towards Data Products • Documentation of intent • Owned and evolved by the publisher Image © Jerome Vial https://flic.kr/p/71KpZy (CC BY-SA 2.0 DEED)
19. Data Contracts With Debezium + Apache Flink | @gunnarmorling Data Contracts datacontract.com
20. Outbox Pattern
21. Data Contracts With Debezium + Apache Flink | @gunnarmorling One Possible Solution The Outbox Pattern
22. Data Contracts With Debezium + Apache Flink | @gunnarmorling Variation on Postgres pg_logical_emit_message() • Directly writing arbitrary messages to the WAL • No need for an outbox table
23. Stream Processing Colin Howley https://flic.kr/p/698F5j (CC BY-ND 2.0)
24. Data Contracts With Debezium + Apache Flink | @gunnarmorling Apache Flink Stateful Computations over Data Streams https://flink.apache.org/
25. Data Contracts With Debezium + Apache Flink | @gunnarmorling Apache Flink APIs for Application Development Image source: “Change Data Capture with Flink SQL and Debezium” by Marta Paes at DataEngBytes (https://noti.st/morsapaes/liQzgs/change-data-capture-with-flink-sql-and-debezium)
26. Data Contracts With Debezium + Apache Flink | @gunnarmorling Example
27. Data Contracts With Debezium + Apache Flink | @gunnarmorling Example
28. Data Contracts With Debezium + Apache Flink | @gunnarmorling Example
29. Data Contracts With Debezium + Apache Flink | @gunnarmorling There’s More… SQL All the Things! • Filters • Derived fields • Consumer-specific contracts
30. Beyond the Basics alvaroreguly https://flic.kr/p/yJVjVY (CC BY 2.0 DEED)
31. Data Contracts With Debezium + Apache Flink | @gunnarmorling Streaming Data Contracts Joining Two Tables
32. Data Contracts With Debezium + Apache Flink | @gunnarmorling Streaming Data Contracts Denormalizing Data
33. Data Contracts With Debezium + Apache Flink | @gunnarmorling Joining Two Streams
34. Data Contracts With Debezium + Apache Flink | @gunnarmorling Nested Data Structures UDFs to the Rescue
35. Data Contracts With Debezium + Apache Flink | @gunnarmorling Nested Data Structures UDFs to the Rescue https://www.youtube.com/@decodable
36. Data Contracts With Debezium + Apache Flink | @gunnarmorling Expanding Partial Events Postgres TOAST Columns
37. Data Contracts With Debezium + Apache Flink | @gunnarmorling Expanding Partial Events
38. Schema Changes Bureau of Land Management Oregon and Washington https://flic.kr/p/26mD2nW (CC BY 2.0 DEED)
39. Data Contracts With Debezium + Apache Flink | @gunnarmorling Schema Evolution Producer-driven Changes
40. Data Contracts With Debezium + Apache Flink | @gunnarmorling Schema Changes Renaming a Column
41. Data Contracts With Debezium + Apache Flink | @gunnarmorling Renaming a Column How To Do It
42. Data Contracts With Debezium + Apache Flink | @gunnarmorling Renaming a Column How To Do It
43. Data Contracts With Debezium + Apache Flink | @gunnarmorling Wrap-Up
44. Data Contracts With Debezium + Apache Flink | @gunnarmorling • Debezium: Real-time change event streams for your data • Apache Flink: Data contracts for… ◦ …encapsulating internal models ✅ ◦ …consciously designed events ✅ ◦ …ensuring compatibility ✅ ◦ …protecting sensitive data ✅ Take Aways 🤩
45. Data Contracts With Debezium + Apache Flink | @gunnarmorling Houston, we may have a problem. If we do, we know how to solve it! Image © NASA Hubble Space Telescope https://flic.kr/p/22tV2DJ (CC BY 2.0 DEED)
47. Data Contracts With Debezium + Apache Flink | @gunnarmorling • Blog post: https://www.decodable.co/blog/change-data-capture-breaks-en capsulation-does-it-though • Example source code: github.com/decodableco/examples → cdc-data-contracts Learn More
48. Data Contracts With Debezium + Apache Flink | @gunnarmorling • ⏰ Today 6pm • 📍 South Gallery Room 200, CentrEd at ExCeL London Open Source Data Infrastructure Meetup Lightning Talks on Apache Kafka
49. Data Contracts With Debezium + Apache Flink | @gunnarmorling gunnar@decodable.co @gunnarmorling 📧 Thank You! Q & A