Change Data Streaming Patterns With Debezium & Apache Flink

Microservices are one of the big trends in software engineering of the last few years: organizing business functionality in several self-contained, loosely coupled services helps engineering teams to work efficiently, make the most suitable technical decisions, and react quickly to new business requirements.

In this session we'll discuss and showcase how open-source change data capture (CDC) with Debezium can help developers with typical challenges they often face when working on microservices. Come and join us to learn how to:

* Employ the outbox pattern for reliable, eventually consistent data exchange between microservices, without incurring unsafe dual writes or tight coupling
* Gradually extract microservices from existing monolithic applications, using CDC, the strangler fig pattern and Apache Flink
* Building auditing logs, containing not only the changed data itself, but also additional metadata like business user, client configuration, or use case identifier

Transcript

1. Image © pixelchecker https://flic.kr/p/ah5sPr (CC BY 2.0) Change Data Streaming Patterns With Debezium & Apache Flink Gunnar Morling Senior Staff Software Engineer, Decodable @gunnarmorling
2. The world is real-time. So should be your data.
4. #ChangeDataStreamingPatterns @gunnarmorling Today’s Mission Learn About…
5. #ChangeDataStreamingPatterns @gunnarmorling • Software engineer at Decodable • Former project lead of Debezium • kcctl 🧸, JfrUnit, ModiTect, MapStruct • Spec Lead for Bean Validation 2.0 • Java Champion Gunnar Morling
6. The Tools https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot
7. #ChangeDataStreamingPatterns @gunnarmorling Debezium Log-Based Change Data Capture • Taps into TX log to capture INSERT/UPDATE/DELETE events • Typically propagated to consumers via Apache Kafka
8. #ChangeDataStreamingPatterns @gunnarmorling Debezium Log-Based Change Data Capture • Taps into TX log to capture INSERT/UPDATE/DELETE events • Typically propagated to consumers via Apache Kafka
9. #ChangeDataStreamingPatterns @gunnarmorling Debezium Open-Source Change Data Capture • A CDC Platform ◦ Based on transaction logs ◦ Snapshotting, filtering, etc. ◦ Outbox support ◦ Web-based UI • Fully open-source, very active community • Large production deployments
10. #ChangeDataStreamingPatterns @gunnarmorling Debezium: Data Change Events • Old and new row state • Metadata on table, TX id, etc. • Operation type, timestamp
11. #ChangeDataStreamingPatterns @gunnarmorling Debezium: Data Change Events • Old and new row state • Metadata on table, TX id, etc. • Operation type, timestamp
12. #ChangeDataStreamingPatterns @gunnarmorling Debezium: Data Change Events • Old and new row state • Metadata on table, TX id, etc. • Operation type, timestamp
13. #ChangeDataStreamingPatterns @gunnarmorling Debezium Deployment Options
14. #ChangeDataStreamingPatterns @gunnarmorling Becoming the De-Facto CDC Standard Debezium Google Cloud Spanner ScyllaDB
15. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot Apache Flink
16. #ChangeDataStreamingPatterns @gunnarmorling Apache Flink Stateful Computations over Data Streams https://flink.apache.org/
17. #ChangeDataStreamingPatterns @gunnarmorling Apache Flink APIs for Application Development Image source: “Change Data Capture with Flink SQL and Debezium, by Marta Paes at DataEngBytes (https://noti.st/morsapaes/liQzgs/change-data-capture-with-flink-sql-and-debezium)
18. #ChangeDataStreamingPatterns @gunnarmorling Apache Flink Stream Processing of Change Data Events
19. #ChangeDataStreamingPatterns @gunnarmorling Apache Flink Stream Processing of Change Data Events
20. #ChangeDataStreamingPatterns @gunnarmorling Apache Flink Stream Processing of Change Data Events
21. #ChangeDataStreamingPatterns @gunnarmorling Apache Flink Stream Processing of Change Data Events
22. #ChangeDataStreamingPatterns @gunnarmorling Apache Flink Stream Processing of Change Data Events
23. #ChangeDataStreamingPatterns @gunnarmorling Apache Flink Stream Processing of Change Data Events
24. Outbox Pattern
25. #ChangeDataStreamingPatterns @gunnarmorling • Services need to update their database, • send messages to other services, • and that consistently! Challenge: Microservices Data Exchange
26. #ChangeDataStreamingPatterns @gunnarmorling “Dual writes” are prone to inconsistencies! Outbox Pattern
27. #ChangeDataStreamingPatterns @gunnarmorling Outbox Pattern
28. #ChangeDataStreamingPatterns @gunnarmorling Outbox Pattern
29. #ChangeDataStreamingPatterns @gunnarmorling Outbox Pattern
30. #ChangeDataStreamingPatterns @gunnarmorling Outbox Pattern
31. #ChangeDataStreamingPatterns @gunnarmorling Variation on Postgres pg_logical_emit_message() • Directly writing arbitrary messages to the WAL • No need for an outbox table
32. #ChangeDataStreamingPatterns @gunnarmorling Outbox Pattern Flink CDC Source
33. #ChangeDataStreamingPatterns @gunnarmorling Outbox Pattern Flink Pipeline
34. #ChangeDataStreamingPatterns @gunnarmorling Outbox Pattern Serializer
35. #ChangeDataStreamingPatterns @gunnarmorling Strangler Fig Pattern
36. #ChangeDataStreamingPatterns @gunnarmorling • Gradually evolve from old into new • Support temporary coexistence • Avoid big bang cut-over Challenge: Migrating Systems
37. #ChangeDataStreamingPatterns @gunnarmorling CDC-based Strangler Fig Pattern
38. #ChangeDataStreamingPatterns @gunnarmorling CDC-based Strangler Fig Pattern
39. #ChangeDataStreamingPatterns @gunnarmorling CDC-based Strangler Fig Pattern
40. #ChangeDataStreamingPatterns @gunnarmorling CDC-based Strangler Fig Pattern
41. #ChangeDataStreamingPatterns @gunnarmorling CDC-based Strangler Fig Pattern
42. #ChangeDataStreamingPatterns @gunnarmorling CDC-based Strangler Fig Pattern
43. #ChangeDataStreamingPatterns @gunnarmorling CDC-based Strangler Fig Pattern
44. #ChangeDataStreamingPatterns @gunnarmorling CDC-based Strangler Fig Pattern Demo Repo: https://bit.ly/ff21-sfp
45. #ChangeDataStreamingPatterns @gunnarmorling • Incremental migration → “baby steps” • Pause or stop migration without losing spent efforts • Migration steps ideally reversible Rationale: ⚠ minimize risk ⚠ Benefits
46. #ChangeDataStreamingPatterns @gunnarmorling CDC Pipeline Considerations • data model leaking from monolith ? • “1:1 replication” → building aggregates ?
47. #ChangeDataStreamingPatterns @gunnarmorling Enhanced CDC Processing Single Message Transforms
48. #ChangeDataStreamingPatterns @gunnarmorling Enhanced CDC Processing Custom Stream Processing with Flink
49. #ChangeDataStreamingPatterns @gunnarmorling Example: Join With Custom Aggregation
50. #ChangeDataStreamingPatterns @gunnarmorling Example: Join With Custom Aggregation Flink CDC Connector
51. #ChangeDataStreamingPatterns @gunnarmorling Example: Join With Custom Aggregation Flink SQL
52. Audit Logs
53. #ChangeDataStreamingPatterns @gunnarmorling Challenge: Capturing Intent pg_logical_emit_message() • Pure CDC events lack metadata like business user, device id, etc. • Solution: emit at TX begin, enrich events using Flink
54. #ChangeDataStreamingPatterns @gunnarmorling Capturing Intent Enriching Change Data Events with Metadata
55. #ChangeDataStreamingPatterns @gunnarmorling Capturing Intent Enriching Change Events Via Apache Flink
56. #ChangeDataStreamingPatterns @gunnarmorling Wrap-Up
57. #ChangeDataStreamingPatterns @gunnarmorling • The fresher data is, the more valuable it is • Change Data Capture and Debezium: Powerful tools for realtime change event feeds • Combining CDC with stream processing: Many more possibilities • Learn more: ◦ https://www.infoq.com/articles/wonders-of-postgres-logical-decoding-messages/ ◦ https://github.com/decodableco/examples/blob/main/postgres-logical-decoding/ Take Aways
58. #ChangeDataStreamingPatterns @gunnarmorling Q & A gunnar@decodable.co @gunnarmorling 📧 Thank You!
60. #ChangeDataStreamingPatterns @gunnarmorling Debezium Correlating Events From Same Transaction
61. #ChangeDataStreamingPatterns @gunnarmorling Debezium Correlating Events From Same Transaction https://www.slideshare.net/FlinkForward/squirreling-away-640-billion-how-stripe-leverages-flink-for-change-data-capture