Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Change Data Streaming Patterns With Debezium & Apache Flink

Change Data Streaming Patterns With Debezium & Apache Flink

Microservices are one of the big trends in software engineering of the last few years: organizing business functionality in several self-contained, loosely coupled services helps engineering teams to work efficiently, make the most suitable technical decisions, and react quickly to new business requirements.

In this session we'll discuss and showcase how open-source change data capture (CDC) with Debezium can help developers with typical challenges they often face when working on microservices. Come and join us to learn how to:

* Employ the outbox pattern for reliable, eventually consistent data exchange between microservices, without incurring unsafe dual writes or tight coupling
* Gradually extract microservices from existing monolithic applications, using CDC, the strangler fig pattern and Apache Flink
* Building auditing logs, containing not only the changed data itself, but also additional metadata like business user, client configuration, or use case identifier

Gunnar Morling

March 30, 2023
Tweet

More Decks by Gunnar Morling

Other Decks in Programming

Transcript

  1. Image © pixelchecker https://flic.kr/p/ah5sPr (CC BY 2.0)
    Change Data Streaming Patterns
    With Debezium & Apache Flink
    Gunnar Morling
    Senior Staff Software Engineer, Decodable
    @gunnarmorling

    View full-size slide

  2. The world is
    real-time.
    So should be
    your data.

    View full-size slide

  3. #ChangeDataStreamingPatterns @gunnarmorling
    Today’s Mission
    Learn About…

    View full-size slide

  4. #ChangeDataStreamingPatterns @gunnarmorling
    ● Software engineer at Decodable
    ● Former project lead of Debezium
    ● kcctl 🧸, JfrUnit, ModiTect,
    MapStruct
    ● Spec Lead for Bean Validation 2.0
    ● Java Champion
    Gunnar Morling

    View full-size slide

  5. The Tools
    https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot

    View full-size slide

  6. #ChangeDataStreamingPatterns @gunnarmorling
    Debezium
    Log-Based Change Data Capture
    ● Taps into TX log to capture INSERT/UPDATE/DELETE events
    ● Typically propagated to consumers via Apache Kafka

    View full-size slide

  7. #ChangeDataStreamingPatterns @gunnarmorling
    Debezium
    Log-Based Change Data Capture
    ● Taps into TX log to capture INSERT/UPDATE/DELETE events
    ● Typically propagated to consumers via Apache Kafka

    View full-size slide

  8. #ChangeDataStreamingPatterns @gunnarmorling
    Debezium
    Open-Source Change Data Capture
    ● A CDC Platform
    ○ Based on transaction logs
    ○ Snapshotting, filtering, etc.
    ○ Outbox support
    ○ Web-based UI
    ● Fully open-source, very active
    community
    ● Large production deployments

    View full-size slide

  9. #ChangeDataStreamingPatterns @gunnarmorling
    Debezium: Data Change Events
    ● Old and new row state
    ● Metadata on table, TX id, etc.
    ● Operation type, timestamp

    View full-size slide

  10. #ChangeDataStreamingPatterns @gunnarmorling
    Debezium: Data Change Events
    ● Old and new row state
    ● Metadata on table, TX id, etc.
    ● Operation type, timestamp

    View full-size slide

  11. #ChangeDataStreamingPatterns @gunnarmorling
    Debezium: Data Change Events
    ● Old and new row state
    ● Metadata on table, TX id, etc.
    ● Operation type, timestamp

    View full-size slide

  12. #ChangeDataStreamingPatterns @gunnarmorling
    Debezium
    Deployment Options

    View full-size slide

  13. #ChangeDataStreamingPatterns @gunnarmorling
    Becoming the De-Facto CDC Standard
    Debezium
    Google Cloud Spanner ScyllaDB

    View full-size slide

  14. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot
    Apache Flink

    View full-size slide

  15. #ChangeDataStreamingPatterns @gunnarmorling
    Apache Flink
    Stateful Computations over Data Streams
    https://flink.apache.org/

    View full-size slide

  16. #ChangeDataStreamingPatterns @gunnarmorling
    Apache Flink
    APIs for Application Development
    Image source: “Change Data Capture with Flink SQL and Debezium, by Marta Paes at DataEngBytes
    (https://noti.st/morsapaes/liQzgs/change-data-capture-with-flink-sql-and-debezium)

    View full-size slide

  17. #ChangeDataStreamingPatterns @gunnarmorling
    Apache Flink
    Stream Processing of Change Data Events

    View full-size slide

  18. #ChangeDataStreamingPatterns @gunnarmorling
    Apache Flink
    Stream Processing of Change Data Events

    View full-size slide

  19. #ChangeDataStreamingPatterns @gunnarmorling
    Apache Flink
    Stream Processing of Change Data Events

    View full-size slide

  20. #ChangeDataStreamingPatterns @gunnarmorling
    Apache Flink
    Stream Processing of Change Data Events

    View full-size slide

  21. #ChangeDataStreamingPatterns @gunnarmorling
    Apache Flink
    Stream Processing of Change Data Events

    View full-size slide

  22. #ChangeDataStreamingPatterns @gunnarmorling
    Apache Flink
    Stream Processing of Change Data Events

    View full-size slide

  23. Outbox Pattern

    View full-size slide

  24. #ChangeDataStreamingPatterns @gunnarmorling
    ● Services need to update their database,
    ● send messages to other services,
    ● and that consistently!
    Challenge: Microservices Data Exchange

    View full-size slide

  25. #ChangeDataStreamingPatterns @gunnarmorling
    “Dual writes” are prone to inconsistencies!
    Outbox Pattern

    View full-size slide

  26. #ChangeDataStreamingPatterns @gunnarmorling
    Outbox Pattern

    View full-size slide

  27. #ChangeDataStreamingPatterns @gunnarmorling
    Outbox Pattern

    View full-size slide

  28. #ChangeDataStreamingPatterns @gunnarmorling
    Outbox Pattern

    View full-size slide

  29. #ChangeDataStreamingPatterns @gunnarmorling
    Outbox Pattern

    View full-size slide

  30. #ChangeDataStreamingPatterns @gunnarmorling
    Variation on Postgres
    pg_logical_emit_message()
    ● Directly writing arbitrary messages to the WAL
    ● No need for an outbox table

    View full-size slide

  31. #ChangeDataStreamingPatterns @gunnarmorling
    Outbox Pattern
    Flink CDC Source

    View full-size slide

  32. #ChangeDataStreamingPatterns @gunnarmorling
    Outbox Pattern
    Flink Pipeline

    View full-size slide

  33. #ChangeDataStreamingPatterns @gunnarmorling
    Outbox Pattern
    Serializer

    View full-size slide

  34. #ChangeDataStreamingPatterns @gunnarmorling
    Strangler Fig Pattern

    View full-size slide

  35. #ChangeDataStreamingPatterns @gunnarmorling
    ● Gradually evolve from old into new
    ● Support temporary coexistence
    ● Avoid big bang cut-over
    Challenge: Migrating Systems

    View full-size slide

  36. #ChangeDataStreamingPatterns @gunnarmorling
    CDC-based Strangler Fig Pattern

    View full-size slide

  37. #ChangeDataStreamingPatterns @gunnarmorling
    CDC-based Strangler Fig Pattern

    View full-size slide

  38. #ChangeDataStreamingPatterns @gunnarmorling
    CDC-based Strangler Fig Pattern

    View full-size slide

  39. #ChangeDataStreamingPatterns @gunnarmorling
    CDC-based Strangler Fig Pattern

    View full-size slide

  40. #ChangeDataStreamingPatterns @gunnarmorling
    CDC-based Strangler Fig Pattern

    View full-size slide

  41. #ChangeDataStreamingPatterns @gunnarmorling
    CDC-based Strangler Fig Pattern

    View full-size slide

  42. #ChangeDataStreamingPatterns @gunnarmorling
    CDC-based Strangler Fig Pattern

    View full-size slide

  43. #ChangeDataStreamingPatterns @gunnarmorling
    CDC-based Strangler Fig Pattern
    Demo Repo: https://bit.ly/ff21-sfp

    View full-size slide

  44. #ChangeDataStreamingPatterns @gunnarmorling
    ● Incremental migration → “baby steps”
    ● Pause or stop migration
    without losing spent efforts
    ● Migration steps ideally reversible
    Rationale: ⚠ minimize risk ⚠
    Benefits

    View full-size slide

  45. #ChangeDataStreamingPatterns @gunnarmorling
    CDC Pipeline Considerations
    ● data model leaking from monolith ?
    ● “1:1 replication” → building aggregates ?

    View full-size slide

  46. #ChangeDataStreamingPatterns @gunnarmorling
    Enhanced CDC Processing
    Single Message Transforms

    View full-size slide

  47. #ChangeDataStreamingPatterns @gunnarmorling
    Enhanced CDC Processing
    Custom Stream Processing with Flink

    View full-size slide

  48. #ChangeDataStreamingPatterns @gunnarmorling
    Example: Join With Custom Aggregation

    View full-size slide

  49. #ChangeDataStreamingPatterns @gunnarmorling
    Example: Join With Custom Aggregation
    Flink CDC Connector

    View full-size slide

  50. #ChangeDataStreamingPatterns @gunnarmorling
    Example: Join With Custom Aggregation
    Flink SQL

    View full-size slide

  51. #ChangeDataStreamingPatterns @gunnarmorling
    Challenge: Capturing Intent
    pg_logical_emit_message()
    ● Pure CDC events lack metadata like business user, device id, etc.
    ● Solution: emit at TX begin, enrich events using Flink

    View full-size slide

  52. #ChangeDataStreamingPatterns @gunnarmorling
    Capturing Intent
    Enriching Change Data Events with Metadata

    View full-size slide

  53. #ChangeDataStreamingPatterns @gunnarmorling
    Capturing Intent
    Enriching Change Events Via Apache Flink

    View full-size slide

  54. #ChangeDataStreamingPatterns @gunnarmorling
    Wrap-Up

    View full-size slide

  55. #ChangeDataStreamingPatterns @gunnarmorling
    ● The fresher data is, the more valuable it is
    ● Change Data Capture and Debezium: Powerful tools for realtime
    change event feeds
    ● Combining CDC with stream processing: Many more possibilities
    ● Learn more:
    ○ https://www.infoq.com/articles/wonders-of-postgres-logical-decoding-messages/
    ○ https://github.com/decodableco/examples/blob/main/postgres-logical-decoding/
    Take Aways

    View full-size slide

  56. #ChangeDataStreamingPatterns @gunnarmorling
    Q & A
    [email protected]
    @gunnarmorling
    📧
    Thank You!

    View full-size slide

  57. #ChangeDataStreamingPatterns @gunnarmorling
    Debezium
    Correlating Events From Same Transaction

    View full-size slide

  58. #ChangeDataStreamingPatterns @gunnarmorling
    Debezium
    Correlating Events From Same Transaction
    https://www.slideshare.net/FlinkForward/squirreling-away-640-billion-how-stripe-leverages-flink-for-change-data-capture

    View full-size slide