Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Advanced Change Data Streaming Patterns in Distributed Systems @ Kafka Summit Europe 2021

Advanced Change Data Streaming Patterns in Distributed Systems @ Kafka Summit Europe 2021

Abstract:

Microservices are one of the big trends in software engineering of the last few years; organising business functionality in several self-contained, loosely coupled services helps teams to work efficiently, make the most suitable technical decisions, and react quickly to new business requirements.

In this session we'll discuss and showcase how open-source change data capture (CDC) with Debezium can help developers with typical challenges they often face when working on microservices. Come and join us to learn how to:

- Employ the outbox pattern for reliable, eventually consistent data exchange between microservices, without incurring unsafe dual writes or tight coupling
- Gradually extract microservices from existing monolithic applications, using CDC and the strangler fig pattern
- Coordinate long-running business transactions across multiple services using CDC-based saga orchestration, ensuring such activity gets consistently applied or aborted by all participating services.

Recording:
http://bit.ly/ks-eu-21 in on-demand library at event page (registration required)

Hans-Peter Grahsl

May 12, 2021
Tweet

More Decks by Hans-Peter Grahsl

Other Decks in Programming

Transcript

  1. Gunnar Morling
    Software Engineer, Red Hat
    @gunnarmorling
    Hans-Peter Grahsl
    Technical Trainer, Netconomy
    @hpgrahsl
    Kafka Summit Europe 2021
    Advanced Change Data Streaming Patterns
    in Distributed Systems

    View Slide

  2. #CDCPatterns @gunnarmorling @hpgrahsl
    Gunnar Morling
    ● Open source software engineer at Red Hat
    ○ Debezium
    ○ Quarkus
    ● Spec Lead for Bean Validation 2.0
    ● Java Champion
    ● @gunnarmorling

    View Slide

  3. #CDCPatterns @gunnarmorling @hpgrahsl
    Hans-Peter Grahsl
    ● Technical Trainer at NETCONOMY
    ● Independent Engineer & Consultant
    ● Confluent Community Catalyst
    ● MongoDB Champion
    ● @hpgrahsl

    View Slide

  4. Today’s Objective: Learn About...
    … implemented using Change Data Capture

    View Slide

  5. #CDCPatterns @gunnarmorling @hpgrahsl
    ● Taps into TX log to capture INSERT/UPDATE/DELETE events
    ● Propagated to consumers via Apache Kafka and Kafka Connect
    Debezium — Log-based Change Data Capture

    View Slide

  6. Outbox Pattern

    View Slide

  7. #CDCPatterns @gunnarmorling @hpgrahsl
    The Problem: Microservices Data Exchange
    ● Services need to update their database,
    ● send messages to other services,
    ● and that consistently!

    View Slide

  8. #CDCPatterns @gunnarmorling @hpgrahsl
    Outbox Pattern
    “Dual writes” are prone to inconsistencies!

    View Slide

  9. #CDCPatterns @gunnarmorling @hpgrahsl
    Outbox Pattern

    View Slide

  10. #CDCPatterns @gunnarmorling @hpgrahsl
    Outbox Pattern

    View Slide

  11. #CDCPatterns @gunnarmorling @hpgrahsl
    Outbox Pattern

    View Slide

  12. #CDCPatterns @gunnarmorling @hpgrahsl
    Outbox Pattern

    View Slide

  13. Strangler Fig Pattern

    View Slide

  14. #CDCPatterns @gunnarmorling @hpgrahsl
    The Problem: Migrating Systems
    ● Gradually evolve from old into new
    ● Support temporary coexistence
    ● Avoid big bang cut-over

    View Slide

  15. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern

    View Slide

  16. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern

    View Slide

  17. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern

    View Slide

  18. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern

    View Slide

  19. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern

    View Slide

  20. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern

    View Slide

  21. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern

    View Slide

  22. #CDCPatterns @gunnarmorling @hpgrahsl
    Benefits
    ● Incremental migration → “baby steps”
    ● Pause or stop migration
    without losing spent efforts
    ● Migration steps ideally reversible
    Rationale: ⚠ minimize risk ⚠

    View Slide

  23. #CDCPatterns @gunnarmorling @hpgrahsl
    CDC Close-Up

    View Slide

  24. #CDCPatterns @gunnarmorling @hpgrahsl
    Enhanced CDC Processing

    View Slide

  25. #CDCPatterns @gunnarmorling @hpgrahsl
    Enhanced CDC Processing

    View Slide

  26. #CDCPatterns @gunnarmorling @hpgrahsl
    Enhanced CDC Processing

    View Slide

  27. Saga Pattern

    View Slide

  28. #CDCPatterns @gunnarmorling @hpgrahsl
    The Problem: Long-running Business Transactions
    ● Multiple services need to act collaboratively
    to achieve a consistent outcome
    ● Without 2-phase commit protocols
    ● Ensure correctness in case of failures

    View Slide

  29. #CDCPatterns @gunnarmorling @hpgrahsl
    Saga Pattern

    View Slide

  30. #CDCPatterns @gunnarmorling @hpgrahsl
    Saga Pattern

    View Slide

  31. #CDCPatterns @gunnarmorling @hpgrahsl
    Saga Pattern

    View Slide

  32. Demo

    View Slide

  33. Wrap-Up

    View Slide

  34. #CDCPatterns @gunnarmorling @hpgrahsl
    ● CDC: a powerful tool in the box for
    event-driven architectures
    ● Debezium: open-source CDC for
    a variety of databases
    ● Call to Action:
    Would you like built-in Saga support?
    Takeaways

    View Slide

  35. #CDCPatterns @gunnarmorling @hpgrahsl
    ● Outbox implementation
    https://debezium.io/blog/2019/02/19/reliable-microservices-data
    -exchange-with-the-outbox-pattern/
    ● Strangler fig pattern
    https://martinfowler.com/bliki/StranglerFigApplication.html
    ● Saga implementation
    https://www.infoq.com/articles/saga-orchestration-outbox/
    Resources

    View Slide

  36. #CDCPatterns @gunnarmorling @hpgrahsl
    Q & A
    [email protected]
    @gunnarmorling
    📧 [email protected]
    @hpgrahsl
    📧

    View Slide

  37. Unsplash https://unsplash.com/license
    © Shiro hatori https://unsplash.com/photos/WR-ifjFy4CI
    © Pablo García Saldaña https://unsplash.com/photos/lPQIndZz8Mo
    © David Clode https://unsplash.com/photos/T49WTav4LgU
    © Aaron Burden https://unsplash.com/photos/GFpxQ2ZyNc0
    © Nathan Dumlao https://unsplash.com/photos/wQDysNUCKfw
    © mari lezhava https://unsplash.com/photos/q65bNe9fW-w
    © Michał Parzuchowski https://unsplash.com/photos/Bt0PM7cNJFQ
    © Charles Forerunner https://unsplash.com/photos/3fPXt37X6UQ
    Flickr
    Attribution 2.0 Generic https://creativecommons.org/licenses/by/2.0/
    © Thomas Kamann https://flic.kr/p/coa2c
    CC0 1.0 Universal Public Domain Dedication https://creativecommons.org/publicdomain/zero/1.0/
    © Wall Boat https://flic.kr/p/Y6zkmX
    Attribution-ShareAlike 2.0 Generic https://creativecommons.org/licenses/by-sa/2.0/
    © Maria Eklind https://flic.kr/p/251HmVw
    Image Credits
    In Order of Appearance

    View Slide