Advanced Change Data Streaming Patterns in Distributed Systems @ Kafka Summit Europe 2021

Abstract:

Microservices are one of the big trends in software engineering of the last few years; organising business functionality in several self-contained, loosely coupled services helps teams to work efficiently, make the most suitable technical decisions, and react quickly to new business requirements.

In this session we'll discuss and showcase how open-source change data capture (CDC) with Debezium can help developers with typical challenges they often face when working on microservices. Come and join us to learn how to:

- Employ the outbox pattern for reliable, eventually consistent data exchange between microservices, without incurring unsafe dual writes or tight coupling
- Gradually extract microservices from existing monolithic applications, using CDC and the strangler fig pattern
- Coordinate long-running business transactions across multiple services using CDC-based saga orchestration, ensuring such activity gets consistently applied or aborted by all participating services.

Recording:
http://bit.ly/ks-eu-21 in on-demand library at event page (registration required)

Transcript

1. Gunnar Morling
   Software Engineer, Red Hat
   @gunnarmorling
   Hans-Peter Grahsl
   Technical Trainer, Netconomy
   @hpgrahsl
   Kafka Summit Europe 2021
   Advanced Change Data Streaming Patterns
   in Distributed Systems
   

   View Slide

2. #CDCPatterns @gunnarmorling @hpgrahsl
   Gunnar Morling
   ● Open source software engineer at Red Hat
   ○ Debezium
   ○ Quarkus
   ● Spec Lead for Bean Validation 2.0
   ● Java Champion
   ● @gunnarmorling
   

   View Slide

3. #CDCPatterns @gunnarmorling @hpgrahsl
   Hans-Peter Grahsl
   ● Technical Trainer at NETCONOMY
   ● Independent Engineer & Consultant
   ● Confluent Community Catalyst
   ● MongoDB Champion
   ● @hpgrahsl
   

   View Slide

4. Today’s Objective: Learn About...
   … implemented using Change Data Capture
   

   View Slide

5. #CDCPatterns @gunnarmorling @hpgrahsl
   ● Taps into TX log to capture INSERT/UPDATE/DELETE events
   ● Propagated to consumers via Apache Kafka and Kafka Connect
   Debezium — Log-based Change Data Capture
   

   View Slide

6. Outbox Pattern
   

   View Slide

7. #CDCPatterns @gunnarmorling @hpgrahsl
   The Problem: Microservices Data Exchange
   ● Services need to update their database,
   ● send messages to other services,
   ● and that consistently!
   

   View Slide

8. #CDCPatterns @gunnarmorling @hpgrahsl
   Outbox Pattern
   “Dual writes” are prone to inconsistencies!
   

   View Slide

9. #CDCPatterns @gunnarmorling @hpgrahsl
   Outbox Pattern
   

   View Slide

10. #CDCPatterns @gunnarmorling @hpgrahsl
    Outbox Pattern
    

    View Slide

11. #CDCPatterns @gunnarmorling @hpgrahsl
    Outbox Pattern
    

    View Slide

12. #CDCPatterns @gunnarmorling @hpgrahsl
    Outbox Pattern
    

    View Slide

13. Strangler Fig Pattern
    

    View Slide

14. #CDCPatterns @gunnarmorling @hpgrahsl
    The Problem: Migrating Systems
    ● Gradually evolve from old into new
    ● Support temporary coexistence
    ● Avoid big bang cut-over
    

    View Slide

15. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern
    

    View Slide

16. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern
    

    View Slide

17. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern
    

    View Slide

18. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern
    

    View Slide

19. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern
    

    View Slide

20. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern
    

    View Slide

21. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern
    

    View Slide

22. #CDCPatterns @gunnarmorling @hpgrahsl
    Benefits
    ● Incremental migration → “baby steps”
    ● Pause or stop migration
    without losing spent efforts
    ● Migration steps ideally reversible
    Rationale: ⚠ minimize risk ⚠
    

    View Slide

23. #CDCPatterns @gunnarmorling @hpgrahsl
    CDC Close-Up
    

    View Slide

24. #CDCPatterns @gunnarmorling @hpgrahsl
    Enhanced CDC Processing
    

    View Slide

25. #CDCPatterns @gunnarmorling @hpgrahsl
    Enhanced CDC Processing
    

    View Slide

26. #CDCPatterns @gunnarmorling @hpgrahsl
    Enhanced CDC Processing
    

    View Slide

27. Saga Pattern
    

    View Slide

28. #CDCPatterns @gunnarmorling @hpgrahsl
    The Problem: Long-running Business Transactions
    ● Multiple services need to act collaboratively
    to achieve a consistent outcome
    ● Without 2-phase commit protocols
    ● Ensure correctness in case of failures
    

    View Slide

29. #CDCPatterns @gunnarmorling @hpgrahsl
    Saga Pattern
    

    View Slide

30. #CDCPatterns @gunnarmorling @hpgrahsl
    Saga Pattern
    

    View Slide

31. #CDCPatterns @gunnarmorling @hpgrahsl
    Saga Pattern
    

    View Slide

32. Demo
    

    View Slide

33. Wrap-Up
    

    View Slide

34. #CDCPatterns @gunnarmorling @hpgrahsl
    ● CDC: a powerful tool in the box for
    event-driven architectures
    ● Debezium: open-source CDC for
    a variety of databases
    ● Call to Action:
    Would you like built-in Saga support?
    Takeaways
    

    View Slide

35. #CDCPatterns @gunnarmorling @hpgrahsl
    ● Outbox implementation
    https://debezium.io/blog/2019/02/19/reliable-microservices-data
    -exchange-with-the-outbox-pattern/
    ● Strangler fig pattern
    https://martinfowler.com/bliki/StranglerFigApplication.html
    ● Saga implementation
    https://www.infoq.com/articles/saga-orchestration-outbox/
    Resources
    

    View Slide

36. #CDCPatterns @gunnarmorling @hpgrahsl
    Q & A
    gunnar@hibernate.org
    @gunnarmorling
    📧 grahslhp@gmail.com
    @hpgrahsl
    📧
    

    View Slide

37. Unsplash https://unsplash.com/license
    © Shiro hatori https://unsplash.com/photos/WR-ifjFy4CI
    © Pablo García Saldaña https://unsplash.com/photos/lPQIndZz8Mo
    © David Clode https://unsplash.com/photos/T49WTav4LgU
    © Aaron Burden https://unsplash.com/photos/GFpxQ2ZyNc0
    © Nathan Dumlao https://unsplash.com/photos/wQDysNUCKfw
    © mari lezhava https://unsplash.com/photos/q65bNe9fW-w
    © Michał Parzuchowski https://unsplash.com/photos/Bt0PM7cNJFQ
    © Charles Forerunner https://unsplash.com/photos/3fPXt37X6UQ
    Flickr
    Attribution 2.0 Generic https://creativecommons.org/licenses/by/2.0/
    © Thomas Kamann https://flic.kr/p/coa2c
    CC0 1.0 Universal Public Domain Dedication https://creativecommons.org/publicdomain/zero/1.0/
    © Wall Boat https://flic.kr/p/Y6zkmX
    Attribution-ShareAlike 2.0 Generic https://creativecommons.org/licenses/by-sa/2.0/
    © Maria Eklind https://flic.kr/p/251HmVw
    Image Credits
    In Order of Appearance
    

    View Slide