Change Data Streaming Patterns in Distributed Systems @ BerlinBuzzwords 2021

Abstract:

Microservices are one of the big trends in software engineering of the last few years; organising business functionality in several self-contained, loosely coupled services helps teams to work efficiently, make the most suitable technical decisions, and react quickly to new business requirements.

In this session we'll discuss and showcase how open-source change data capture (CDC) with Debezium can help developers with typical challenges they often face when working on microservices. Come and join us to learn how to:

* Employ the outbox pattern for reliable, eventually consistent data exchange between microservices, without incurring unsafe dual writes or tight coupling

* Gradually extract microservices from existing monolithic applications, using CDC and the strangler fig pattern

* Coordinate long-running business transactions across multiple services using CDC-based saga orchestration, ensuring such activity gets consistently applied or aborted by all participating services

Event Link: https://2021.berlinbuzzwords.de/session/change-data-streaming-patterns-distributed-systems

Recording Link:
https://www.youtube.com/watch?v=CLv2EcYnr2g

Transcript

1. Change Data Streaming Patterns in
   Distributed Systems
   Gunnar Morling
   Software Engineer, Red Hat
   @gunnarmorling
   Hans-Peter Grahsl
   Technical Trainer, Netconomy
   @hpgrahsl
   

   View Slide

2. #CDCPatterns @gunnarmorling @hpgrahsl
   … implemented using Change Data Capture
   Today’s Objectives
   

   View Slide

3. #CDCPatterns @gunnarmorling @hpgrahsl
   ● Open source software engineer at Red Hat
   ○ Debezium
   ○ Quarkus
   ● Spec Lead for Bean Validation 2.0
   ● Java Champion
   ● @gunnarmorling
   Gunnar Morling
   

   View Slide

4. #CDCPatterns @gunnarmorling @hpgrahsl
   ● Technical Trainer at NETCONOMY
   ● Independent Engineer & Consultant
   ● Confluent Community Catalyst
   ● MongoDB Champion
   ● @hpgrahsl
   Hans-Peter Grahsl
   

   View Slide

5. #CDCPatterns @gunnarmorling @hpgrahsl
   ● Taps into TX log to capture INSERT/UPDATE/DELETE events
   ● Propagated to consumers via Apache Kafka and Kafka Connect
   Debezium — Log-based Change Data Capture
   

   View Slide

6. #CDCPatterns @gunnarmorling @hpgrahsl
   Detour: Data Change Events
   ● Old and new row state
   ● Metadata on table, TX id, etc.
   ● Operation type, timestamp
   

   View Slide

7. #CDCPatterns @gunnarmorling @hpgrahsl
   ● Old and new row state
   ● Metadata on table, TX id, etc.
   ● Operation type, timestamp
   Detour: Data Change Events
   

   View Slide

8. #CDCPatterns @gunnarmorling @hpgrahsl
   ● Old and new row state
   ● Metadata on table, TX id, etc.
   ● Operation type, timestamp
   Detour: Data Change Events
   

   View Slide

9. Outbox Pattern
   

   View Slide

10. #CDCPatterns @gunnarmorling @hpgrahsl
    ● Services need to update their database,
    ● send messages to other services,
    ● and that consistently!
    The Problem: Microservices Data Exchange
    

    View Slide

11. #CDCPatterns @gunnarmorling @hpgrahsl
    “Dual writes” are prone to inconsistencies!
    Outbox Pattern
    

    View Slide

12. #CDCPatterns @gunnarmorling @hpgrahsl
    Outbox Pattern
    

    View Slide

13. #CDCPatterns @gunnarmorling @hpgrahsl
    Outbox Pattern
    

    View Slide

14. #CDCPatterns @gunnarmorling @hpgrahsl
    Outbox Pattern
    

    View Slide

15. #CDCPatterns @gunnarmorling @hpgrahsl
    Outbox Pattern
    

    View Slide

16. Strangler Fig Pattern
    

    View Slide

17. #CDCPatterns @gunnarmorling @hpgrahsl
    ● Gradually evolve from old into new
    ● Support temporary coexistence
    ● Avoid big bang cut-over
    The Problem: Migrating Systems
    

    View Slide

18. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern
    

    View Slide

19. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern
    

    View Slide

20. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern
    

    View Slide

21. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern
    

    View Slide

22. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern
    

    View Slide

23. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern
    

    View Slide

24. #CDCPatterns @gunnarmorling @hpgrahsl
    Strangler Fig Pattern
    

    View Slide

25. #CDCPatterns @gunnarmorling @hpgrahsl
    ● Incremental migration → “baby steps”
    ● Pause or stop migration
    without losing spent efforts
    ● Migration steps ideally reversible
    Rationale: ⚠ minimize risk ⚠
    Benefits
    

    View Slide

26. #CDCPatterns @gunnarmorling @hpgrahsl
    CDC Close-Up
    

    View Slide

27. #CDCPatterns @gunnarmorling @hpgrahsl
    Enhanced CDC Processing
    

    View Slide

28. #CDCPatterns @gunnarmorling @hpgrahsl
    Enhanced CDC Processing
    

    View Slide

29. #CDCPatterns @gunnarmorling @hpgrahsl
    Enhanced CDC Processing
    

    View Slide

30. Saga Pattern
    

    View Slide

31. #CDCPatterns @gunnarmorling @hpgrahsl
    ● Multiple services need to act collaboratively
    to achieve a consistent outcome
    ● Without 2-phase commit protocols
    ● Ensure correctness in case of failures
    The Problem: Long-running Business Transactions
    

    View Slide

32. #CDCPatterns @gunnarmorling @hpgrahsl
    Saga Pattern
    

    View Slide

33. #CDCPatterns @gunnarmorling @hpgrahsl
    Saga Pattern
    

    View Slide

34. #CDCPatterns @gunnarmorling @hpgrahsl
    Saga Pattern
    

    View Slide

35. Demo
    

    View Slide

36. Wrap-Up
    

    View Slide

37. #CDCPatterns @gunnarmorling @hpgrahsl
    ● CDC: a powerful tool in the box for
    event-driven architectures
    ● Debezium: open-source CDC for
    a variety of databases
    ● Call to Action:
    Would you like built-in Saga support?
    Takeaways
    

    View Slide

38. #CDCPatterns @gunnarmorling @hpgrahsl
    ● Outbox implementation
    https://debezium.io/blog/2019/02/19/reliable-microservices-data
    -exchange-with-the-outbox-pattern/
    ● Strangler fig pattern
    https://martinfowler.com/bliki/StranglerFigApplication.html
    ● Saga implementation
    https://www.infoq.com/articles/saga-orchestration-outbox/
    ● Demo repo
    https://github.com/debezium/debezium-examples
    Resources
    

    View Slide

39. #CDCPatterns @gunnarmorling @hpgrahsl
    Q & A
    gunnar@hibernate.org
    @gunnarmorling
    📧 grahslhp@gmail.com
    @hpgrahsl
    📧
    Thank You!
    

    View Slide

40. #CDCPatterns @gunnarmorling @hpgrahsl
    Unsplash https://unsplash.com/license
    © Pablo García Saldaña https://unsplash.com/photos/lPQIndZz8Mo
    © David Clode https://unsplash.com/photos/T49WTav4LgU
    © Aaron Burden https://unsplash.com/photos/GFpxQ2ZyNc0
    © Nathan Dumlao https://unsplash.com/photos/wQDysNUCKfw
    © mari lezhava https://unsplash.com/photos/q65bNe9fW-w
    © Michał Parzuchowski https://unsplash.com/photos/Bt0PM7cNJFQ
    © Charles Forerunner https://unsplash.com/photos/3fPXt37X6UQ
    Flickr
    Attribution 2.0 Generic https://creativecommons.org/licenses/by/2.0/
    © Thomas Kamann https://flic.kr/p/coa2c
    CC0 1.0 Universal Public Domain Dedication https://creativecommons.org/publicdomain/zero/1.0/
    © Wall Boat https://flic.kr/p/Y6zkmX
    Attribution-ShareAlike 2.0 Generic https://creativecommons.org/licenses/by-sa/2.0/
    © Andrew Hart https://flic.kr/p/dmjkSk
    Image Credits
    In Order of Appearance
    

    View Slide