Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Change Data Streaming Patterns in Distributed Systems @ TheDevConf (TDC) Connections 2021

Change Data Streaming Patterns in Distributed Systems @ TheDevConf (TDC) Connections 2021

Abstract

Microservices are one of the biggest trends in software engineering: organising functionality in self-contained, loosely coupled services lets teams make the most suitable technical decisions and react quickly to new requirements.

In this session we'll discuss how change data capture (CDC) with Debezium can help developers with challenges they face when working on microservices. Join us to learn how to:

* Employ the outbox pattern for reliable data exchange between microservices
* Gradually extract microservices from existing monolithic applications with the strangler fig pattern
* Coordinate long-running business transactions across multiple services using CDC-based saga orchestration

NOTE: There was no recording of this session AFAIK

744f1c2c6cbea2ff5104b0ac512936bd?s=128

Hans-Peter Grahsl

June 10, 2021
Tweet

Transcript

  1. Gunnar Morling Software Engineer, Red Hat @gunnarmorling Hans-Peter Grahsl Technical

    Trainer, Netconomy @hpgrahsl Change Data Streaming Patterns in Distributed Systems
  2. #CDCPatterns @gunnarmorling @hpgrahsl … implemented using Change Data Capture Today’s

    Objectives
  3. #CDCPatterns @gunnarmorling @hpgrahsl • Open source software engineer at Red

    Hat ◦ Debezium ◦ Quarkus • Spec Lead for Bean Validation 2.0 • Java Champion • @gunnarmorling Gunnar Morling
  4. #CDCPatterns @gunnarmorling @hpgrahsl • Technical Trainer at NETCONOMY • Independent

    Engineer & Consultant • Confluent Community Catalyst • MongoDB Champion • @hpgrahsl Hans-Peter Grahsl
  5. #CDCPatterns @gunnarmorling @hpgrahsl • Taps into TX log to capture

    INSERT/UPDATE/DELETE events • Propagated to consumers via Apache Kafka and Kafka Connect Debezium — Log-based Change Data Capture
  6. #CDCPatterns @gunnarmorling @hpgrahsl • Message Key: table PK • Message

    Value: ▪ Old and new row state ▪ Metadata on table, TX id, etc. ▪ Operation type, timestamp • Serialized as JSON, Avro, etc. Detour: Data Change Events
  7. #CDCPatterns @gunnarmorling @hpgrahsl • Message Key: table PK • Message

    Value: ▪ Old and new row state ▪ Metadata on table, TX id, etc. ▪ Operation type, timestamp • Serialized as JSON, Avro, etc. Detour: Data Change Events
  8. #CDCPatterns @gunnarmorling @hpgrahsl • Message Key: table PK • Message

    Value: ▪ Old and new row state ▪ Metadata on table, TX id, etc. ▪ Operation type, timestamp • Serialized as JSON, Avro, etc. Detour: Data Change Events
  9. Outbox Pattern

  10. #CDCPatterns @gunnarmorling @hpgrahsl • Services need to update their database,

    • send messages to other services, • and that consistently! The Problem: Microservices Data Exchange
  11. #CDCPatterns @gunnarmorling @hpgrahsl “Dual writes” are prone to inconsistencies! Outbox

    Pattern
  12. #CDCPatterns @gunnarmorling @hpgrahsl Outbox Pattern

  13. #CDCPatterns @gunnarmorling @hpgrahsl Outbox Pattern

  14. #CDCPatterns @gunnarmorling @hpgrahsl Outbox Pattern

  15. #CDCPatterns @gunnarmorling @hpgrahsl Outbox Pattern

  16. Strangler Fig Pattern

  17. #CDCPatterns @gunnarmorling @hpgrahsl • Gradually evolve from old into new

    • Support temporary coexistence • Avoid big bang cut-over The Problem: Migrating Systems
  18. #CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

  19. #CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

  20. #CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

  21. #CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

  22. #CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

  23. #CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

  24. #CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

  25. #CDCPatterns @gunnarmorling @hpgrahsl • Incremental migration → “baby steps” •

    Pause or stop migration without losing spent efforts • Migration steps ideally reversible Rationale: ⚠ minimize risk ⚠ Benefits
  26. #CDCPatterns @gunnarmorling @hpgrahsl CDC Close-Up

  27. #CDCPatterns @gunnarmorling @hpgrahsl Enhanced CDC Processing

  28. #CDCPatterns @gunnarmorling @hpgrahsl Enhanced CDC Processing

  29. #CDCPatterns @gunnarmorling @hpgrahsl Enhanced CDC Processing

  30. Saga Pattern

  31. #CDCPatterns @gunnarmorling @hpgrahsl • Multiple services need to act collaboratively

    to achieve a consistent outcome • Without 2-phase commit protocols • Ensure correctness in case of failures The Problem: Long-running Business Transactions
  32. #CDCPatterns @gunnarmorling @hpgrahsl Saga Pattern

  33. #CDCPatterns @gunnarmorling @hpgrahsl Saga Pattern

  34. #CDCPatterns @gunnarmorling @hpgrahsl Saga Pattern

  35. Demo

  36. Wrap-Up

  37. #CDCPatterns @gunnarmorling @hpgrahsl • CDC: a powerful tool in the

    box for event-driven architectures • Debezium: open-source CDC for a variety of databases • Call to Action: Would you like built-in Saga support? Takeaways
  38. #CDCPatterns @gunnarmorling @hpgrahsl • Outbox implementation https://debezium.io/blog/2019/02/19/reliable-microservices-data -exchange-with-the-outbox-pattern/ • Strangler

    fig pattern https://martinfowler.com/bliki/StranglerFigApplication.html • Saga implementation https://www.infoq.com/articles/saga-orchestration-outbox/ • Demo repo https://github.com/debezium/debezium-examples Resources
  39. #CDCPatterns @gunnarmorling @hpgrahsl Q & A gunnar@hibernate.org @gunnarmorling 📧 grahslhp@gmail.com

    @hpgrahsl 📧 Thank You!
  40. #CDCPatterns @gunnarmorling @hpgrahsl Unsplash https://unsplash.com/license © Jérôme Prax https://unsplash.com/photos/U_m-mPOZzMI ©

    Pablo García Saldaña https://unsplash.com/photos/lPQIndZz8Mo © David Clode https://unsplash.com/photos/T49WTav4LgU © Aaron Burden https://unsplash.com/photos/GFpxQ2ZyNc0 © Nathan Dumlao https://unsplash.com/photos/wQDysNUCKfw © mari lezhava https://unsplash.com/photos/q65bNe9fW-w © Michał Parzuchowski https://unsplash.com/photos/Bt0PM7cNJFQ © Charles Forerunner https://unsplash.com/photos/3fPXt37X6UQ Flickr Attribution 2.0 Generic https://creativecommons.org/licenses/by/2.0/ © Thomas Kamann https://flic.kr/p/coa2c CC0 1.0 Universal Public Domain Dedication https://creativecommons.org/publicdomain/zero/1.0/ © Wall Boat https://flic.kr/p/Y6zkmX Attribution-ShareAlike 2.0 Generic https://creativecommons.org/licenses/by-sa/2.0/ © Maria Eklind https://flic.kr/p/251HmVw Image Credits In Order of Appearance