Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Advanced Change Data Streaming Patterns in Distributed Systems @ Kafka Summit Europe 2021

Advanced Change Data Streaming Patterns in Distributed Systems @ Kafka Summit Europe 2021

Abstract:

Microservices are one of the big trends in software engineering of the last few years; organising business functionality in several self-contained, loosely coupled services helps teams to work efficiently, make the most suitable technical decisions, and react quickly to new business requirements.

In this session we'll discuss and showcase how open-source change data capture (CDC) with Debezium can help developers with typical challenges they often face when working on microservices. Come and join us to learn how to:

- Employ the outbox pattern for reliable, eventually consistent data exchange between microservices, without incurring unsafe dual writes or tight coupling
- Gradually extract microservices from existing monolithic applications, using CDC and the strangler fig pattern
- Coordinate long-running business transactions across multiple services using CDC-based saga orchestration, ensuring such activity gets consistently applied or aborted by all participating services.

Recording:
http://bit.ly/ks-eu-21 in on-demand library at event page (registration required)

744f1c2c6cbea2ff5104b0ac512936bd?s=128

Hans-Peter Grahsl

May 12, 2021
Tweet

Transcript

  1. Gunnar Morling Software Engineer, Red Hat @gunnarmorling Hans-Peter Grahsl Technical

    Trainer, Netconomy @hpgrahsl Kafka Summit Europe 2021 Advanced Change Data Streaming Patterns in Distributed Systems
  2. #CDCPatterns @gunnarmorling @hpgrahsl Gunnar Morling • Open source software engineer

    at Red Hat ◦ Debezium ◦ Quarkus • Spec Lead for Bean Validation 2.0 • Java Champion • @gunnarmorling
  3. #CDCPatterns @gunnarmorling @hpgrahsl Hans-Peter Grahsl • Technical Trainer at NETCONOMY

    • Independent Engineer & Consultant • Confluent Community Catalyst • MongoDB Champion • @hpgrahsl
  4. Today’s Objective: Learn About... … implemented using Change Data Capture

  5. #CDCPatterns @gunnarmorling @hpgrahsl • Taps into TX log to capture

    INSERT/UPDATE/DELETE events • Propagated to consumers via Apache Kafka and Kafka Connect Debezium — Log-based Change Data Capture
  6. Outbox Pattern

  7. #CDCPatterns @gunnarmorling @hpgrahsl The Problem: Microservices Data Exchange • Services

    need to update their database, • send messages to other services, • and that consistently!
  8. #CDCPatterns @gunnarmorling @hpgrahsl Outbox Pattern “Dual writes” are prone to

    inconsistencies!
  9. #CDCPatterns @gunnarmorling @hpgrahsl Outbox Pattern

  10. #CDCPatterns @gunnarmorling @hpgrahsl Outbox Pattern

  11. #CDCPatterns @gunnarmorling @hpgrahsl Outbox Pattern

  12. #CDCPatterns @gunnarmorling @hpgrahsl Outbox Pattern

  13. Strangler Fig Pattern

  14. #CDCPatterns @gunnarmorling @hpgrahsl The Problem: Migrating Systems • Gradually evolve

    from old into new • Support temporary coexistence • Avoid big bang cut-over
  15. #CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

  16. #CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

  17. #CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

  18. #CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

  19. #CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

  20. #CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

  21. #CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

  22. #CDCPatterns @gunnarmorling @hpgrahsl Benefits • Incremental migration → “baby steps”

    • Pause or stop migration without losing spent efforts • Migration steps ideally reversible Rationale: ⚠ minimize risk ⚠
  23. #CDCPatterns @gunnarmorling @hpgrahsl CDC Close-Up

  24. #CDCPatterns @gunnarmorling @hpgrahsl Enhanced CDC Processing

  25. #CDCPatterns @gunnarmorling @hpgrahsl Enhanced CDC Processing

  26. #CDCPatterns @gunnarmorling @hpgrahsl Enhanced CDC Processing

  27. Saga Pattern

  28. #CDCPatterns @gunnarmorling @hpgrahsl The Problem: Long-running Business Transactions • Multiple

    services need to act collaboratively to achieve a consistent outcome • Without 2-phase commit protocols • Ensure correctness in case of failures
  29. #CDCPatterns @gunnarmorling @hpgrahsl Saga Pattern

  30. #CDCPatterns @gunnarmorling @hpgrahsl Saga Pattern

  31. #CDCPatterns @gunnarmorling @hpgrahsl Saga Pattern

  32. Demo

  33. Wrap-Up

  34. #CDCPatterns @gunnarmorling @hpgrahsl • CDC: a powerful tool in the

    box for event-driven architectures • Debezium: open-source CDC for a variety of databases • Call to Action: Would you like built-in Saga support? Takeaways
  35. #CDCPatterns @gunnarmorling @hpgrahsl • Outbox implementation https://debezium.io/blog/2019/02/19/reliable-microservices-data -exchange-with-the-outbox-pattern/ • Strangler

    fig pattern https://martinfowler.com/bliki/StranglerFigApplication.html • Saga implementation https://www.infoq.com/articles/saga-orchestration-outbox/ Resources
  36. #CDCPatterns @gunnarmorling @hpgrahsl Q & A gunnar@hibernate.org @gunnarmorling 📧 grahslhp@gmail.com

    @hpgrahsl 📧
  37. Unsplash https://unsplash.com/license © Shiro hatori https://unsplash.com/photos/WR-ifjFy4CI © Pablo García Saldaña

    https://unsplash.com/photos/lPQIndZz8Mo © David Clode https://unsplash.com/photos/T49WTav4LgU © Aaron Burden https://unsplash.com/photos/GFpxQ2ZyNc0 © Nathan Dumlao https://unsplash.com/photos/wQDysNUCKfw © mari lezhava https://unsplash.com/photos/q65bNe9fW-w © Michał Parzuchowski https://unsplash.com/photos/Bt0PM7cNJFQ © Charles Forerunner https://unsplash.com/photos/3fPXt37X6UQ Flickr Attribution 2.0 Generic https://creativecommons.org/licenses/by/2.0/ © Thomas Kamann https://flic.kr/p/coa2c CC0 1.0 Universal Public Domain Dedication https://creativecommons.org/publicdomain/zero/1.0/ © Wall Boat https://flic.kr/p/Y6zkmX Attribution-ShareAlike 2.0 Generic https://creativecommons.org/licenses/by-sa/2.0/ © Maria Eklind https://flic.kr/p/251HmVw Image Credits In Order of Appearance