Tweet
Tweet

Slide 1

Slide 1 text

Change Data Streaming Patterns in Distributed Systems Gunnar Morling Software Engineer, Red Hat @gunnarmorling Hans-Peter Grahsl Technical Trainer, Netconomy @hpgrahsl

Slide 2

Slide 2 text

#CDCPatterns @gunnarmorling @hpgrahsl … implemented using Change Data Capture Today’s Objectives

Slide 3

Slide 3 text

#CDCPatterns @gunnarmorling @hpgrahsl ● Open source software engineer at Red Hat ○ Debezium ○ Quarkus ● Spec Lead for Bean Validation 2.0 ● Java Champion ● @gunnarmorling Gunnar Morling

Slide 4

Slide 4 text

#CDCPatterns @gunnarmorling @hpgrahsl ● Technical Trainer at NETCONOMY ● Independent Engineer & Consultant ● Confluent Community Catalyst ● MongoDB Champion ● @hpgrahsl Hans-Peter Grahsl

Slide 5

Slide 5 text

#CDCPatterns @gunnarmorling @hpgrahsl ● Taps into TX log to capture INSERT/UPDATE/DELETE events ● Propagated to consumers via Apache Kafka and Kafka Connect Debezium — Log-based Change Data Capture

Slide 6

Slide 6 text

#CDCPatterns @gunnarmorling @hpgrahsl Detour: Data Change Events ● Old and new row state ● Metadata on table, TX id, etc. ● Operation type, timestamp

Slide 7

Slide 7 text

#CDCPatterns @gunnarmorling @hpgrahsl ● Old and new row state ● Metadata on table, TX id, etc. ● Operation type, timestamp Detour: Data Change Events

Slide 8

Slide 8 text

#CDCPatterns @gunnarmorling @hpgrahsl ● Old and new row state ● Metadata on table, TX id, etc. ● Operation type, timestamp Detour: Data Change Events

Slide 9

Slide 9 text

Outbox Pattern

Slide 10

Slide 10 text

#CDCPatterns @gunnarmorling @hpgrahsl ● Services need to update their database, ● send messages to other services, ● and that consistently! The Problem: Microservices Data Exchange

Slide 11

Slide 11 text

#CDCPatterns @gunnarmorling @hpgrahsl “Dual writes” are prone to inconsistencies! Outbox Pattern

Slide 12

Slide 12 text

#CDCPatterns @gunnarmorling @hpgrahsl Outbox Pattern

Slide 13

Slide 13 text

#CDCPatterns @gunnarmorling @hpgrahsl Outbox Pattern

Slide 14

Slide 14 text

#CDCPatterns @gunnarmorling @hpgrahsl Outbox Pattern

Slide 15

Slide 15 text

#CDCPatterns @gunnarmorling @hpgrahsl Outbox Pattern

Slide 16

Slide 16 text

Strangler Fig Pattern

Slide 17

Slide 17 text

#CDCPatterns @gunnarmorling @hpgrahsl ● Gradually evolve from old into new ● Support temporary coexistence ● Avoid big bang cut-over The Problem: Migrating Systems

Slide 18

Slide 18 text

#CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

Slide 19

Slide 19 text

#CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

Slide 20

Slide 20 text

#CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

Slide 21

Slide 21 text

#CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

Slide 22

Slide 22 text

#CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

Slide 23

Slide 23 text

#CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

Slide 24

Slide 24 text

#CDCPatterns @gunnarmorling @hpgrahsl Strangler Fig Pattern

Slide 25

Slide 25 text

#CDCPatterns @gunnarmorling @hpgrahsl ● Incremental migration → “baby steps” ● Pause or stop migration without losing spent efforts ● Migration steps ideally reversible Rationale: ⚠ minimize risk ⚠ Benefits

Slide 26

Slide 26 text

#CDCPatterns @gunnarmorling @hpgrahsl CDC Close-Up

Slide 27

Slide 27 text

#CDCPatterns @gunnarmorling @hpgrahsl Enhanced CDC Processing

Slide 28

Slide 28 text

#CDCPatterns @gunnarmorling @hpgrahsl Enhanced CDC Processing

Slide 29

Slide 29 text

#CDCPatterns @gunnarmorling @hpgrahsl Enhanced CDC Processing

Slide 30

Slide 30 text

Saga Pattern

Slide 31

Slide 31 text

#CDCPatterns @gunnarmorling @hpgrahsl ● Multiple services need to act collaboratively to achieve a consistent outcome ● Without 2-phase commit protocols ● Ensure correctness in case of failures The Problem: Long-running Business Transactions

Slide 32

Slide 32 text

#CDCPatterns @gunnarmorling @hpgrahsl Saga Pattern

Slide 33

Slide 33 text

#CDCPatterns @gunnarmorling @hpgrahsl Saga Pattern

Slide 34

Slide 34 text

#CDCPatterns @gunnarmorling @hpgrahsl Saga Pattern

Slide 35

Slide 35 text

Demo

Slide 36

Slide 36 text

Wrap-Up

Slide 37

Slide 37 text

#CDCPatterns @gunnarmorling @hpgrahsl ● CDC: a powerful tool in the box for event-driven architectures ● Debezium: open-source CDC for a variety of databases ● Call to Action: Would you like built-in Saga support? Takeaways

Slide 38

Slide 38 text

#CDCPatterns @gunnarmorling @hpgrahsl ● Outbox implementation https://debezium.io/blog/2019/02/19/reliable-microservices-data -exchange-with-the-outbox-pattern/ ● Strangler fig pattern https://martinfowler.com/bliki/StranglerFigApplication.html ● Saga implementation https://www.infoq.com/articles/saga-orchestration-outbox/ ● Demo repo https://github.com/debezium/debezium-examples Resources

Slide 39

Slide 39 text

#CDCPatterns @gunnarmorling @hpgrahsl Q & A [email protected] @gunnarmorling 📧 [email protected] @hpgrahsl 📧 Thank You!

Slide 40

Slide 40 text

#CDCPatterns @gunnarmorling @hpgrahsl Unsplash https://unsplash.com/license © Pablo García Saldaña https://unsplash.com/photos/lPQIndZz8Mo © David Clode https://unsplash.com/photos/T49WTav4LgU © Aaron Burden https://unsplash.com/photos/GFpxQ2ZyNc0 © Nathan Dumlao https://unsplash.com/photos/wQDysNUCKfw © mari lezhava https://unsplash.com/photos/q65bNe9fW-w © Michał Parzuchowski https://unsplash.com/photos/Bt0PM7cNJFQ © Charles Forerunner https://unsplash.com/photos/3fPXt37X6UQ Flickr Attribution 2.0 Generic https://creativecommons.org/licenses/by/2.0/ © Thomas Kamann https://flic.kr/p/coa2c CC0 1.0 Universal Public Domain Dedication https://creativecommons.org/publicdomain/zero/1.0/ © Wall Boat https://flic.kr/p/Y6zkmX Attribution-ShareAlike 2.0 Generic https://creativecommons.org/licenses/by-sa/2.0/ © Andrew Hart https://flic.kr/p/dmjkSk Image Credits In Order of Appearance