Dissecting our Legacy: The Strangler Fig Pattern with Apache Kafka, Debezium and MongoDB @ Kafka Summit Americas 2021

Abstract:

There is no denying the fact that many development efforts have to be spent on existing applications - legacy that is - which typically exhibit a monolithic design based on traditional tech stacks. Thus, affected companies strive to move towards distributed architectures and modern technologies. This talk introduces you to the strangler fig pattern, which aids a smooth and step-wise migration of monolithic applications into separate services. The practical part shows how to apply this pattern to extract parts of a fictional monolith into their own service by featuring:

* Apache Kafka, the de-facto standard for event streaming
* MongoDB and its official connector for Apache Kafka
* plus Debezium, a distributed open-source change data capture platform.

After this talk, you’ll have a better understanding of and a concrete blueprint for extracting functionality from your monoliths, thereby gradually evolving into a (micro)service architecture and an en vogue tech stack.

Demo Repository: https://github.com/hpgrahsl/ks21-strangler-fig-pattern

Video Recording: https://lnkd.in/ewRQg-Y3 (behind registration)

Transcript

1. Dissecting our Legacy The Strangler Fig Pattern with Apache Kafka, Debezium and MongoDB Hans-Peter Grahsl Technical Trainer @hpgrahsl Gunnar Morling Software Engineer @gunnarmorling
2. @gunnarmorling @hpgrahsl
3. @gunnarmorling @hpgrahsl • Technical Trainer at NETCONOMY • Independent Consultant & Engineer • Confluent Community Catalyst • MongoDB Champion Hans-Peter Grahsl
4. @gunnarmorling @hpgrahsl • Open source software engineer at Red Hat ◦ Debezium ◦ Quarkus • Spec Lead for Bean Validation 2.0 • Java Champion Gunnar Morling
5. @gunnarmorling @hpgrahsl system migrations are challenging @hpgrahsl
6. @gunnarmorling @hpgrahsl hard cut-overs are very risky @gunnarmorling
7. @gunnarmorling @hpgrahsl BIG BANG migrations often doomed to failure @hpgrahsl
8. @gunnarmorling @hpgrahsl strangler fig pattern to the rescue @gunnarmorling
9. @gunnarmorling @hpgrahsl 3 Strangler Fig Phases Eliminate Extract Co-Exist
10. @gunnarmorling @hpgrahsl System Migration Goal • gradually evolve from old into new • support temporary coexistence • avoid big bang cut-over @hpgrahsl
11. @gunnarmorling @hpgrahsl Detour: Log-based Change-Data-Capture • captures INSERT/UPDATE/DELETE events by tapping into DB’s TX log • changes propagated via Apache Kafka & Kafka Connect
12. @gunnarmorling @hpgrahsl Detour: Data Change Events • Message Key: table PK • Message Value: ◦ Old and new row state ◦ Metadata on table, TX id, etc. ◦ Operation type, timestamp • Serialized as JSON, Avro, etc.
13. @gunnarmorling @hpgrahsl Detour: Data Change Events • Message Key: table PK • Message Value: ◦ Old and new row state ◦ Metadata on table, TX id, etc. ◦ Operation type, timestamp • Serialized as JSON, Avro, etc.
14. @gunnarmorling @hpgrahsl Detour: Data Change Events • Message Key: table PK • Message Value: ◦ Old and new row state ◦ Metadata on table, TX id, etc. ◦ Operation type, timestamp • Serialized as JSON, Avro, etc.
15. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern
16. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern
17. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern
18. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern
19. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern
20. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern
21. @gunnarmorling @hpgrahsl CDC Pipeline Considerations • data model leaking from monolith ? • “1:1 replication” → building aggregates ? • transactional boundaries in the monolith ? @hpgrahsl
22. @gunnarmorling @hpgrahsl Enhanced CDC Pipelines
23. @gunnarmorling @hpgrahsl Enhanced CDC Pipelines
24. @gunnarmorling @hpgrahsl Enhanced CDC Pipelines
25. @hpgrahsl @gunnarmorling
26. @gunnarmorling @hpgrahsl Wrapping Up Strangler Fig Pattern Benefits • incremental migration → “baby steps” • pause or stop without losing spent efforts • extraction steps ideally reversible • rationale: ⚠ minimize risk ⚠ @hpgrahsl
27. @gunnarmorling @hpgrahsl Resources • MongoDB Connector for Apache Kafka: https://www.mongodb.com/kafka-connector • Debezium Website: https://debezium.io/ • Demo Repository: https://bit.ly/ks21-am-sfp
28. Thank You! Join our Live Q & A Hans-Peter Grahsl Technical Trainer @hpgrahsl Gunnar Morling Software Engineer @gunnarmorling
29. @gunnarmorling @hpgrahsl Image Credits in order of appearance: @davidclode https://unsplash.com/photos/T49WTav4LgU @mparzuchowski https://unsplash.com/photos/geNNFqfvw48 @florianolv https://unsplash.com/photos/ca7tJ48YD64 @mischievous_penguins https://unsplash.com/photos/wqAuyugJIeU @jernejcic https://unsplash.com/photos/Oi31uKsnM1Q @ahmad_elsafty https://unsplash.com/photos/x97saLWeMNI @anniespratt https://unsplash.com/photos/t3IYuQZRDNE @janoschphotos https://unsplash.com/photos/w9rYOScXf6E