Dissecting our Legacy: The Strangler Fig Pattern with Apache Kafka, Debezium and MongoDB @ VoxxedDays Romania 2021

Abstract:

There is no denying the fact that many development efforts have to be spent on existing applications - legacy that is - which typically exhibit a monolithic design based on traditional tech stacks. Thus, affected companies strive to move towards distributed architectures and modern technologies. This talk introduces you to the strangler fig pattern, which aids a smooth and step-wise migration of monolithic applications into separate services. The practical part shows how to apply this pattern to extract parts of a fictional monolith into their own service by featuring:

* Apache Kafka, the de-facto standard for event streaming
* MongoDB and its official connector for Apache Kafka
* plus Debezium, a distributed open-source change data capture platform.

After this talk, you’ll have a better understanding of and a concrete blueprint for extracting functionality from your monoliths, thereby gradually evolving into a (micro)service architecture and an en vogue tech stack.

Demo Repository: https://github.com/hpgrahsl/voxxedromania21-sfp-demo

Video Recording:
https://www.youtube.com/watch?v=rl06nAIdIhQ

Transcript

1. Dissecting our Legacy The Strangler Fig Pattern with Apache Kafka, Debezium and MongoDB Hans-Peter Grahsl Technical Trainer @hpgrahsl Gunnar Morling Software Engineer @gunnarmorling
2. @gunnarmorling @hpgrahsl
3. @gunnarmorling @hpgrahsl • Technical Trainer at NETCONOMY • Independent Consultant & Engineer • Confluent Community Catalyst • MongoDB Champion Hans-Peter Grahsl
4. @gunnarmorling @hpgrahsl • Open source software engineer at Red Hat ◦ Debezium ◦ Quarkus • Spec Lead for Bean Validation 2.0 • Java Champion Gunnar Morling
5. @gunnarmorling @hpgrahsl system migrations are challenging @hpgrahsl
6. @gunnarmorling @hpgrahsl hard cut-overs are very risky @gunnarmorling
7. @gunnarmorling @hpgrahsl BIG BANG migrations often doomed to failure @hpgrahsl
8. @gunnarmorling @hpgrahsl strangler fig pattern to the rescue @gunnarmorling
9. @gunnarmorling @hpgrahsl 3 Strangler Fig Phases Eliminate Extract Co-Exist
10. @gunnarmorling @hpgrahsl System Migration Goal • gradually evolve from old into new • support temporary coexistence • avoid big bang cut-over @hpgrahsl
11. @gunnarmorling @hpgrahsl Open-Source Change-Data-Capture With Debezium • Captures INSERT/UPDATE/DELETE events by tapping into DB’s TX log • Changes propagated via Apache Kafka & Kafka Connect
12. @gunnarmorling @hpgrahsl Debezium in a Nutshell • Feature-rich CDC Platform ◦ Based on transaction logs ◦ Snapshotting, filtering, etc. ◦ Outbox support ◦ Web-based UI • Very active OSS community • Large production deployments
13. @gunnarmorling @hpgrahsl Debezium Connectors • Stable ◦ MySQL ◦ Postgres ◦ MongoDB ◦ SQL Server ◦ Db2 ◦ Oracle • Incubating ◦ Vitess ◦ Cassandra
14. @gunnarmorling @hpgrahsl Debezium – Deployment Alternatives
15. @gunnarmorling @hpgrahsl Debezium – Data Change Events • Message Key: table PK • Message Value: ◦ Old and new row state ◦ Metadata on table, TX id, etc. ◦ Operation type, timestamp • Serialized as JSON, Avro, etc.
16. @gunnarmorling @hpgrahsl Debezium – Data Change Events • Message Key: table PK • Message Value: ◦ Old and new row state ◦ Metadata on table, TX id, etc. ◦ Operation type, timestamp • Serialized as JSON, Avro, etc.
17. @gunnarmorling @hpgrahsl • Message Key: table PK • Message Value: ◦ Old and new row state ◦ Metadata on table, TX id, etc. ◦ Operation type, timestamp • Serialized as JSON, Avro, etc. Debezium – Data Change Events
18. @gunnarmorling @hpgrahsl
19. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern
20. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern
21. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern
22. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern
23. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern
24. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern
25. @gunnarmorling @hpgrahsl CDC Pipeline Considerations • data model leaking from monolith ? • “1:1 replication” → building aggregates ? • transactional boundaries in the monolith ? @hpgrahsl
26. @gunnarmorling @hpgrahsl Enhanced CDC Pipelines
27. @gunnarmorling @hpgrahsl Enhanced CDC Pipelines
28. @gunnarmorling @hpgrahsl Enhanced CDC Pipelines
29. @hpgrahsl @gunnarmorling
30. @gunnarmorling @hpgrahsl Wrapping Up Strangler Fig Pattern Benefits • incremental migration → “baby steps” • pause or stop without losing spent efforts • extraction steps ideally reversible • rationale: ⚠ minimize risk ⚠ @hpgrahsl
31. @gunnarmorling @hpgrahsl Resources • Debezium Website: https://debezium.io/ • MongoDB Connector for Apache Kafka: https://www.mongodb.com/kafka-connector • Debezium Example Repositories: https://github.com/debezium/debezium-examples • Demo Repository: https://bit.ly/vdr21-sfp-demo
32. Hans-Peter Grahsl Technical Trainer @hpgrahsl Gunnar Morling Software Engineer @gunnarmorling
33. @gunnarmorling @hpgrahsl Image Credits in order of appearance: @davidclode https://unsplash.com/photos/T49WTav4LgU @mparzuchowski https://unsplash.com/photos/geNNFqfvw48 @florianolv https://unsplash.com/photos/ca7tJ48YD64 @mischievous_penguins https://unsplash.com/photos/wqAuyugJIeU @jernejcic https://unsplash.com/photos/Oi31uKsnM1Q @ahmad_elsafty https://unsplash.com/photos/x97saLWeMNI @anniespratt https://unsplash.com/photos/t3IYuQZRDNE @janoschphotos https://unsplash.com/photos/w9rYOScXf6E @morvanic https://unsplash.com/photos/GiUJ02Yj_io