Save 37% off PRO during our Black Friday Sale! »

Dissecting our Legacy: The Strangler Fig Pattern with Apache Kafka, Debezium and MongoDB @ VoxxedDays Romania 2021

Dissecting our Legacy: The Strangler Fig Pattern with Apache Kafka, Debezium and MongoDB @ VoxxedDays Romania 2021

Abstract:

There is no denying the fact that many development efforts have to be spent on existing applications - legacy that is - which typically exhibit a monolithic design based on traditional tech stacks. Thus, affected companies strive to move towards distributed architectures and modern technologies. This talk introduces you to the strangler fig pattern, which aids a smooth and step-wise migration of monolithic applications into separate services. The practical part shows how to apply this pattern to extract parts of a fictional monolith into their own service by featuring:

* Apache Kafka, the de-facto standard for event streaming
* MongoDB and its official connector for Apache Kafka
* plus Debezium, a distributed open-source change data capture platform.

After this talk, you’ll have a better understanding of and a concrete blueprint for extracting functionality from your monoliths, thereby gradually evolving into a (micro)service architecture and an en vogue tech stack.

Demo Repository: https://github.com/hpgrahsl/voxxedromania21-sfp-demo

Video Recording: pending...

744f1c2c6cbea2ff5104b0ac512936bd?s=128

Hans-Peter Grahsl

October 21, 2021
Tweet

Transcript

  1. Dissecting our Legacy The Strangler Fig Pattern with Apache Kafka,

    Debezium and MongoDB Hans-Peter Grahsl Technical Trainer @hpgrahsl Gunnar Morling Software Engineer @gunnarmorling
  2. @gunnarmorling @hpgrahsl

  3. @gunnarmorling @hpgrahsl • Technical Trainer at NETCONOMY • Independent Consultant

    & Engineer • Confluent Community Catalyst • MongoDB Champion Hans-Peter Grahsl
  4. @gunnarmorling @hpgrahsl • Open source software engineer at Red Hat

    ◦ Debezium ◦ Quarkus • Spec Lead for Bean Validation 2.0 • Java Champion Gunnar Morling
  5. @gunnarmorling @hpgrahsl system migrations are challenging @hpgrahsl

  6. @gunnarmorling @hpgrahsl hard cut-overs are very risky @gunnarmorling

  7. @gunnarmorling @hpgrahsl BIG BANG migrations often doomed to failure @hpgrahsl

  8. @gunnarmorling @hpgrahsl strangler fig pattern to the rescue @gunnarmorling

  9. @gunnarmorling @hpgrahsl 3 Strangler Fig Phases Eliminate Extract Co-Exist

  10. @gunnarmorling @hpgrahsl System Migration Goal • gradually evolve from old

    into new • support temporary coexistence • avoid big bang cut-over @hpgrahsl
  11. @gunnarmorling @hpgrahsl Open-Source Change-Data-Capture With Debezium • Captures INSERT/UPDATE/DELETE events

    by tapping into DB’s TX log • Changes propagated via Apache Kafka & Kafka Connect
  12. @gunnarmorling @hpgrahsl Debezium in a Nutshell • Feature-rich CDC Platform

    ◦ Based on transaction logs ◦ Snapshotting, filtering, etc. ◦ Outbox support ◦ Web-based UI • Very active OSS community • Large production deployments
  13. @gunnarmorling @hpgrahsl Debezium Connectors • Stable ◦ MySQL ◦ Postgres

    ◦ MongoDB ◦ SQL Server ◦ Db2 ◦ Oracle • Incubating ◦ Vitess ◦ Cassandra
  14. @gunnarmorling @hpgrahsl Debezium – Deployment Alternatives

  15. @gunnarmorling @hpgrahsl Debezium – Data Change Events • Message Key:

    table PK • Message Value: ◦ Old and new row state ◦ Metadata on table, TX id, etc. ◦ Operation type, timestamp • Serialized as JSON, Avro, etc.
  16. @gunnarmorling @hpgrahsl Debezium – Data Change Events • Message Key:

    table PK • Message Value: ◦ Old and new row state ◦ Metadata on table, TX id, etc. ◦ Operation type, timestamp • Serialized as JSON, Avro, etc.
  17. @gunnarmorling @hpgrahsl • Message Key: table PK • Message Value:

    ◦ Old and new row state ◦ Metadata on table, TX id, etc. ◦ Operation type, timestamp • Serialized as JSON, Avro, etc. Debezium – Data Change Events
  18. @gunnarmorling @hpgrahsl

  19. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern

  20. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern

  21. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern

  22. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern

  23. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern

  24. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern

  25. @gunnarmorling @hpgrahsl CDC Pipeline Considerations • data model leaking from

    monolith ? • “1:1 replication” → building aggregates ? • transactional boundaries in the monolith ? @hpgrahsl
  26. @gunnarmorling @hpgrahsl Enhanced CDC Pipelines

  27. @gunnarmorling @hpgrahsl Enhanced CDC Pipelines

  28. @gunnarmorling @hpgrahsl Enhanced CDC Pipelines

  29. @hpgrahsl @gunnarmorling

  30. @gunnarmorling @hpgrahsl Wrapping Up Strangler Fig Pattern Benefits • incremental

    migration → “baby steps” • pause or stop without losing spent efforts • extraction steps ideally reversible • rationale: ⚠ minimize risk ⚠ @hpgrahsl
  31. @gunnarmorling @hpgrahsl Resources • Debezium Website: https://debezium.io/ • MongoDB Connector

    for Apache Kafka: https://www.mongodb.com/kafka-connector • Debezium Example Repositories: https://github.com/debezium/debezium-examples • Demo Repository: https://bit.ly/vdr21-sfp-demo
  32. Hans-Peter Grahsl Technical Trainer @hpgrahsl Gunnar Morling Software Engineer @gunnarmorling

  33. @gunnarmorling @hpgrahsl Image Credits in order of appearance: @davidclode https://unsplash.com/photos/T49WTav4LgU

    @mparzuchowski https://unsplash.com/photos/geNNFqfvw48 @florianolv https://unsplash.com/photos/ca7tJ48YD64 @mischievous_penguins https://unsplash.com/photos/wqAuyugJIeU @jernejcic https://unsplash.com/photos/Oi31uKsnM1Q @ahmad_elsafty https://unsplash.com/photos/x97saLWeMNI @anniespratt https://unsplash.com/photos/t3IYuQZRDNE @janoschphotos https://unsplash.com/photos/w9rYOScXf6E @morvanic https://unsplash.com/photos/GiUJ02Yj_io