Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dissecting our Legacy: The Strangler Fig Pattern with Apache Kafka, Debezium and MongoDB @ MongoDB.live 2021

Dissecting our Legacy: The Strangler Fig Pattern with Apache Kafka, Debezium and MongoDB @ MongoDB.live 2021

Abstract:

There is no denying the fact that many development efforts have to be spent on existing applications - legacy that is - which typically exhibit a monolithic design based on traditional tech stacks. Thus, affected companies strive to move towards distributed architectures and modern technologies. This talk introduces you to the strangler fig pattern, which aids a smooth and step-wise migration of monolithic applications into separate services. The practical part shows how to apply this pattern to extract parts of a fictional monolith into their own service by featuring:

* Apache Kafka, the de-facto standard for event streaming
* MongoDB and its official connector for Apache Kafka
* plus Debezium, a distributed open-source change data capture platform.

After this talk, you’ll have a better understanding of and a concrete blueprint for extracting functionality from your monoliths, thereby gradually evolving into a (micro)service architecture and an en vogue tech stack.

Demo Repository:
https://github.com/hpgrahsl/strangler-fig-pattern-demo

Video Recording:
https://www.youtube.com/watch?v=R1kOuvLYcYo

744f1c2c6cbea2ff5104b0ac512936bd?s=128

Hans-Peter Grahsl

July 13, 2021
Tweet

Transcript

  1. Dissecting our Legacy The Strangler Fig Pattern with Apache Kafka,

    Debezium and MongoDB Hans-Peter Grahsl Technical Trainer @hpgrahsl Gunnar Morling Software Engineer @gunnarmorling
  2. @hpgrahsl @gunnarmorling

  3. @gunnarmorling @hpgrahsl • Open source software engineer at Red Hat

    ◦ Debezium ◦ Quarkus • Spec Lead for Bean Validation 2.0 • Java Champion Gunnar Morling
  4. @gunnarmorling @hpgrahsl • Technical Trainer at NETCONOMY • Independent Consultant

    & Engineer • Confluent Community Catalyst • MongoDB Champion Hans-Peter Grahsl
  5. @gunnarmorling @hpgrahsl system migrations are challenging @hpgrahsl

  6. @gunnarmorling @hpgrahsl hard cut-overs are very risky @gunnarmorling

  7. @gunnarmorling @hpgrahsl BIG BANG migrations often doomed to failure @hpgrahsl

  8. @gunnarmorling @hpgrahsl strangler fig pattern to the rescue @gunnarmorling

  9. @gunnarmorling @hpgrahsl 3 Strangler Fig Phases Eliminate Extract Co-Exist

  10. @gunnarmorling @hpgrahsl System Migration Goal • gradually evolve from old

    into new • support temporary coexistence • avoid big bang cut-over @hpgrahsl
  11. @gunnarmorling @hpgrahsl Detour: Log-based Change-Data-Capture • captures INSERT/UPDATE/DELETE events by

    tapping into DB’s TX log • changes propagated via Apache Kafka & Kafka Connect
  12. @gunnarmorling @hpgrahsl Detour: Data Change Events • Message Key: table

    PK • Message Value: ◦ Old and new row state ◦ Metadata on table, TX id, etc. ◦ Operation type, timestamp • Serialized as JSON, Avro, etc.
  13. @gunnarmorling @hpgrahsl Detour: Data Change Events • Message Key: table

    PK • Message Value: ◦ Old and new row state ◦ Metadata on table, TX id, etc. ◦ Operation type, timestamp • Serialized as JSON, Avro, etc.
  14. @gunnarmorling @hpgrahsl Detour: Data Change Events • Message Key: table

    PK • Message Value: ◦ Old and new row state ◦ Metadata on table, TX id, etc. ◦ Operation type, timestamp • Serialized as JSON, Avro, etc.
  15. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern

  16. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern

  17. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern

  18. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern

  19. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern

  20. @gunnarmorling @hpgrahsl CDC-based Strangler Fig Pattern

  21. @gunnarmorling @hpgrahsl CDC Pipeline Considerations • data model leaking from

    monolith ? • “1:1 replication” → building aggregates ? • transactional boundaries in the monolith ? @hpgrahsl
  22. @gunnarmorling @hpgrahsl Enhanced CDC Pipelines

  23. @gunnarmorling @hpgrahsl Enhanced CDC Pipelines

  24. @gunnarmorling @hpgrahsl Enhanced CDC Pipelines

  25. @hpgrahsl @gunnarmorling

  26. @gunnarmorling @hpgrahsl Wrapping Up Strangler Fig Pattern Benefits • incremental

    migration → “baby steps” • pause or stop without losing spent efforts • extraction steps ideally reversible • rationale: ⚠ minimize risk ⚠ @hpgrahsl
  27. @gunnarmorling @hpgrahsl Resources • MongoDB Connector for Apache Kafka: https://www.mongodb.com/kafka-connector

    • Debezium Website: https://debezium.io/ • Demo Repository: https://github.com/hpgrahsl/strangler-fig-pattern-demo
  28. Thank You! Join our Live Q & A Hans-Peter Grahsl

    Technical Trainer @hpgrahsl Gunnar Morling Software Engineer @gunnarmorling
  29. @gunnarmorling @hpgrahsl Image Credits in order of appearance: @davidclode https://unsplash.com/photos/T49WTav4LgU

    @mparzuchowski https://unsplash.com/photos/geNNFqfvw48 @florianolv https://unsplash.com/photos/ca7tJ48YD64 @mischievous_penguins https://unsplash.com/photos/wqAuyugJIeU @jernejcic https://unsplash.com/photos/Oi31uKsnM1Q @ahmad_elsafty https://unsplash.com/photos/x97saLWeMNI @anniespratt https://unsplash.com/photos/t3IYuQZRDNE @janoschphotos https://unsplash.com/photos/w9rYOScXf6E