Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dissecting our Legacy: The Strangler Fig Pattern with Apache Kafka, Debezium and MongoDB @ Kafka Summit Americas 2021

Dissecting our Legacy: The Strangler Fig Pattern with Apache Kafka, Debezium and MongoDB @ Kafka Summit Americas 2021

Abstract:

There is no denying the fact that many development efforts have to be spent on existing applications - legacy that is - which typically exhibit a monolithic design based on traditional tech stacks. Thus, affected companies strive to move towards distributed architectures and modern technologies. This talk introduces you to the strangler fig pattern, which aids a smooth and step-wise migration of monolithic applications into separate services. The practical part shows how to apply this pattern to extract parts of a fictional monolith into their own service by featuring:

* Apache Kafka, the de-facto standard for event streaming
* MongoDB and its official connector for Apache Kafka
* plus Debezium, a distributed open-source change data capture platform.

After this talk, you’ll have a better understanding of and a concrete blueprint for extracting functionality from your monoliths, thereby gradually evolving into a (micro)service architecture and an en vogue tech stack.

Demo Repository: https://github.com/hpgrahsl/ks21-strangler-fig-pattern

Video Recording: https://lnkd.in/ewRQg-Y3 (behind registration)

Hans-Peter Grahsl

September 14, 2021
Tweet

More Decks by Hans-Peter Grahsl

Other Decks in Programming

Transcript

  1. Dissecting our Legacy
    The Strangler Fig Pattern
    with Apache Kafka, Debezium and MongoDB
    Hans-Peter Grahsl
    Technical Trainer
    @hpgrahsl
    Gunnar Morling
    Software Engineer
    @gunnarmorling

    View full-size slide

  2. @gunnarmorling @hpgrahsl

    View full-size slide

  3. @gunnarmorling @hpgrahsl
    ● Technical Trainer at NETCONOMY
    ● Independent Consultant & Engineer
    ● Confluent Community Catalyst
    ● MongoDB Champion
    Hans-Peter Grahsl

    View full-size slide

  4. @gunnarmorling @hpgrahsl
    ● Open source software engineer at Red Hat
    ○ Debezium
    ○ Quarkus
    ● Spec Lead for Bean Validation 2.0
    ● Java Champion
    Gunnar Morling

    View full-size slide

  5. @gunnarmorling @hpgrahsl
    system migrations
    are challenging
    @hpgrahsl

    View full-size slide

  6. @gunnarmorling @hpgrahsl
    hard cut-overs
    are very risky
    @gunnarmorling

    View full-size slide

  7. @gunnarmorling @hpgrahsl
    BIG BANG
    migrations often
    doomed to failure
    @hpgrahsl

    View full-size slide

  8. @gunnarmorling @hpgrahsl
    strangler fig
    pattern to
    the rescue
    @gunnarmorling

    View full-size slide

  9. @gunnarmorling @hpgrahsl
    3 Strangler Fig Phases
    Eliminate
    Extract Co-Exist

    View full-size slide

  10. @gunnarmorling @hpgrahsl
    System Migration Goal
    ● gradually evolve from old into new
    ● support temporary coexistence
    ● avoid big bang cut-over
    @hpgrahsl

    View full-size slide

  11. @gunnarmorling @hpgrahsl
    Detour: Log-based Change-Data-Capture
    ● captures INSERT/UPDATE/DELETE events by tapping into DB’s TX log
    ● changes propagated via Apache Kafka & Kafka Connect

    View full-size slide

  12. @gunnarmorling @hpgrahsl
    Detour: Data Change Events
    ● Message Key: table PK
    ● Message Value:
    ○ Old and new row state
    ○ Metadata on table, TX id, etc.
    ○ Operation type, timestamp
    ● Serialized as JSON, Avro, etc.

    View full-size slide

  13. @gunnarmorling @hpgrahsl
    Detour: Data Change Events
    ● Message Key: table PK
    ● Message Value:
    ○ Old and new row state
    ○ Metadata on table, TX id, etc.
    ○ Operation type, timestamp
    ● Serialized as JSON, Avro, etc.

    View full-size slide

  14. @gunnarmorling @hpgrahsl
    Detour: Data Change Events
    ● Message Key: table PK
    ● Message Value:
    ○ Old and new row state
    ○ Metadata on table, TX id, etc.
    ○ Operation type, timestamp
    ● Serialized as JSON, Avro, etc.

    View full-size slide

  15. @gunnarmorling @hpgrahsl
    CDC-based Strangler Fig Pattern

    View full-size slide

  16. @gunnarmorling @hpgrahsl
    CDC-based Strangler Fig Pattern

    View full-size slide

  17. @gunnarmorling @hpgrahsl
    CDC-based Strangler Fig Pattern

    View full-size slide

  18. @gunnarmorling @hpgrahsl
    CDC-based Strangler Fig Pattern

    View full-size slide

  19. @gunnarmorling @hpgrahsl
    CDC-based Strangler Fig Pattern

    View full-size slide

  20. @gunnarmorling @hpgrahsl
    CDC-based Strangler Fig Pattern

    View full-size slide

  21. @gunnarmorling @hpgrahsl
    CDC Pipeline Considerations
    ● data model leaking from monolith ?
    ● “1:1 replication” → building aggregates ?
    ● transactional boundaries in the monolith ?
    @hpgrahsl

    View full-size slide

  22. @gunnarmorling @hpgrahsl
    Enhanced CDC Pipelines

    View full-size slide

  23. @gunnarmorling @hpgrahsl
    Enhanced CDC Pipelines

    View full-size slide

  24. @gunnarmorling @hpgrahsl
    Enhanced CDC Pipelines

    View full-size slide

  25. @hpgrahsl
    @gunnarmorling

    View full-size slide

  26. @gunnarmorling @hpgrahsl
    Wrapping Up
    Strangler Fig Pattern Benefits
    ● incremental migration → “baby steps”
    ● pause or stop without losing spent efforts
    ● extraction steps ideally reversible
    ● rationale: ⚠ minimize risk ⚠
    @hpgrahsl

    View full-size slide

  27. @gunnarmorling @hpgrahsl
    Resources
    ● MongoDB Connector for Apache Kafka: https://www.mongodb.com/kafka-connector
    ● Debezium Website: https://debezium.io/
    ● Demo Repository: https://bit.ly/ks21-am-sfp

    View full-size slide

  28. Thank You!
    Join our Live Q & A
    Hans-Peter Grahsl
    Technical Trainer
    @hpgrahsl
    Gunnar Morling
    Software Engineer
    @gunnarmorling

    View full-size slide

  29. @gunnarmorling @hpgrahsl
    Image Credits
    in order of appearance:
    @davidclode https://unsplash.com/photos/T49WTav4LgU
    @mparzuchowski https://unsplash.com/photos/geNNFqfvw48
    @florianolv https://unsplash.com/photos/ca7tJ48YD64
    @mischievous_penguins https://unsplash.com/photos/wqAuyugJIeU
    @jernejcic https://unsplash.com/photos/Oi31uKsnM1Q
    @ahmad_elsafty https://unsplash.com/photos/x97saLWeMNI
    @anniespratt https://unsplash.com/photos/t3IYuQZRDNE
    @janoschphotos https://unsplash.com/photos/w9rYOScXf6E

    View full-size slide