Dissecting our Legacy: The Strangler Fig Pattern with Apache Kafka, Debezium and MongoDB @ Kafka Summit Americas 2021

Abstract:

There is no denying the fact that many development efforts have to be spent on existing applications - legacy that is - which typically exhibit a monolithic design based on traditional tech stacks. Thus, affected companies strive to move towards distributed architectures and modern technologies. This talk introduces you to the strangler fig pattern, which aids a smooth and step-wise migration of monolithic applications into separate services. The practical part shows how to apply this pattern to extract parts of a fictional monolith into their own service by featuring:

* Apache Kafka, the de-facto standard for event streaming
* MongoDB and its official connector for Apache Kafka
* plus Debezium, a distributed open-source change data capture platform.

After this talk, you’ll have a better understanding of and a concrete blueprint for extracting functionality from your monoliths, thereby gradually evolving into a (micro)service architecture and an en vogue tech stack.

Demo Repository: https://github.com/hpgrahsl/ks21-strangler-fig-pattern

Video Recording: https://lnkd.in/ewRQg-Y3 (behind registration)

Transcript

1. Dissecting our Legacy
   The Strangler Fig Pattern
   with Apache Kafka, Debezium and MongoDB
   Hans-Peter Grahsl
   Technical Trainer
   @hpgrahsl
   Gunnar Morling
   Software Engineer
   @gunnarmorling
   

   View Slide

2. @gunnarmorling @hpgrahsl
   

   View Slide

3. @gunnarmorling @hpgrahsl
   ● Technical Trainer at NETCONOMY
   ● Independent Consultant & Engineer
   ● Confluent Community Catalyst
   ● MongoDB Champion
   Hans-Peter Grahsl
   

   View Slide

4. @gunnarmorling @hpgrahsl
   ● Open source software engineer at Red Hat
   ○ Debezium
   ○ Quarkus
   ● Spec Lead for Bean Validation 2.0
   ● Java Champion
   Gunnar Morling
   

   View Slide

5. @gunnarmorling @hpgrahsl
   system migrations
   are challenging
   @hpgrahsl
   

   View Slide

6. @gunnarmorling @hpgrahsl
   hard cut-overs
   are very risky
   @gunnarmorling
   

   View Slide

7. @gunnarmorling @hpgrahsl
   BIG BANG
   migrations often
   doomed to failure
   @hpgrahsl
   

   View Slide

8. @gunnarmorling @hpgrahsl
   strangler fig
   pattern to
   the rescue
   @gunnarmorling
   

   View Slide

9. @gunnarmorling @hpgrahsl
   3 Strangler Fig Phases
   Eliminate
   Extract Co-Exist
   

   View Slide

10. @gunnarmorling @hpgrahsl
    System Migration Goal
    ● gradually evolve from old into new
    ● support temporary coexistence
    ● avoid big bang cut-over
    @hpgrahsl
    

    View Slide

11. @gunnarmorling @hpgrahsl
    Detour: Log-based Change-Data-Capture
    ● captures INSERT/UPDATE/DELETE events by tapping into DB’s TX log
    ● changes propagated via Apache Kafka & Kafka Connect
    

    View Slide

12. @gunnarmorling @hpgrahsl
    Detour: Data Change Events
    ● Message Key: table PK
    ● Message Value:
    ○ Old and new row state
    ○ Metadata on table, TX id, etc.
    ○ Operation type, timestamp
    ● Serialized as JSON, Avro, etc.
    

    View Slide

13. @gunnarmorling @hpgrahsl
    Detour: Data Change Events
    ● Message Key: table PK
    ● Message Value:
    ○ Old and new row state
    ○ Metadata on table, TX id, etc.
    ○ Operation type, timestamp
    ● Serialized as JSON, Avro, etc.
    

    View Slide

14. @gunnarmorling @hpgrahsl
    Detour: Data Change Events
    ● Message Key: table PK
    ● Message Value:
    ○ Old and new row state
    ○ Metadata on table, TX id, etc.
    ○ Operation type, timestamp
    ● Serialized as JSON, Avro, etc.
    

    View Slide

15. @gunnarmorling @hpgrahsl
    CDC-based Strangler Fig Pattern
    

    View Slide

16. @gunnarmorling @hpgrahsl
    CDC-based Strangler Fig Pattern
    

    View Slide

17. @gunnarmorling @hpgrahsl
    CDC-based Strangler Fig Pattern
    

    View Slide

18. @gunnarmorling @hpgrahsl
    CDC-based Strangler Fig Pattern
    

    View Slide

19. @gunnarmorling @hpgrahsl
    CDC-based Strangler Fig Pattern
    

    View Slide

20. @gunnarmorling @hpgrahsl
    CDC-based Strangler Fig Pattern
    

    View Slide

21. @gunnarmorling @hpgrahsl
    CDC Pipeline Considerations
    ● data model leaking from monolith ?
    ● “1:1 replication” → building aggregates ?
    ● transactional boundaries in the monolith ?
    @hpgrahsl
    

    View Slide

22. @gunnarmorling @hpgrahsl
    Enhanced CDC Pipelines
    

    View Slide

23. @gunnarmorling @hpgrahsl
    Enhanced CDC Pipelines
    

    View Slide

24. @gunnarmorling @hpgrahsl
    Enhanced CDC Pipelines
    

    View Slide

25. @hpgrahsl
    @gunnarmorling
    

    View Slide

26. @gunnarmorling @hpgrahsl
    Wrapping Up
    Strangler Fig Pattern Benefits
    ● incremental migration → “baby steps”
    ● pause or stop without losing spent efforts
    ● extraction steps ideally reversible
    ● rationale: ⚠ minimize risk ⚠
    @hpgrahsl
    

    View Slide

27. @gunnarmorling @hpgrahsl
    Resources
    ● MongoDB Connector for Apache Kafka: https://www.mongodb.com/kafka-connector
    ● Debezium Website: https://debezium.io/
    ● Demo Repository: https://bit.ly/ks21-am-sfp
    

    View Slide

28. Thank You!
    Join our Live Q & A
    Hans-Peter Grahsl
    Technical Trainer
    @hpgrahsl
    Gunnar Morling
    Software Engineer
    @gunnarmorling
    

    View Slide

29. @gunnarmorling @hpgrahsl
    Image Credits
    in order of appearance:
    @davidclode https://unsplash.com/photos/T49WTav4LgU
    @mparzuchowski https://unsplash.com/photos/geNNFqfvw48
    @florianolv https://unsplash.com/photos/ca7tJ48YD64
    @mischievous_penguins https://unsplash.com/photos/wqAuyugJIeU
    @jernejcic https://unsplash.com/photos/Oi31uKsnM1Q
    @ahmad_elsafty https://unsplash.com/photos/x97saLWeMNI
    @anniespratt https://unsplash.com/photos/t3IYuQZRDNE
    @janoschphotos https://unsplash.com/photos/w9rYOScXf6E
    

    View Slide