KanDDDinsky 2017 - The Dark Side of Event Sourcing: Managing Data Conversion

KanDDDinsky 2017 - The Dark Side of Event Sourcing: Managing Data Conversion

Evolving software systems often includes data schema changes, and those cause the need for data conversion. This is no different in event sourced systems.
Converting data between schemas while continuing the operation of the system is a challenge.
In this talk, we will look at the possible conversion operations, the techniques to execute them, and strategies for deployment with zero-downtime.
We will look at the possibilities and the trade-offs for designing and implementing upgrades for event sourced systems.

03dc1d993fa7d5de422c1f35d53f80e6?s=128

Michiel Overeem

October 20, 2017
Tweet

Transcript

  1. 8.
  2. 9.
  3. 10.

    Structure of an Event Store Store A (distributed) collection of

    streams. An ordered list of events Stream An ordered list of events. Event A name, attributes with values, and metadata.
  4. 11.

    The schema is implicit and hidden! Which streams are known

    in the system? Which events are allowed in which streams? Which attributes are allowed in which events?
  5. 12.

    Webshop EventStore Customer Stream ShoppingCart Stream CustomerSignedUp { … }

    UpgradedCustomerToPremiumService { … } ShoppingCartCreated { … } ArticleRemovedFromChart { … } ArticleAddedToChart { … }
  6. 13.

    Change will happen “An software system written to perform some

    real-world activity must be continually adapted or it becomes progressively less satisfactory” Lehman’s law of continuing change
  7. 14.

    Add attribute to event Customer Stream CustomerSignedUp { … }

    UpgradedCustomerToPremiumService { … }
  8. 15.

    Add attribute to event Customer Stream CustomerSignedUp { …, newsletter:

    <bool> } UpgradedCustomerToPremiumService { … }
  9. 19.

    Customer Stream CustomerSignedUp { …, firstName: <string>, surName: <string> }

    UpgradedCustomerToPremiumService { … } CustomerSignedUp { …, fullName: <string> } Split attribute
  10. 20.

    Customer Stream CustomerSignedUp { …, firstName: <string>, surName: <string> }

    UpgradedCustomerToPremiumService { … } Split attribute
  11. 21.

    Customer Stream CustomerSignedUp { … } UpgradedCustomerToPremiumService { … }

    CustomerSignedUp { …, newsletter: <bool> } Split events
  12. 23.

    Customer Stream CustomerSignedUp { … } UpgradedCustomerToPremiumService { … }

    OptInOnNewsLetter { … } CustomerSignedUp { … } UpgradedCustomerToPremiumService { … } Split streams
  13. 24.

    CustomerSignedUp { … } UpgradedCustomerToPremiumService { … } Customer Stream

    CustomerSignedUp { … } UpgradedCustomerToPremiumService { … } NewsletterSubscription Stream OptInOnNewsLetter { … } Split streams
  14. 25.

    Store level operations Add / remove streams Merge streams /

    split / … streams Stream level operations Add / remove events Merge events / split / … events Event level operations Add / update / remove attributes Merge attributes / split / … attributes The operations
  15. 28.

    Our codebase would only grow. Blue Cross Blue Shield Tower

    under construction in Chicago, by Photogonal, wikipedia.org
  16. 29.

    CustomerSignedUp { “customerId”: …, “fullName”: … } Weak schema &

    serialization CustomerSignedUp { “customerId”: …, “firstName”: …, “surName”: … }
  17. 32.
  18. 34.

    CustomerSignedUp { “customerId”: …, “fullName”: … } Upcasting CustomerSignedUp {

    “customerId”: …, “firstName”: …, “surName”: … } CustomerSignedUp { “customerId”: …, “firstName”: …, “surName”: …, “newsletter”: … } Upcaster Upcaster
  19. 38.
  20. 39.
  21. 40.
  22. 41.
  23. 42.
  24. 54.
  25. 55.

    Store A (distributed) collection of streams. An ordered list of

    events Store level Stream level Event level Add / remove streams Merge streams / split / … streams Should we perform all operations at runtime?
  26. 57.

    Event operations Stream operations Store operations simple Store operations complex

    Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform
  27. 58.

    Always extend Weak schema Upcasting Lazy transformation In place transformation

    Copy and transformation Functional suitability +/- - +/- +/- + +
  28. 59.

    Always extend Weak schema Upcasting Lazy transformation In place transformation

    Copy and transformation Maintainability - +/- +/- +/- + +
  29. 60.

    Always extend Weak schema Upcasting Lazy transformation In place transformation

    Copy and transformation Performance efficiency + + + +/- +/- -
  30. 61.
  31. 62.

    Event operations Stream operations Store operations simple Store operations complex

    Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform
  32. 64.

    Event operations Stream operations Store operations simple Store operations complex

    Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform These techniques are executed within the application These techniques need a separate stage
  33. 66.
  34. 67.
  35. 68.
  36. 69.
  37. 70.
  38. 71.
  39. 80.

    Event operations Stream operations Store operations simple Store operations complex

    Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform Big Flip Blue / Green Rolling Upgrade
  40. 89.

    Event operations Stream operations Store operations simple Store operations complex

    Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform Big Flip Blue / Green Rolling Upgrade Expand & Contract Blue / Green
  41. 90.

    Event operations Stream operations Store operations simple Store operations complex

    In place transformation Copy and transform Big Flip Blue / Green Rolling Upgrade Expand & Contract Blue / Green
  42. 91.

    Event operations Stream operations Store operations simple Store operations complex

    In place transformation Copy and transform Big Flip Blue / Green Rolling Upgrade Expand & Contract Blue / Green
  43. 92.

    Event operations Stream operations Store operations simple Store operations complex

    Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform Big Flip Blue / Green Rolling Upgrade Expand & Contract Blue / Green Mix & Match!
  44. 94.