Upgrade to Pro — share decks privately, control downloads, hide ads and more …

KanDDDinsky 2017 - The Dark Side of Event Sourcing: Managing Data Conversion

KanDDDinsky 2017 - The Dark Side of Event Sourcing: Managing Data Conversion

Evolving software systems often includes data schema changes, and those cause the need for data conversion. This is no different in event sourced systems.
Converting data between schemas while continuing the operation of the system is a challenge.
In this talk, we will look at the possible conversion operations, the techniques to execute them, and strategies for deployment with zero-downtime.
We will look at the possibilities and the trade-offs for designing and implementing upgrades for event sourced systems.

Michiel Overeem

October 20, 2017
Tweet

More Decks by Michiel Overeem

Other Decks in Technology

Transcript

  1. Copyright Nasa Goddard
    Michiel Overeem
    @michielovereem

    View full-size slide

  2. AFAS Software
    400+ employees
    10.000+ organizations

    View full-size slide

  3. AFAS Profit
    HRM, CRM, finance, order
    management, project management

    View full-size slide

  4. Model-Driven
    Generate a customized
    application with
    standardized software

    View full-size slide

  5. Management
    Zero downtime upgrades

    View full-size slide

  6. daptable
    odel-based
    and
    ser-specific
    oftware
    cosystems
    A
    M
    U
    S
    E

    View full-size slide

  7. Copyright Nasa Goddard

    View full-size slide

  8. How can an event sourced system be upgraded efficiently
    when the (implicit) event schema changes?

    View full-size slide

  9. Structure of an Event Store
    Store
    A (distributed)
    collection of streams.
    An ordered list of events
    Stream
    An ordered list of
    events.
    Event
    A name, attributes with
    values, and metadata.

    View full-size slide

  10. The schema is implicit and hidden!
    Which streams are known in the system?
    Which events are allowed in which streams?
    Which attributes are allowed in which events?

    View full-size slide

  11. Webshop EventStore
    Customer Stream
    ShoppingCart Stream
    CustomerSignedUp { … }
    UpgradedCustomerToPremiumService { … }
    ShoppingCartCreated { … }
    ArticleRemovedFromChart { … }
    ArticleAddedToChart { … }

    View full-size slide

  12. Change will happen
    “An software system written to
    perform some real-world activity
    must be continually adapted or it
    becomes progressively less
    satisfactory”
    Lehman’s law of continuing change

    View full-size slide

  13. Add attribute to event
    Customer Stream
    CustomerSignedUp { … }
    UpgradedCustomerToPremiumService { … }

    View full-size slide

  14. Add attribute to event
    Customer Stream
    CustomerSignedUp { …, newsletter: }
    UpgradedCustomerToPremiumService { … }

    View full-size slide

  15. Customer Stream
    CustomerSignedUp { … }
    UpgradedCustomerToPremiumService { … }
    Add event to stream

    View full-size slide

  16. Customer Stream
    CustomerSignedUp { … }
    UpgradedCustomerToPremiumService { … }
    OptInOnNewsLetter { … }
    Add event to stream

    View full-size slide

  17. Newsletter Stream
    CreateNewsletter { … }
    PublishNewsletter { … }
    Add stream to store

    View full-size slide

  18. Customer Stream
    CustomerSignedUp { …, firstName: ,
    surName: }
    UpgradedCustomerToPremiumService { … }
    CustomerSignedUp { …, fullName: }
    Split attribute

    View full-size slide

  19. Customer Stream
    CustomerSignedUp { …, firstName: ,
    surName: }
    UpgradedCustomerToPremiumService { … }
    Split attribute

    View full-size slide

  20. Customer Stream
    CustomerSignedUp { … }
    UpgradedCustomerToPremiumService { … }
    CustomerSignedUp { …, newsletter: }
    Split events

    View full-size slide

  21. Customer Stream
    CustomerSignedUp { … }
    UpgradedCustomerToPremiumService { … }
    OptInOnNewsLetter { … }
    Split events

    View full-size slide

  22. Customer Stream
    CustomerSignedUp { … }
    UpgradedCustomerToPremiumService { … }
    OptInOnNewsLetter { … }
    CustomerSignedUp { … }
    UpgradedCustomerToPremiumService { … }
    Split streams

    View full-size slide

  23. CustomerSignedUp { … }
    UpgradedCustomerToPremiumService { … }
    Customer Stream
    CustomerSignedUp { … }
    UpgradedCustomerToPremiumService { … }
    NewsletterSubscription Stream
    OptInOnNewsLetter { … }
    Split streams

    View full-size slide

  24. Store level operations
    Add / remove streams
    Merge streams / split / … streams
    Stream level operations
    Add / remove events
    Merge events / split / … events
    Event level operations
    Add / update / remove attributes
    Merge attributes / split / … attributes
    The operations

    View full-size slide

  25. How can we execute these operations?

    View full-size slide

  26. Always extend
    Customer Stream
    CustomerSignedUp { … }
    UpgradedCustomerToPremiumService { … }
    CustomerSignedUp_V2 { … }

    View full-size slide

  27. Our codebase
    would only grow.
    Blue Cross Blue Shield Tower under construction in Chicago, by Photogonal, wikipedia.org

    View full-size slide

  28. CustomerSignedUp {
    “customerId”: …,
    “fullName”: …
    }
    Weak schema & serialization
    CustomerSignedUp {
    “customerId”: …,
    “firstName”: …,
    “surName”: …
    }

    View full-size slide

  29. CustomerSignedUp {
    “customerId”: …,

    }
    Weak schema & serialization

    View full-size slide

  30. If-then-else explosion?
    pexels.com

    View full-size slide

  31. Upcasters
    Upcasting

    View full-size slide

  32. CustomerSignedUp {
    “customerId”: …,
    “fullName”: …
    }
    Upcasting
    CustomerSignedUp {
    “customerId”: …,
    “firstName”: …,
    “surName”: …
    }
    CustomerSignedUp {
    “customerId”: …,
    “firstName”: …,
    “surName”: …,
    “newsletter”: …
    }
    Upcaster
    Upcaster

    View full-size slide

  33. Upcasters
    Upcasting

    View full-size slide

  34. Runtime overhead?
    pexels.com

    View full-size slide

  35. Think about the order of upgrading components!

    View full-size slide

  36. Do we need immutability?
    Copyright Stephen Cairns

    View full-size slide

  37. Data Information

    View full-size slide

  38. Upcasters
    Lazy transformation

    View full-size slide

  39. Upcasters
    Lazy transformation

    View full-size slide

  40. When will this be done?

    View full-size slide

  41. In place transformation

    View full-size slide

  42. In place transformation

    View full-size slide

  43. Not for the faint of heart?

    View full-size slide

  44. Copy and transform

    View full-size slide

  45. Copy and transform

    View full-size slide

  46. Copy and transform

    View full-size slide

  47. Store
    A (distributed)
    collection of streams.
    An ordered list of events
    Store level
    Stream level
    Event level
    Add / remove streams
    Merge streams / split / … streams
    Should we perform all operations at runtime?

    View full-size slide

  48. Let’s put all these operations together!

    View full-size slide

  49. Event operations
    Stream operations
    Store operations
    simple
    Store operations
    complex
    Always extend
    Weak schema
    Upcasting
    Lazy transformation
    In place
    transformation
    Copy and transform

    View full-size slide

  50. Always extend
    Weak schema
    Upcasting
    Lazy transformation
    In place transformation
    Copy and transformation
    Functional
    suitability
    +/-
    -
    +/-
    +/-
    +
    +

    View full-size slide

  51. Always extend
    Weak schema
    Upcasting
    Lazy transformation
    In place transformation
    Copy and transformation
    Maintainability
    -
    +/-
    +/-
    +/-
    +
    +

    View full-size slide

  52. Always extend
    Weak schema
    Upcasting
    Lazy transformation
    In place transformation
    Copy and transformation
    Performance
    efficiency
    +
    +
    +
    +/-
    +/-
    -

    View full-size slide

  53. Always extend
    Weak schema
    Upcasting
    Lazy transformation
    In place transformation
    Copy and transformation
    Reliability
    +
    +
    +
    -
    -
    +

    View full-size slide

  54. Event operations
    Stream operations
    Store operations
    simple
    Store operations
    complex
    Always extend
    Weak schema
    Upcasting
    Lazy transformation
    In place
    transformation
    Copy and transform

    View full-size slide

  55. Can we execute this upgrade with zero downtime?
    Copyright SpaceX

    View full-size slide

  56. Event operations
    Stream operations
    Store operations
    simple
    Store operations
    complex
    Always extend
    Weak schema
    Upcasting
    Lazy transformation
    In place
    transformation
    Copy and transform
    These techniques
    are executed
    within the
    application
    These techniques
    need a separate
    stage

    View full-size slide

  57. Application deployments strategies

    View full-size slide

  58. Big Flip Blue / Green

    View full-size slide

  59. Big Flip Blue / Green

    View full-size slide

  60. Big Flip Blue / Green

    View full-size slide

  61. Big Flip Blue / Green

    View full-size slide

  62. Big Flip Blue / Green

    View full-size slide

  63. Big Flip Blue / Green Rolling Upgrade

    View full-size slide

  64. Big Flip Blue / Green Rolling Upgrade

    View full-size slide

  65. Big Flip Blue / Green Rolling Upgrade

    View full-size slide

  66. Event operations
    Stream operations
    Store operations
    simple
    Store operations
    complex
    Always extend
    Weak schema
    Upcasting
    Lazy transformation
    In place
    transformation
    Copy and transform
    Big Flip
    Blue / Green
    Rolling Upgrade

    View full-size slide

  67. Database deployments strategies

    View full-size slide

  68. Blue / Green

    View full-size slide

  69. Blue / Green

    View full-size slide

  70. Blue / Green

    View full-size slide

  71. Blue / Green

    View full-size slide

  72. Blue / Green Expand & Contract

    View full-size slide

  73. Blue / Green Expand & Contract

    View full-size slide

  74. Blue / Green Expand & Contract

    View full-size slide

  75. Event operations
    Stream operations
    Store operations
    simple
    Store operations
    complex
    Always extend
    Weak schema
    Upcasting
    Lazy transformation
    In place
    transformation
    Copy and transform
    Big Flip
    Blue / Green
    Rolling Upgrade
    Expand & Contract
    Blue / Green

    View full-size slide

  76. Event operations
    Stream operations
    Store operations
    simple
    Store operations
    complex
    In place
    transformation
    Copy and transform
    Big Flip
    Blue / Green
    Rolling Upgrade
    Expand & Contract
    Blue / Green

    View full-size slide

  77. Event operations
    Stream operations
    Store operations
    simple
    Store operations
    complex
    In place
    transformation
    Copy and transform
    Big Flip
    Blue / Green
    Rolling Upgrade
    Expand & Contract
    Blue / Green

    View full-size slide

  78. Event operations
    Stream operations
    Store operations
    simple
    Store operations
    complex
    Always extend
    Weak schema
    Upcasting
    Lazy transformation
    In place
    transformation
    Copy and transform
    Big Flip
    Blue / Green
    Rolling Upgrade
    Expand & Contract
    Blue / Green
    Mix & Match!

    View full-size slide

  79. What about the projections?

    View full-size slide

  80. How to keep your clients in sync?

    View full-size slide

  81. It depends. Your context. Your solution.

    View full-size slide

  82. Throw away with a reason
    It is an option!

    View full-size slide

  83. Time can be of help
    Expirations?

    View full-size slide

  84. You might not be Google
    Small enough to replay?

    View full-size slide

  85. Domain design helps
    Design and test before production

    View full-size slide

  86. Read our paper for more
    en
    And more is coming!

    View full-size slide

  87. Recommended!

    View full-size slide

  88. Copyright Nasa Goddard
    Michiel Overeem
    @michielovereem

    View full-size slide