Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Drukwerkdeal.nl Developer Meetup - Event Sourcing After Launch

Drukwerkdeal.nl Developer Meetup - Event Sourcing After Launch

Event sourcing can be a solution for certain problems in your application, but it also comes with its own set of challenges. One of those is the evolution of your event schema. How should you handle changes to events? Michiel et al. found many approaches while doing research (they published earlier results in their paper “The Dark Side of Event Sourcing”) and they gained a lot of experienced from building a full-size ERP application using CQRS and event sourcing. In this talk he will show what kind of approaches they’ve found and what they used to deal with our specific challenges. He will also share early research results from interviews with over 20 different companies using event sourcing. Every request for advice can be answered with ‘it depends’, and this talk will not solve that. But he will tell you what your options are, what he think works, and what many others have done.

Michiel Overeem

March 01, 2018
Tweet

More Decks by Michiel Overeem

Other Decks in Technology

Transcript

  1. Event Sourcing after launch
    how to evolve your event store along with your application
    Michiel Overeem, AFAS, @michielovereem

    View Slide

  2. 400+ employees (4 locations)
    20 years of ERP experience
    10.500 customers (companies)
    AFAS Software

    View Slide

  3. HRM, CRM, finance, order
    management, project
    management, workflow, ...
    AFAS Profit

    View Slide

  4. Web
    Scalable
    CQRS
    Event sourcing

    View Slide

  5. Adaptable
    Model-based
    User-specific
    Software
    Ecosystems

    View Slide

  6. Event sourcing 101
    Current state Event sourcing

    View Slide

  7. Event sourcing 101
    accountId balance owner
    1234567 0 Michiel
    Current state Event sourcing
    Event
    BankAccountCreated
    (accountId: 1234567, balance: 0, owner: Michiel)

    View Slide

  8. Event sourcing 101
    accountId balance owner
    1234567 100 Michiel
    Current state Event sourcing
    Event
    BankAccountCreated
    (accountId: 1234567, balance: 0, owner: Michiel)
    DepositPerformed
    (accountId : 1234567, amount: 100, balance: 100)

    View Slide

  9. Event sourcing 101
    Event
    BankAccountCreated
    (accountId: 1234567, balance: 0, owner: Michiel)
    DepositPerformed
    (accountId : 1234567, amount: 100, balance: 100)
    WithdrawalPerformed
    (accountId: 1234567, amount: 50, balance: 50)
    accountId balance owner
    1234567 50 Michiel
    Current state Event sourcing

    View Slide

  10. Event sourcing 101
    Event
    BankAccountCreated
    (accountId: 1234567, balance: 0, owner: Michiel)
    DepositPerformed
    (accountId : 1234567, amount: 100, balance: 100)
    WithdrawalPerformed
    (accountId: 1234567, amount: 50, balance: 50)
    OwnerChanged
    (accountId: 1234567, newOwner: Marten)
    accountId balance owner
    1234567 50 Marten
    Current state Event sourcing

    View Slide

  11. Sure, but this is normal
    for transactions …

    View Slide

  12. Event sourcing 101
    Event
    ShoppingCartCreated
    (cartid: 1234567, user: Michiel)
    cartid articles user
    1234567 0 Michiel
    Current state Event sourcing

    View Slide

  13. Event sourcing 101
    Event
    ShoppingCartCreated
    (cartid: 1234567, user: Michiel)
    ArticleAddedToCart(cartid: 1234567, article: Lumia
    950)
    cartid articles user
    1234567 1 Michiel
    Current state Event sourcing
    cartid article
    1234567 Lumia 950

    View Slide

  14. Event sourcing 101
    cartid articles user
    1234567 0 Michiel
    Current state Event sourcing
    cartid article
    Event
    ShoppingCartCreated
    (cartid: 1234567, user: Michiel)
    ArticleAddedToCart(cartid: 1234567, article: Lumia
    950)
    ArticleRemovedFromCart(cartid: 1234567, article:
    Lumia 950)

    View Slide

  15. Event sourcing 101
    cartid articles user
    1234567 1 Michiel
    Current state Event sourcing
    cartid article
    1234567 iPhone X
    Event
    ShoppingCartCreated
    (cartid: 1234567, user: Michiel)
    ArticleAddedToCart(cartid: 1234567, article: Lumia
    950)
    ArticleRemovedFromCart(cartid: 1234567, article:
    Lumia 950)
    ArticleAddedToCart(cartid: 1234567, article: iPhone
    X)

    View Slide

  16. Event sourcing 101
    cartid articles user
    1234567 1 Michiel
    Current state Event sourcing
    cartid article
    1234567 iPhone X
    Event
    ShoppingCartCreated
    (cartid: 1234567, user: Michiel)
    ArticleAddedToCart(cartid: 1234567, article: Lumia
    950)
    ArticleRemovedFromCart(cartid: 1234567, article:
    Lumia 950)
    ArticleAddedToCart(cartid: 1234567, article: iPhone
    X)

    View Slide

  17. Why do people use this?

    View Slide

  18. “Why was this
    changed?”

    View Slide

  19. “Multiple datamodels”

    View Slide

  20. “I know that I don't know
    what the future will
    bring”.

    View Slide

  21. Audit, flexibility, temporal, debugging, …

    View Slide

  22. “Versioning in
    event sourced systems
    is a big problem”

    View Slide

  23. “We dreaded the upgrading.
    We had some fear in advance.”

    View Slide

  24. “There is not much literature about it”

    View Slide

  25. How to evolve your event store along with your application?

    View Slide

  26. View Slide

  27. 24 interviews
    on event sourcing
    and upgrades

    View Slide

  28. Branches, domains

    View Slide

  29. Prevention Techniques Pruning Privacy
    Read
    models

    View Slide

  30. View Slide

  31. Which streams are in the store?
    Which event types are in which stream?
    Which attributes does an event have?

    View Slide

  32. Applications that do not evolve in response to changing
    requirements or changing technology become less useful
    -Lehman, 1974

    View Slide

  33. View Slide

  34. Prevention is better than a cure

    View Slide

  35. View Slide

  36. State
    State’
    Event

    View Slide

  37. Events = Transactions
    Or not?

    View Slide

  38. Domain-driven design
    Events = real world events

    View Slide

  39. “You align the events with real world events.
    You are dealing with changes that
    have a native equivalence.
    Doing DDD leads to less fragile design.”

    View Slide

  40. See what sticks
    Life before and after release

    View Slide

  41. How implicit is your schema?

    View Slide

  42. What if prevention is no longer possible?

    View Slide

  43. Versioned Events
    only introduce new events and streams
    2/24

    View Slide

  44. “I try to keep my
    domain abstraction pure."

    View Slide

  45. Weak Schema
    aka tolerant reader
    11/24

    View Slide

  46. Easy, but incomplete

    View Slide

  47. Upcasters
    Transform at runtime
    12/24

    View Slide

  48. “The approach I took, and which is my impression that
    everyone takes …”

    View Slide

  49. “Then we made our upcasters permanent…”

    View Slide

  50. Immutability, not a binary option

    View Slide

  51. “So one of the things you should never ever do is change existing
    events. An event written persisted to datastore is not like a hand
    wavy thing. It is a fact.”

    View Slide

  52. “Historians rewrite the past too…”

    View Slide

  53. “These rewrites aren't destructive first of all.”

    View Slide

  54. Data Information

    View Slide

  55. “When your events form a contract with the outside, you
    have to keep those.”

    View Slide

  56. In-place transformation
    Just as in the old days
    5/24

    View Slide

  57. “Now that would pretty much break the whole premise of
    having a proper audit trail.”

    View Slide

  58. Copy-transform
    When you just need to start over
    14/24

    View Slide

  59. Variations
    1. Create a new store from the old one.
    2. Create a new stream inside of the store, based on old streams.
    3. Create new events in an existing stream.

    View Slide

  60. “if you have like 10 billion events you just can't do it”

    View Slide

  61. Versioned
    Events
    Weak
    Schema
    Upcasters In-Place
    Copy-
    Transform
    14/24
    5/24
    12/24
    11/24
    2/24

    View Slide

  62. Does my event store look big?

    View Slide

  63. Year ending Classified ads

    View Slide

  64. Privacy…
    Privacy

    View Slide

  65. “I really don’t know”

    View Slide

  66. Custom event
    store structure Encryption
    Destructive
    technique

    View Slide

  67. What about all those read models?

    View Slide

  68. Rebuild Sync

    View Slide

  69. Prevention Techniques Pruning Privacy
    Read
    models
    24 interviews on event sourcing and
    upgrades
    Event sourced systems after launch are not scary

    View Slide

  70. It depends. Your context. Your solution.

    View Slide

  71. So, what is our solution?
    1. Copy-transform of the whole store
    1. 100.000 events in 4.5 minute (around 360 per sec)
    2. Rebuild query side
    1. All projectors run in parallel
    3. We deploy in blue-green

    View Slide

  72. What do we (think we) want?
    1. No renames
    2. Weak schema: handle new attributes with ease
    3. Upcasters: solve complex cases without long delays
    4. Copy-transform:
    1. Remove upcasters
    2. Only those streams that need it
    5. Only rebuild projections when necessary

    View Slide

  73. Copyright Nasa Goddard
    Michiel Overeem
    @michielovereem
    Event Sourced Systems after launch:
    1. Prevention
    2. Techniques for upgrading
    3. Prune your store
    4. Think about privacy
    5. Rebuild your read models

    View Slide