Drukwerkdeal.nl Developer Meetup - Event Sourcing After Launch

Drukwerkdeal.nl Developer Meetup - Event Sourcing After Launch

Event sourcing can be a solution for certain problems in your application, but it also comes with its own set of challenges. One of those is the evolution of your event schema. How should you handle changes to events? Michiel et al. found many approaches while doing research (they published earlier results in their paper “The Dark Side of Event Sourcing”) and they gained a lot of experienced from building a full-size ERP application using CQRS and event sourcing. In this talk he will show what kind of approaches they’ve found and what they used to deal with our specific challenges. He will also share early research results from interviews with over 20 different companies using event sourcing. Every request for advice can be answered with ‘it depends’, and this talk will not solve that. But he will tell you what your options are, what he think works, and what many others have done.

03dc1d993fa7d5de422c1f35d53f80e6?s=128

Michiel Overeem

March 01, 2018
Tweet

Transcript

  1. Event Sourcing after launch how to evolve your event store

    along with your application Michiel Overeem, AFAS, @michielovereem
  2. 400+ employees (4 locations) 20 years of ERP experience 10.500

    customers (companies) AFAS Software
  3. HRM, CRM, finance, order management, project management, workflow, ... AFAS

    Profit
  4. Web Scalable CQRS Event sourcing

  5. Adaptable Model-based User-specific Software Ecosystems

  6. Event sourcing 101 Current state Event sourcing

  7. Event sourcing 101 accountId balance owner 1234567 0 Michiel Current

    state Event sourcing Event BankAccountCreated (accountId: 1234567, balance: 0, owner: Michiel)
  8. Event sourcing 101 accountId balance owner 1234567 100 Michiel Current

    state Event sourcing Event BankAccountCreated (accountId: 1234567, balance: 0, owner: Michiel) DepositPerformed (accountId : 1234567, amount: 100, balance: 100)
  9. Event sourcing 101 Event BankAccountCreated (accountId: 1234567, balance: 0, owner:

    Michiel) DepositPerformed (accountId : 1234567, amount: 100, balance: 100) WithdrawalPerformed (accountId: 1234567, amount: 50, balance: 50) accountId balance owner 1234567 50 Michiel Current state Event sourcing
  10. Event sourcing 101 Event BankAccountCreated (accountId: 1234567, balance: 0, owner:

    Michiel) DepositPerformed (accountId : 1234567, amount: 100, balance: 100) WithdrawalPerformed (accountId: 1234567, amount: 50, balance: 50) OwnerChanged (accountId: 1234567, newOwner: Marten) accountId balance owner 1234567 50 Marten Current state Event sourcing
  11. Sure, but this is normal for transactions …

  12. Event sourcing 101 Event ShoppingCartCreated (cartid: 1234567, user: Michiel) cartid

    articles user 1234567 0 Michiel Current state Event sourcing
  13. Event sourcing 101 Event ShoppingCartCreated (cartid: 1234567, user: Michiel) ArticleAddedToCart(cartid:

    1234567, article: Lumia 950) cartid articles user 1234567 1 Michiel Current state Event sourcing cartid article 1234567 Lumia 950
  14. Event sourcing 101 cartid articles user 1234567 0 Michiel Current

    state Event sourcing cartid article Event ShoppingCartCreated (cartid: 1234567, user: Michiel) ArticleAddedToCart(cartid: 1234567, article: Lumia 950) ArticleRemovedFromCart(cartid: 1234567, article: Lumia 950)
  15. Event sourcing 101 cartid articles user 1234567 1 Michiel Current

    state Event sourcing cartid article 1234567 iPhone X Event ShoppingCartCreated (cartid: 1234567, user: Michiel) ArticleAddedToCart(cartid: 1234567, article: Lumia 950) ArticleRemovedFromCart(cartid: 1234567, article: Lumia 950) ArticleAddedToCart(cartid: 1234567, article: iPhone X)
  16. Event sourcing 101 cartid articles user 1234567 1 Michiel Current

    state Event sourcing cartid article 1234567 iPhone X Event ShoppingCartCreated (cartid: 1234567, user: Michiel) ArticleAddedToCart(cartid: 1234567, article: Lumia 950) ArticleRemovedFromCart(cartid: 1234567, article: Lumia 950) ArticleAddedToCart(cartid: 1234567, article: iPhone X)
  17. Why do people use this?

  18. “Why was this changed?”

  19. “Multiple datamodels”

  20. “I know that I don't know what the future will

    bring”.
  21. Audit, flexibility, temporal, debugging, …

  22. “Versioning in event sourced systems is a big problem”

  23. “We dreaded the upgrading. We had some fear in advance.”

  24. “There is not much literature about it”

  25. How to evolve your event store along with your application?

  26. None
  27. 24 interviews on event sourcing and upgrades

  28. Branches, domains

  29. Prevention Techniques Pruning Privacy Read models

  30. None
  31. Which streams are in the store? Which event types are

    in which stream? Which attributes does an event have?
  32. Applications that do not evolve in response to changing requirements

    or changing technology become less useful -Lehman, 1974
  33. None
  34. Prevention is better than a cure

  35. None
  36. State State’ Event

  37. Events = Transactions Or not?

  38. Domain-driven design Events = real world events

  39. “You align the events with real world events. You are

    dealing with changes that have a native equivalence. Doing DDD leads to less fragile design.”
  40. See what sticks Life before and after release

  41. How implicit is your schema?

  42. What if prevention is no longer possible?

  43. Versioned Events only introduce new events and streams 2/24

  44. “I try to keep my domain abstraction pure."

  45. Weak Schema aka tolerant reader 11/24

  46. Easy, but incomplete

  47. Upcasters Transform at runtime 12/24

  48. “The approach I took, and which is my impression that

    everyone takes …”
  49. “Then we made our upcasters permanent…”

  50. Immutability, not a binary option

  51. “So one of the things you should never ever do

    is change existing events. An event written persisted to datastore is not like a hand wavy thing. It is a fact.”
  52. “Historians rewrite the past too…”

  53. “These rewrites aren't destructive first of all.”

  54. Data Information

  55. “When your events form a contract with the outside, you

    have to keep those.”
  56. In-place transformation Just as in the old days 5/24

  57. “Now that would pretty much break the whole premise of

    having a proper audit trail.”
  58. Copy-transform When you just need to start over 14/24

  59. Variations 1. Create a new store from the old one.

    2. Create a new stream inside of the store, based on old streams. 3. Create new events in an existing stream.
  60. “if you have like 10 billion events you just can't

    do it”
  61. Versioned Events Weak Schema Upcasters In-Place Copy- Transform 14/24 5/24

    12/24 11/24 2/24
  62. Does my event store look big?

  63. Year ending Classified ads

  64. Privacy… Privacy

  65. “I really don’t know”

  66. Custom event store structure Encryption Destructive technique

  67. What about all those read models?

  68. Rebuild Sync

  69. Prevention Techniques Pruning Privacy Read models 24 interviews on event

    sourcing and upgrades Event sourced systems after launch are not scary
  70. It depends. Your context. Your solution.

  71. So, what is our solution? 1. Copy-transform of the whole

    store 1. 100.000 events in 4.5 minute (around 360 per sec) 2. Rebuild query side 1. All projectors run in parallel 3. We deploy in blue-green
  72. What do we (think we) want? 1. No renames 2.

    Weak schema: handle new attributes with ease 3. Upcasters: solve complex cases without long delays 4. Copy-transform: 1. Remove upcasters 2. Only those streams that need it 5. Only rebuild projections when necessary
  73. Copyright Nasa Goddard Michiel Overeem @michielovereem Event Sourced Systems after

    launch: 1. Prevention 2. Techniques for upgrading 3. Prune your store 4. Think about privacy 5. Rebuild your read models