Drukwerkdeal.nl Developer Meetup - Event Sourcing After Launch

Event sourcing can be a solution for certain problems in your application, but it also comes with its own set of challenges. One of those is the evolution of your event schema. How should you handle changes to events? Michiel et al. found many approaches while doing research (they published earlier results in their paper “The Dark Side of Event Sourcing”) and they gained a lot of experienced from building a full-size ERP application using CQRS and event sourcing. In this talk he will show what kind of approaches they’ve found and what they used to deal with our specific challenges. He will also share early research results from interviews with over 20 different companies using event sourcing. Every request for advice can be answered with ‘it depends’, and this talk will not solve that. But he will tell you what your options are, what he think works, and what many others have done.

Transcript

1. Event Sourcing after launch
   how to evolve your event store along with your application
   Michiel Overeem, AFAS, @michielovereem
   

   View Slide

2. 400+ employees (4 locations)
   20 years of ERP experience
   10.500 customers (companies)
   AFAS Software
   

   View Slide

3. HRM, CRM, finance, order
   management, project
   management, workflow, ...
   AFAS Profit
   

   View Slide

4. Web
   Scalable
   CQRS
   Event sourcing
   

   View Slide

5. Adaptable
   Model-based
   User-specific
   Software
   Ecosystems
   

   View Slide

6. Event sourcing 101
   Current state Event sourcing
   

   View Slide

7. Event sourcing 101
   accountId balance owner
   1234567 0 Michiel
   Current state Event sourcing
   Event
   BankAccountCreated
   (accountId: 1234567, balance: 0, owner: Michiel)
   

   View Slide

8. Event sourcing 101
   accountId balance owner
   1234567 100 Michiel
   Current state Event sourcing
   Event
   BankAccountCreated
   (accountId: 1234567, balance: 0, owner: Michiel)
   DepositPerformed
   (accountId : 1234567, amount: 100, balance: 100)
   

   View Slide

9. Event sourcing 101
   Event
   BankAccountCreated
   (accountId: 1234567, balance: 0, owner: Michiel)
   DepositPerformed
   (accountId : 1234567, amount: 100, balance: 100)
   WithdrawalPerformed
   (accountId: 1234567, amount: 50, balance: 50)
   accountId balance owner
   1234567 50 Michiel
   Current state Event sourcing
   

   View Slide

10. Event sourcing 101
    Event
    BankAccountCreated
    (accountId: 1234567, balance: 0, owner: Michiel)
    DepositPerformed
    (accountId : 1234567, amount: 100, balance: 100)
    WithdrawalPerformed
    (accountId: 1234567, amount: 50, balance: 50)
    OwnerChanged
    (accountId: 1234567, newOwner: Marten)
    accountId balance owner
    1234567 50 Marten
    Current state Event sourcing
    

    View Slide

11. Sure, but this is normal
    for transactions …
    

    View Slide

12. Event sourcing 101
    Event
    ShoppingCartCreated
    (cartid: 1234567, user: Michiel)
    cartid articles user
    1234567 0 Michiel
    Current state Event sourcing
    

    View Slide

13. Event sourcing 101
    Event
    ShoppingCartCreated
    (cartid: 1234567, user: Michiel)
    ArticleAddedToCart(cartid: 1234567, article: Lumia
    950)
    cartid articles user
    1234567 1 Michiel
    Current state Event sourcing
    cartid article
    1234567 Lumia 950
    

    View Slide

14. Event sourcing 101
    cartid articles user
    1234567 0 Michiel
    Current state Event sourcing
    cartid article
    Event
    ShoppingCartCreated
    (cartid: 1234567, user: Michiel)
    ArticleAddedToCart(cartid: 1234567, article: Lumia
    950)
    ArticleRemovedFromCart(cartid: 1234567, article:
    Lumia 950)
    

    View Slide

15. Event sourcing 101
    cartid articles user
    1234567 1 Michiel
    Current state Event sourcing
    cartid article
    1234567 iPhone X
    Event
    ShoppingCartCreated
    (cartid: 1234567, user: Michiel)
    ArticleAddedToCart(cartid: 1234567, article: Lumia
    950)
    ArticleRemovedFromCart(cartid: 1234567, article:
    Lumia 950)
    ArticleAddedToCart(cartid: 1234567, article: iPhone
    X)
    

    View Slide

16. Event sourcing 101
    cartid articles user
    1234567 1 Michiel
    Current state Event sourcing
    cartid article
    1234567 iPhone X
    Event
    ShoppingCartCreated
    (cartid: 1234567, user: Michiel)
    ArticleAddedToCart(cartid: 1234567, article: Lumia
    950)
    ArticleRemovedFromCart(cartid: 1234567, article:
    Lumia 950)
    ArticleAddedToCart(cartid: 1234567, article: iPhone
    X)
    

    View Slide

17. Why do people use this?
    

    View Slide

18. “Why was this
    changed?”
    

    View Slide

19. “Multiple datamodels”
    

    View Slide

20. “I know that I don't know
    what the future will
    bring”.
    

    View Slide

21. Audit, flexibility, temporal, debugging, …
    

    View Slide

22. “Versioning in
    event sourced systems
    is a big problem”
    

    View Slide

23. “We dreaded the upgrading.
    We had some fear in advance.”
    

    View Slide

24. “There is not much literature about it”
    

    View Slide

25. How to evolve your event store along with your application?
    

    View Slide

26. View Slide

27. 24 interviews
    on event sourcing
    and upgrades
    

    View Slide

28. Branches, domains
    

    View Slide

29. Prevention Techniques Pruning Privacy
    Read
    models
    

    View Slide

30. View Slide

31. Which streams are in the store?
    Which event types are in which stream?
    Which attributes does an event have?
    

    View Slide

32. Applications that do not evolve in response to changing
    requirements or changing technology become less useful
    -Lehman, 1974
    

    View Slide

33. View Slide

34. Prevention is better than a cure
    

    View Slide

35. View Slide

36. State
    State’
    Event
    

    View Slide

37. Events = Transactions
    Or not?
    

    View Slide

38. Domain-driven design
    Events = real world events
    

    View Slide

39. “You align the events with real world events.
    You are dealing with changes that
    have a native equivalence.
    Doing DDD leads to less fragile design.”
    

    View Slide

40. See what sticks
    Life before and after release
    

    View Slide

41. How implicit is your schema?
    

    View Slide

42. What if prevention is no longer possible?
    

    View Slide

43. Versioned Events
    only introduce new events and streams
    2/24
    

    View Slide

44. “I try to keep my
    domain abstraction pure."
    

    View Slide

45. Weak Schema
    aka tolerant reader
    11/24
    

    View Slide

46. Easy, but incomplete
    

    View Slide

47. Upcasters
    Transform at runtime
    12/24
    

    View Slide

48. “The approach I took, and which is my impression that
    everyone takes …”
    

    View Slide

49. “Then we made our upcasters permanent…”
    

    View Slide

50. Immutability, not a binary option
    

    View Slide

51. “So one of the things you should never ever do is change existing
    events. An event written persisted to datastore is not like a hand
    wavy thing. It is a fact.”
    

    View Slide

52. “Historians rewrite the past too…”
    

    View Slide

53. “These rewrites aren't destructive first of all.”
    

    View Slide

54. Data Information
    

    View Slide

55. “When your events form a contract with the outside, you
    have to keep those.”
    

    View Slide

56. In-place transformation
    Just as in the old days
    5/24
    

    View Slide

57. “Now that would pretty much break the whole premise of
    having a proper audit trail.”
    

    View Slide

58. Copy-transform
    When you just need to start over
    14/24
    

    View Slide

59. Variations
    1. Create a new store from the old one.
    2. Create a new stream inside of the store, based on old streams.
    3. Create new events in an existing stream.
    

    View Slide

60. “if you have like 10 billion events you just can't do it”
    

    View Slide

61. Versioned
    Events
    Weak
    Schema
    Upcasters In-Place
    Copy-
    Transform
    14/24
    5/24
    12/24
    11/24
    2/24
    

    View Slide

62. Does my event store look big?
    

    View Slide

63. Year ending Classified ads
    

    View Slide

64. Privacy…
    Privacy
    

    View Slide

65. “I really don’t know”
    

    View Slide

66. Custom event
    store structure Encryption
    Destructive
    technique
    

    View Slide

67. What about all those read models?
    

    View Slide

68. Rebuild Sync
    

    View Slide

69. Prevention Techniques Pruning Privacy
    Read
    models
    24 interviews on event sourcing and
    upgrades
    Event sourced systems after launch are not scary
    

    View Slide

70. It depends. Your context. Your solution.
    

    View Slide

71. So, what is our solution?
    1. Copy-transform of the whole store
    1. 100.000 events in 4.5 minute (around 360 per sec)
    2. Rebuild query side
    1. All projectors run in parallel
    3. We deploy in blue-green
    

    View Slide

72. What do we (think we) want?
    1. No renames
    2. Weak schema: handle new attributes with ease
    3. Upcasters: solve complex cases without long delays
    4. Copy-transform:
    1. Remove upcasters
    2. Only those streams that need it
    5. Only rebuild projections when necessary
    

    View Slide

73. Copyright Nasa Goddard
    Michiel Overeem
    @michielovereem
    Event Sourced Systems after launch:
    1. Prevention
    2. Techniques for upgrading
    3. Prune your store
    4. Think about privacy
    5. Rebuild your read models
    

    View Slide