DDDVienna - 103 years of event sourcing experience

DDDVienna - 103 years of event sourcing experience

All this talk about event sourcing, and why it is the best. But who uses it? Why do they use it? And what are the challenges that others have run into? We had exactly these questions. In order to find the answers we have interviewed 25 engineers from all over the world. Some with many years of experience, others with just a few. In total they had 103 years of experience. We asked them all about event sourcing: why would you use it, why not, what did you like, what not, and so on. Topics such as schema evolution, optimization of replay, and the steep learning curve will be discussed. This talk presents a summary of 103 years of experience with event sourcing.

03dc1d993fa7d5de422c1f35d53f80e6?s=128

Michiel Overeem

October 04, 2019
Tweet

Transcript

  1. 103 years of event sourcing experience Results of our empirical

    research on event sourcing
  2. Michiel Lead Software Architect @ AFAS Software PhD candidate @

    University Utrecht michiel.overeem@afas.nl @michielovereem Overeem
  3. Continuous Migration of Mass Customized Applications • Cloud architectures •

    Model-driven development • Evolution
  4. M. Overeem, M. Spoor, and S. Jansen, “The Dark Side

    of Event Sourcing: Managing Data Conversion,” in IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2017, pp. 193–204.
  5. Data upgrade Deployed with Executed by Executed by Data upgrade

    Application upgrade Lazy transformation Upcasting In place transfor- mation Multiple versions Copy and transfor- mation Basic & Complex Event Basic & Complex Stream Basic Store Complex Store Big Flip Rolling Upgrade Blue-Green Expand- Contract Blue- Green Deployed with Combined with Weak schema A framework based on literature and a prototype. But do we really understand event sourcing?
  6. RQ2 RQ3 RQ1 What types of systems apply event sourcing

    and why? How can event sourced systems be defined? What are the challenges faced in applying event sourcing?
  7. 1. Problem 2. Solution 3. Consequences

  8. RQ2 RQ3 RQ1 What types of systems apply event sourcing

    and why? How can event sourced systems be defined? What are the challenges faced in applying event sourcing?
  9. 4 3 18 25 engineers with 103 years of combined

    experience
  10. And 3 engineers who are not doing event sourcing...

  11. None
  12. None
  13. State State’ Event

  14. Events = Transactions Or not?

  15. THE PROBLEM

  16. Project administration Marketing automation Website building Payment processing Content management

    Classified advertising …
  17. Store (19) S M L Traffic (19) S M L

    Schema (14) S M L
  18. Rationale Flexibility Complexity Trending Audit Immutability Strict Cut-off Mutable Technology

    .NET JVM PHP Ruby,Scala,Go
  19. DDD Yes No MSA Yes No CQRS Yes No

  20. No one returned to ‘current state’!

  21. THE SOLUTION

  22. “An event is a discrete data object specified in domain

    terms that represents a state change in an ESS.”
  23. “The project function takes one or more event streams and

    creates a projection with the data from the given events.”
  24. “The accept function takes a projection P and a command

    C. The command is validated using the data in the projection, and the accept function either results in an error or in an event “
  25. What does a event sourced system with CQRS looks like?

  26. None
  27. THE CONSEQUENCES

  28. Steep learning curve Learning curve Finding your way

  29. Evolution Small steps, but how to stay stable?

  30. Missing tools Related to the learning curve? Or the trend?

  31. Duplication?

  32. How long do these take?

  33. Eventually consistent Things might take a while

  34. Evolution

  35. Applications that do not evolve in response to changing requirements

    or changing technology become less useful -Lehman, 1974
  36. Which streams are in the store? Which event types are

    in which stream? Which attributes does an event have?
  37. DDD Yes No “You align the events with real world

    events. You are dealing with changes that have a native equivalence. Doing DDD leads to less fragile design.”
  38. Versioned Events (2/24) only introduce new events and streams “I

    try to keep my domain abstraction pure."
  39. Weak Schema (11/24) aka tolerant reader Easy, but incomplete

  40. Upcasters (12/24) Transform at runtime “The approach I took, and

    which is my impression that everyone takes …”
  41. Upcasters (12/24) Transform at runtime “Then we made our upcasters

    permanent…”
  42. Immutability, not a binaire option

  43. Immutability Strict Cut-off Mutable Rationale Flexibility Complexity Trending Audit

  44. “So one of the things you should never ever do

    is change existing events. An event written persisted to datastore is not like a hand wavy thing. It is a fact.”
  45. “So one of the things you should never ever do

    is change existing events. An event written persisted to datastore is not like a hand wavy thing. It is a fact.” “Historians rewrite the past too…”
  46. “So one of the things you should never ever do

    is change existing events. An event written persisted to datastore is not like a hand wavy thing. It is a fact.” “Historians rewrite the past too…” “These rewrites aren't destructive first of all.”
  47. “So one of the things you should never ever do

    is change existing events. An event written persisted to datastore is not like a hand wavy thing. It is a fact.” “Historians rewrite the past too…” “These rewrites aren't destructive first of all.” “When your events form a contract with the outside, you have to keep those.”
  48. In-place transformation (5/24) Just as in the old days “Now

    that would pretty much break the whole premise of having a proper audit trail.”
  49. Copy-transform (14/24) When you just need to start over “if

    you have like 10 billion events you just can't do it”
  50. Copy-transform (14/24) When you just need to start over “if

    you have like 10 billion events you just can't do it” 1. Create a new store from the old one. 2. Create a new stream inside of the store, based on old streams. 3. Create new events in an existing stream.
  51. Versioned Events Weak Schema Upcasters In-Place Copy- Transform 14/24 5/24

    12/24 11/24 2/24
  52. Some last topics

  53. Does my event store look big?

  54. Year ending Classified ads

  55. Privacy… Privacy

  56. Custom event store structure Encryption Destructive technique

  57. What about all those read models?

  58. Rebuilt Sync

  59. It depends. Your context. Your solution.

  60. Hopefully, published soon! M. Overeem, M. Spoor, S. Jansen, and

    S. Brinkkemper, “An Empirical Characterization of Event Sourced Systems and their Schema Evolution,” to be published.
  61. Copyright Nasa Goddard Michiel Overeem @michielovereem All (business) applications can

    apply event sourcing and benefit from it. There is room for your solution!