DDDVienna - 103 years of event sourcing experience

All this talk about event sourcing, and why it is the best. But who uses it? Why do they use it? And what are the challenges that others have run into? We had exactly these questions. In order to find the answers we have interviewed 25 engineers from all over the world. Some with many years of experience, others with just a few. In total they had 103 years of experience. We asked them all about event sourcing: why would you use it, why not, what did you like, what not, and so on. Topics such as schema evolution, optimization of replay, and the steep learning curve will be discussed. This talk presents a summary of 103 years of experience with event sourcing.

Transcript

1. 103 years of event sourcing experience
   Results of our empirical research on event sourcing
   

   View Slide

2. Michiel
   Lead Software Architect @ AFAS Software
   PhD candidate @ University Utrecht
   michiel.overeem@afas.nl
   @michielovereem
   Overeem
   

   View Slide

3. Continuous Migration of Mass Customized Applications
   • Cloud architectures
   • Model-driven development
   • Evolution
   

   View Slide

4. M. Overeem, M. Spoor, and S. Jansen, “The Dark Side of Event
   Sourcing: Managing Data Conversion,” in IEEE 24th
   International Conference on Software Analysis, Evolution and
   Reengineering (SANER), 2017, pp. 193–204.
   

   View Slide

5. Data
   upgrade
   Deployed with
   Executed by
   Executed by
   Data
   upgrade
   Application upgrade
   Lazy transformation
   Upcasting
   In place
   transfor-
   mation
   Multiple versions
   Copy and
   transfor-
   mation
   Basic &
   Complex
   Event
   Basic &
   Complex
   Stream
   Basic
   Store
   Complex
   Store
   Big Flip
   Rolling
   Upgrade
   Blue-Green
   Expand-
   Contract
   Blue-
   Green
   Deployed with
   Combined with
   Weak schema
   A framework based on literature and a prototype.
   But do we really understand event sourcing?
   

   View Slide

6. RQ2 RQ3
   RQ1
   What types of
   systems apply
   event sourcing and
   why?
   How can event
   sourced systems be
   defined?
   What are the
   challenges faced in
   applying event
   sourcing?
   

   View Slide

7. 1. Problem
   2. Solution
   3. Consequences
   

   View Slide

8. RQ2 RQ3
   RQ1
   What types of
   systems apply
   event sourcing and
   why?
   How can event
   sourced systems be
   defined?
   What are the
   challenges faced in
   applying event
   sourcing?
   

   View Slide

9. 4
   3
   18
   25 engineers with 103 years of
   combined experience
   

   View Slide

10. And 3 engineers who are not doing event sourcing...
    

    View Slide

11. View Slide

12. View Slide

13. State
    State’
    Event
    

    View Slide

14. Events = Transactions
    Or not?
    

    View Slide

15. THE PROBLEM
    

    View Slide

16. Project administration
    Marketing automation
    Website building
    Payment processing
    Content management
    Classified advertising
    …
    

    View Slide

17. Store (19)
    S M L
    Traffic (19)
    S M L
    Schema (14)
    S M L
    

    View Slide

18. Rationale
    Flexibility Complexity Trending Audit
    Immutability
    Strict Cut-off Mutable
    Technology
    .NET JVM PHP Ruby,Scala,Go
    

    View Slide

19. DDD
    Yes No
    MSA
    Yes No
    CQRS
    Yes No
    

    View Slide

20. No one returned to ‘current state’!
    

    View Slide

21. THE SOLUTION
    

    View Slide

22. “An event is a discrete data object specified in domain
    terms that represents a state change in an ESS.”
    

    View Slide

23. “The project function takes one or more event streams
    and creates a projection with the data
    from the given events.”
    

    View Slide

24. “The accept function takes a projection P and a command C.
    The command is validated using the data in the projection,
    and the accept function either results in an error or in an event “
    

    View Slide

25. What does
    a event sourced system with CQRS
    looks like?
    

    View Slide

26. View Slide

27. THE CONSEQUENCES
    

    View Slide

28. Steep learning curve
    Learning curve
    Finding your way
    

    View Slide

29. Evolution
    Small steps, but how to stay stable?
    

    View Slide

30. Missing tools
    Related to the learning curve? Or the trend?
    

    View Slide

31. Duplication?
    

    View Slide

32. How long do
    these take?
    

    View Slide

33. Eventually consistent
    Things might take a while
    

    View Slide

34. Evolution
    

    View Slide

35. Applications that do not evolve in response to changing
    requirements or changing technology become less useful
    -Lehman, 1974
    

    View Slide

36. Which streams are in the store?
    Which event types are in which stream?
    Which attributes does an event have?
    

    View Slide

37. DDD
    Yes No
    “You align the events with real
    world events.
    You are dealing with changes that
    have a native equivalence.
    Doing DDD leads to less fragile
    design.”
    

    View Slide

38. Versioned Events (2/24)
    only introduce new events and streams
    “I try to keep my
    domain abstraction
    pure."
    

    View Slide

39. Weak Schema (11/24)
    aka tolerant reader
    Easy, but incomplete
    

    View Slide

40. Upcasters (12/24)
    Transform at runtime
    “The approach I took,
    and which is my
    impression that
    everyone takes …”
    

    View Slide

41. Upcasters (12/24)
    Transform at runtime
    “Then we made our
    upcasters
    permanent…”
    

    View Slide

42. Immutability, not a binaire option
    

    View Slide

43. Immutability
    Strict Cut-off Mutable
    Rationale
    Flexibility Complexity Trending Audit
    

    View Slide

44. “So one of the things you
    should never ever do is
    change existing events. An
    event written persisted to
    datastore is not like a hand
    wavy thing. It is a fact.”
    

    View Slide

45. “So one of the things you
    should never ever do is
    change existing events. An
    event written persisted to
    datastore is not like a hand
    wavy thing. It is a fact.”
    “Historians
    rewrite the past
    too…”
    

    View Slide

46. “So one of the things you
    should never ever do is
    change existing events. An
    event written persisted to
    datastore is not like a hand
    wavy thing. It is a fact.”
    “Historians
    rewrite the past
    too…”
    “These rewrites
    aren't destructive
    first of all.”
    

    View Slide

47. “So one of the things you
    should never ever do is
    change existing events. An
    event written persisted to
    datastore is not like a hand
    wavy thing. It is a fact.”
    “Historians
    rewrite the past
    too…”
    “These rewrites
    aren't destructive
    first of all.”
    “When your events form a
    contract with the outside,
    you have to keep those.”
    

    View Slide

48. In-place transformation (5/24)
    Just as in the old days
    “Now that would
    pretty much break the
    whole premise of
    having a proper audit
    trail.”
    

    View Slide

49. Copy-transform (14/24)
    When you just need to start over
    “if you have like 10
    billion events you just
    can't do it”
    

    View Slide

50. Copy-transform (14/24)
    When you just need to start over
    “if you have like 10
    billion events you just
    can't do it”
    1. Create a new store from the old one.
    2. Create a new stream inside of the store, based on old streams.
    3. Create new events in an existing stream.
    

    View Slide

51. Versioned
    Events
    Weak
    Schema
    Upcasters In-Place
    Copy-
    Transform
    14/24
    5/24
    12/24
    11/24
    2/24
    

    View Slide

52. Some last topics
    

    View Slide

53. Does my event store look big?
    

    View Slide

54. Year ending Classified ads
    

    View Slide

55. Privacy…
    Privacy
    

    View Slide

56. Custom event
    store structure Encryption
    Destructive
    technique
    

    View Slide

57. What about all those read models?
    

    View Slide

58. Rebuilt Sync
    

    View Slide

59. It depends. Your context. Your solution.
    

    View Slide

60. Hopefully, published soon!
    M. Overeem, M. Spoor, S. Jansen, and S. Brinkkemper, “An
    Empirical Characterization of Event Sourced Systems and
    their Schema Evolution,” to be published.
    

    View Slide

61. Copyright Nasa Goddard
    Michiel Overeem
    @michielovereem
    All (business) applications can apply event sourcing and
    benefit from it. There is room for your solution!
    

    View Slide