Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DDDVienna - 103 years of event sourcing experience

DDDVienna - 103 years of event sourcing experience

All this talk about event sourcing, and why it is the best. But who uses it? Why do they use it? And what are the challenges that others have run into? We had exactly these questions. In order to find the answers we have interviewed 25 engineers from all over the world. Some with many years of experience, others with just a few. In total they had 103 years of experience. We asked them all about event sourcing: why would you use it, why not, what did you like, what not, and so on. Topics such as schema evolution, optimization of replay, and the steep learning curve will be discussed. This talk presents a summary of 103 years of experience with event sourcing.

Michiel Overeem

October 04, 2019
Tweet

More Decks by Michiel Overeem

Other Decks in Technology

Transcript

  1. 103 years of event sourcing experience
    Results of our empirical research on event sourcing

    View Slide

  2. Michiel
    Lead Software Architect @ AFAS Software
    PhD candidate @ University Utrecht
    [email protected]
    @michielovereem
    Overeem

    View Slide

  3. Continuous Migration of Mass Customized Applications
    • Cloud architectures
    • Model-driven development
    • Evolution

    View Slide

  4. M. Overeem, M. Spoor, and S. Jansen, “The Dark Side of Event
    Sourcing: Managing Data Conversion,” in IEEE 24th
    International Conference on Software Analysis, Evolution and
    Reengineering (SANER), 2017, pp. 193–204.

    View Slide

  5. Data
    upgrade
    Deployed with
    Executed by
    Executed by
    Data
    upgrade
    Application upgrade
    Lazy transformation
    Upcasting
    In place
    transfor-
    mation
    Multiple versions
    Copy and
    transfor-
    mation
    Basic &
    Complex
    Event
    Basic &
    Complex
    Stream
    Basic
    Store
    Complex
    Store
    Big Flip
    Rolling
    Upgrade
    Blue-Green
    Expand-
    Contract
    Blue-
    Green
    Deployed with
    Combined with
    Weak schema
    A framework based on literature and a prototype.
    But do we really understand event sourcing?

    View Slide

  6. RQ2 RQ3
    RQ1
    What types of
    systems apply
    event sourcing and
    why?
    How can event
    sourced systems be
    defined?
    What are the
    challenges faced in
    applying event
    sourcing?

    View Slide

  7. 1. Problem
    2. Solution
    3. Consequences

    View Slide

  8. RQ2 RQ3
    RQ1
    What types of
    systems apply
    event sourcing and
    why?
    How can event
    sourced systems be
    defined?
    What are the
    challenges faced in
    applying event
    sourcing?

    View Slide

  9. 4
    3
    18
    25 engineers with 103 years of
    combined experience

    View Slide

  10. And 3 engineers who are not doing event sourcing...

    View Slide

  11. View Slide

  12. View Slide

  13. State
    State’
    Event

    View Slide

  14. Events = Transactions
    Or not?

    View Slide

  15. THE PROBLEM

    View Slide

  16. Project administration
    Marketing automation
    Website building
    Payment processing
    Content management
    Classified advertising

    View Slide

  17. Store (19)
    S M L
    Traffic (19)
    S M L
    Schema (14)
    S M L

    View Slide

  18. Rationale
    Flexibility Complexity Trending Audit
    Immutability
    Strict Cut-off Mutable
    Technology
    .NET JVM PHP Ruby,Scala,Go

    View Slide

  19. DDD
    Yes No
    MSA
    Yes No
    CQRS
    Yes No

    View Slide

  20. No one returned to ‘current state’!

    View Slide

  21. THE SOLUTION

    View Slide

  22. “An event is a discrete data object specified in domain
    terms that represents a state change in an ESS.”

    View Slide

  23. “The project function takes one or more event streams
    and creates a projection with the data
    from the given events.”

    View Slide

  24. “The accept function takes a projection P and a command C.
    The command is validated using the data in the projection,
    and the accept function either results in an error or in an event “

    View Slide

  25. What does
    a event sourced system with CQRS
    looks like?

    View Slide

  26. View Slide

  27. THE CONSEQUENCES

    View Slide

  28. Steep learning curve
    Learning curve
    Finding your way

    View Slide

  29. Evolution
    Small steps, but how to stay stable?

    View Slide

  30. Missing tools
    Related to the learning curve? Or the trend?

    View Slide

  31. Duplication?

    View Slide

  32. How long do
    these take?

    View Slide

  33. Eventually consistent
    Things might take a while

    View Slide

  34. Evolution

    View Slide

  35. Applications that do not evolve in response to changing
    requirements or changing technology become less useful
    -Lehman, 1974

    View Slide

  36. Which streams are in the store?
    Which event types are in which stream?
    Which attributes does an event have?

    View Slide

  37. DDD
    Yes No
    “You align the events with real
    world events.
    You are dealing with changes that
    have a native equivalence.
    Doing DDD leads to less fragile
    design.”

    View Slide

  38. Versioned Events (2/24)
    only introduce new events and streams
    “I try to keep my
    domain abstraction
    pure."

    View Slide

  39. Weak Schema (11/24)
    aka tolerant reader
    Easy, but incomplete

    View Slide

  40. Upcasters (12/24)
    Transform at runtime
    “The approach I took,
    and which is my
    impression that
    everyone takes …”

    View Slide

  41. Upcasters (12/24)
    Transform at runtime
    “Then we made our
    upcasters
    permanent…”

    View Slide

  42. Immutability, not a binaire option

    View Slide

  43. Immutability
    Strict Cut-off Mutable
    Rationale
    Flexibility Complexity Trending Audit

    View Slide

  44. “So one of the things you
    should never ever do is
    change existing events. An
    event written persisted to
    datastore is not like a hand
    wavy thing. It is a fact.”

    View Slide

  45. “So one of the things you
    should never ever do is
    change existing events. An
    event written persisted to
    datastore is not like a hand
    wavy thing. It is a fact.”
    “Historians
    rewrite the past
    too…”

    View Slide

  46. “So one of the things you
    should never ever do is
    change existing events. An
    event written persisted to
    datastore is not like a hand
    wavy thing. It is a fact.”
    “Historians
    rewrite the past
    too…”
    “These rewrites
    aren't destructive
    first of all.”

    View Slide

  47. “So one of the things you
    should never ever do is
    change existing events. An
    event written persisted to
    datastore is not like a hand
    wavy thing. It is a fact.”
    “Historians
    rewrite the past
    too…”
    “These rewrites
    aren't destructive
    first of all.”
    “When your events form a
    contract with the outside,
    you have to keep those.”

    View Slide

  48. In-place transformation (5/24)
    Just as in the old days
    “Now that would
    pretty much break the
    whole premise of
    having a proper audit
    trail.”

    View Slide

  49. Copy-transform (14/24)
    When you just need to start over
    “if you have like 10
    billion events you just
    can't do it”

    View Slide

  50. Copy-transform (14/24)
    When you just need to start over
    “if you have like 10
    billion events you just
    can't do it”
    1. Create a new store from the old one.
    2. Create a new stream inside of the store, based on old streams.
    3. Create new events in an existing stream.

    View Slide

  51. Versioned
    Events
    Weak
    Schema
    Upcasters In-Place
    Copy-
    Transform
    14/24
    5/24
    12/24
    11/24
    2/24

    View Slide

  52. Some last topics

    View Slide

  53. Does my event store look big?

    View Slide

  54. Year ending Classified ads

    View Slide

  55. Privacy…
    Privacy

    View Slide

  56. Custom event
    store structure Encryption
    Destructive
    technique

    View Slide

  57. What about all those read models?

    View Slide

  58. Rebuilt Sync

    View Slide

  59. It depends. Your context. Your solution.

    View Slide

  60. Hopefully, published soon!
    M. Overeem, M. Spoor, S. Jansen, and S. Brinkkemper, “An
    Empirical Characterization of Event Sourced Systems and
    their Schema Evolution,” to be published.

    View Slide

  61. Copyright Nasa Goddard
    Michiel Overeem
    @michielovereem
    All (business) applications can apply event sourcing and
    benefit from it. There is room for your solution!

    View Slide