103 years of event sourcing experience
Results of our empirical research on event sourcing
Slide 2
Slide 2 text
Michiel
Lead Software Architect @ AFAS Software
PhD candidate @ University Utrecht
michiel.overeem@afas.nl
@michielovereem
Overeem
Slide 3
Slide 3 text
Continuous Migration of Mass Customized Applications
• Cloud architectures
• Model-driven development
• Evolution
Slide 4
Slide 4 text
M. Overeem, M. Spoor, and S. Jansen, “The Dark Side of Event
Sourcing: Managing Data Conversion,” in IEEE 24th
International Conference on Software Analysis, Evolution and
Reengineering (SANER), 2017, pp. 193–204.
Slide 5
Slide 5 text
Data
upgrade
Deployed with
Executed by
Executed by
Data
upgrade
Application upgrade
Lazy transformation
Upcasting
In place
transfor-
mation
Multiple versions
Copy and
transfor-
mation
Basic &
Complex
Event
Basic &
Complex
Stream
Basic
Store
Complex
Store
Big Flip
Rolling
Upgrade
Blue-Green
Expand-
Contract
Blue-
Green
Deployed with
Combined with
Weak schema
A framework based on literature and a prototype.
But do we really understand event sourcing?
Slide 6
Slide 6 text
RQ2 RQ3
RQ1
What types of
systems apply
event sourcing and
why?
How can event
sourced systems be
defined?
What are the
challenges faced in
applying event
sourcing?
Slide 7
Slide 7 text
1. Problem
2. Solution
3. Consequences
Slide 8
Slide 8 text
RQ2 RQ3
RQ1
What types of
systems apply
event sourcing and
why?
How can event
sourced systems be
defined?
What are the
challenges faced in
applying event
sourcing?
Slide 9
Slide 9 text
4
3
18
25 engineers with 103 years of
combined experience
Slide 10
Slide 10 text
And 3 engineers who are not doing event sourcing...
“An event is a discrete data object specified in domain
terms that represents a state change in an ESS.”
Slide 23
Slide 23 text
“The project function takes one or more event streams
and creates a projection with the data
from the given events.”
Slide 24
Slide 24 text
“The accept function takes a projection P and a command C.
The command is validated using the data in the projection,
and the accept function either results in an error or in an event “
Slide 25
Slide 25 text
What does
a event sourced system with CQRS
looks like?
Slide 26
Slide 26 text
No content
Slide 27
Slide 27 text
THE CONSEQUENCES
Slide 28
Slide 28 text
Steep learning curve
Learning curve
Finding your way
Slide 29
Slide 29 text
Evolution
Small steps, but how to stay stable?
Slide 30
Slide 30 text
Missing tools
Related to the learning curve? Or the trend?
Slide 31
Slide 31 text
Duplication?
Slide 32
Slide 32 text
How long do
these take?
Slide 33
Slide 33 text
Eventually consistent
Things might take a while
Slide 34
Slide 34 text
Evolution
Slide 35
Slide 35 text
Applications that do not evolve in response to changing
requirements or changing technology become less useful
-Lehman, 1974
Slide 36
Slide 36 text
Which streams are in the store?
Which event types are in which stream?
Which attributes does an event have?
Slide 37
Slide 37 text
DDD
Yes No
“You align the events with real
world events.
You are dealing with changes that
have a native equivalence.
Doing DDD leads to less fragile
design.”
Slide 38
Slide 38 text
Versioned Events (2/24)
only introduce new events and streams
“I try to keep my
domain abstraction
pure."
Slide 39
Slide 39 text
Weak Schema (11/24)
aka tolerant reader
Easy, but incomplete
Slide 40
Slide 40 text
Upcasters (12/24)
Transform at runtime
“The approach I took,
and which is my
impression that
everyone takes …”
Slide 41
Slide 41 text
Upcasters (12/24)
Transform at runtime
“Then we made our
upcasters
permanent…”
“So one of the things you
should never ever do is
change existing events. An
event written persisted to
datastore is not like a hand
wavy thing. It is a fact.”
Slide 45
Slide 45 text
“So one of the things you
should never ever do is
change existing events. An
event written persisted to
datastore is not like a hand
wavy thing. It is a fact.”
“Historians
rewrite the past
too…”
Slide 46
Slide 46 text
“So one of the things you
should never ever do is
change existing events. An
event written persisted to
datastore is not like a hand
wavy thing. It is a fact.”
“Historians
rewrite the past
too…”
“These rewrites
aren't destructive
first of all.”
Slide 47
Slide 47 text
“So one of the things you
should never ever do is
change existing events. An
event written persisted to
datastore is not like a hand
wavy thing. It is a fact.”
“Historians
rewrite the past
too…”
“These rewrites
aren't destructive
first of all.”
“When your events form a
contract with the outside,
you have to keep those.”
Slide 48
Slide 48 text
In-place transformation (5/24)
Just as in the old days
“Now that would
pretty much break the
whole premise of
having a proper audit
trail.”
Slide 49
Slide 49 text
Copy-transform (14/24)
When you just need to start over
“if you have like 10
billion events you just
can't do it”
Slide 50
Slide 50 text
Copy-transform (14/24)
When you just need to start over
“if you have like 10
billion events you just
can't do it”
1. Create a new store from the old one.
2. Create a new stream inside of the store, based on old streams.
3. Create new events in an existing stream.
Custom event
store structure Encryption
Destructive
technique
Slide 57
Slide 57 text
What about all those read models?
Slide 58
Slide 58 text
Rebuilt Sync
Slide 59
Slide 59 text
It depends. Your context. Your solution.
Slide 60
Slide 60 text
Hopefully, published soon!
M. Overeem, M. Spoor, S. Jansen, and S. Brinkkemper, “An
Empirical Characterization of Event Sourced Systems and
their Schema Evolution,” to be published.
Slide 61
Slide 61 text
Copyright Nasa Goddard
Michiel Overeem
@michielovereem
All (business) applications can apply event sourcing and
benefit from it. There is room for your solution!