The Dark Side of Event Sourcing
Managing Data Conversion
Michiel Overeem, Marten Spoor, Slinger Jansen
Slide 2
Slide 2 text
AFAS Software
400 employees
100 million euro revenue
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
x 10.000
Slide 5
Slide 5 text
Adaptable Model-based
and
User-specific
Software Ecosystems
Slide 6
Slide 6 text
Copyright Nasa Goddard
Slide 7
Slide 7 text
How can an event sourced
system be upgraded efficiently
when the (implicit) event
schema changes?
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
type Attribute = string * System.Object
type Event = string * Attribute[]
type EventStream = string * (Event[] * System.DateTime)[]
type EventStore = EventStream[]
The structure of
an Event Store
Slide 10
Slide 10 text
But there is also a schema
Which streams are known in the system?
Which events are allowed in which streams?
Which attributes are allowed in which events?
Copyright Stephen Cairns
Technical versus functional
immutability
Slide 30
Slide 30 text
Lazy transformation
Slide 31
Slide 31 text
But when is this finished?
Mixed appends and updates.
Slide 32
Slide 32 text
In place transformation
Just find the
events and fix
them.
Slide 33
Slide 33 text
Not for the faint of heart!
Slide 34
Slide 34 text
Copy and transformation
Slide 35
Slide 35 text
Change all the things!
Auto backup.
Could be very slow.
Slide 36
Slide 36 text
Multiple versions
Weak schema
Upcasters
Lazy transformation
In place transformation
Copy and transformation
Functional
suitability
+/-
-
+/-
+/-
+
+
Comparing the approaches
Slide 37
Slide 37 text
Multiple versions
Weak schema
Upcasters
Lazy transformation
In place transformation
Copy and transformation
Maintainability
Comparing the approaches
-
+/-
+/-
+/-
+
+
Slide 38
Slide 38 text
Multiple versions
Weak schema
Upcasters
Lazy transformation
In place transformation
Copy and transformation
Performance
efficiency
+
+
+
+/-
+/-
-
Comparing the approaches
Slide 39
Slide 39 text
Multiple versions
Weak schema
Upcasters
Lazy transformation
In place transformation
Copy and transformation
Reliability
+
+
+
-
-
+
Comparing the approaches
Slide 40
Slide 40 text
Copyright SpaceX
Slide 41
Slide 41 text
Schema
versioning
Multiple versions
Weak schema
Upcasters
Lazy transformation
In place transformation
Copy and transformation
Schema
evolution
Slide 42
Slide 42 text
Runtime
techniques
Multiple versions
Weak schema
Upcasters
Lazy transformation
In place transformation
Copy and transformation
Non-runtime
techniques
Slide 43
Slide 43 text
Big flip
Slide 44
Slide 44 text
Big flip
Slide 45
Slide 45 text
Big flip
Slide 46
Slide 46 text
Big flip
Slide 47
Slide 47 text
Big flip
Slide 48
Slide 48 text
Blue-Green
Slide 49
Slide 49 text
Blue-Green
Slide 50
Slide 50 text
Blue-Green
Slide 51
Slide 51 text
Blue-Green
Slide 52
Slide 52 text
Blue-Green
Slide 53
Slide 53 text
Rolling upgrade
Slide 54
Slide 54 text
Rolling upgrade
Slide 55
Slide 55 text
Rolling upgrade
Slide 56
Slide 56 text
Rolling upgrade
Slide 57
Slide 57 text
Rolling upgrade
Slide 58
Slide 58 text
But for schema evolution we
need another strategy for
database deployment.
Slide 59
Slide 59 text
Blue-Green
Slide 60
Slide 60 text
Blue-Green
Slide 61
Slide 61 text
Blue-Green
Slide 62
Slide 62 text
Expand-
Contract
Slide 63
Slide 63 text
Expand-
Contract
Slide 64
Slide 64 text
Expand-
Contract
Slide 65
Slide 65 text
Expand-
Contract
Slide 66
Slide 66 text
Combining
everything
into a
framework
Data
upgrade
Deployed with
Executed by
Executed by
Data
upgrade
Application upgrade
Lazy transformation
Upcasting
In place
transfor-
mation
Multiple versions
Copy and
transfor-
mation
Basic &
Complex
Event
Basic &
Complex
Stream
Basic
Store
Complex
Store
Big Flip
Rolling
Upgrade
Blue-Green
Expand-
Contract
Blue-
Green
Deployed with
Combined with
Weak schema
Slide 67
Slide 67 text
But what about the query side?
Slide 68
Slide 68 text
Early interview results
Slide 69
Slide 69 text
Do you read your events ever
again? No?
Then just change the
snapshot.
Slide 70
Slide 70 text
Move out all events belonging
to old aggregates.
Slide 71
Slide 71 text
Nobody uses lazy
transformation.
Slide 72
Slide 72 text
Domain driven design helps
in preventing changes.
Slide 73
Slide 73 text
There is a lack of
experience sharing.
Slide 74
Slide 74 text
What’s next?
1. Do interviews. Interested? Contact me!
https://goo.gl/forms/7tuBKW39EPkWnZLG3
1. Write a follow up paper with more
information.
2. Publish the experiences in some form.
Slide 75
Slide 75 text
Your context determines the
best upgrade strategy.
m.overeem@afas.nl
@michielovereem