Event Sourcing after launch
how to evolve your event store along with your application
Michiel Overeem, AFAS, @michielovereem
Slide 2
Slide 2 text
“Versioning in
event sourced systems
is a big problem”
Slide 3
Slide 3 text
“We dreaded the upgrading.
We had some fear in advance.”
Slide 4
Slide 4 text
“There is not much literature about it”
Slide 5
Slide 5 text
How to evolve your event store along with your application?
Slide 6
Slide 6 text
20 years of ERP
10.000 customers
90% cloud
Web
Scalable
CQRS
Event sourcing
Slide 7
Slide 7 text
20 years of ERP
10.000 customers
90% cloud
Person
Entity
Customer
Role
Order
Agreement
party
Spending limit
Address
Organisation
Entity
Delivery
BusinessAc tivity
party
Invoice
BusinessAc tivity
party
Address
Address
Spending limit
Spending limit
Own Organisation
Workarea
workarea
Spending limit
Put all our knowledge and
experience in a model
Slide 8
Slide 8 text
Customized
Web
Scalable
CQRS
Event sourcing
Person
Entity
Customer
Role
Order
Agreement
party
Spending limit
Address
Organisation
Entity
Delivery
BusinessAc tivity
party
Invoice
BusinessAc tivity
party
Address
Address
Spending limit
Spending limit
Own Organisation
Workarea
workarea
Spending limit
Slide 9
Slide 9 text
How to evolve your event store along with your application?
Slide 10
Slide 10 text
No content
Slide 11
Slide 11 text
24 interviews
on event sourcing
and upgrades
Slide 12
Slide 12 text
Branches, domains
Slide 13
Slide 13 text
Prevention Techniques Pruning Privacy
Read
models
Slide 14
Slide 14 text
No content
Slide 15
Slide 15 text
Which streams are in the store?
Which event types are in which stream?
Which attributes does an event have?
Slide 16
Slide 16 text
Applications that do not evolve in response to changing
requirements or changing technology become less useful
-Lehman, 1974
Slide 17
Slide 17 text
No content
Slide 18
Slide 18 text
Prevention is better than a cure
Slide 19
Slide 19 text
No content
Slide 20
Slide 20 text
No content
Slide 21
Slide 21 text
State
State’
Event
Slide 22
Slide 22 text
Events = Transactions
Or not?
Slide 23
Slide 23 text
Domain-driven design
Events = real world events
Slide 24
Slide 24 text
“You align the events with real world events.
You are dealing with changes that
have a native equivalence.
Doing DDD leads to less fragile design.”
Slide 25
Slide 25 text
See what sticks
Life before and after release
Slide 26
Slide 26 text
How implicit is your schema?
Slide 27
Slide 27 text
What if prevention is no longer possible?
Slide 28
Slide 28 text
Versioned Events
only introduce new events and streams
2/24
Slide 29
Slide 29 text
“I try to keep my
domain abstraction pure."
Slide 30
Slide 30 text
Weak Schema
aka tolerant reader
11/24
Slide 31
Slide 31 text
Easy, but incomplete
Slide 32
Slide 32 text
Upcasters
Transform at runtime
12/24
Slide 33
Slide 33 text
“The approach I took, and which is my impression that
everyone takes …”
Slide 34
Slide 34 text
“Then we made our upcasters permanent…”
Slide 35
Slide 35 text
Immutability, not a binaire option
Slide 36
Slide 36 text
“So one of the things you should never ever do is change existing
events. An event written persisted to datastore is not like a hand
wavy thing. It is a fact.”
Slide 37
Slide 37 text
“Historians rewrite the past too…”
Slide 38
Slide 38 text
“These rewrites aren't destructive first of all.”
Slide 39
Slide 39 text
Data Information
Slide 40
Slide 40 text
“When your events form a contract with the outside, you
have to keep those.”
Slide 41
Slide 41 text
In-place transformation
Just as in the old days
5/24
Slide 42
Slide 42 text
“Now that would pretty much break the whole premise of
having a proper audit trail.”
Slide 43
Slide 43 text
Copy-transform
When you just need to start over
14/24
Slide 44
Slide 44 text
Variations
1. Create a new store from the old one.
2. Create a new stream inside of the store, based on old streams.
3. Create new events in an existing stream.
Slide 45
Slide 45 text
“if you have like 10 billion events you just can't do it”
Custom event
store structure Encryption
Destructive
technique
Slide 52
Slide 52 text
What about all those read models?
Slide 53
Slide 53 text
Rebuilt Sync
Slide 54
Slide 54 text
Prevention Techniques Pruning Privacy
Read
models
24 interviews on event sourcing and
upgrades
Event sourced systems after launch are not scary
Slide 55
Slide 55 text
It depends. Your context. Your solution.
Slide 56
Slide 56 text
So, what is our solution?
1. Copy-transform of the whole store
1. 100.000 events in 4.5 minute (around 360 per sec)
2. Rebuilt query side
1. All projectors run in parallel
3. We deploy in blue-green
Slide 57
Slide 57 text
What do we (think we) want?
1. Renames are solved on the model level
2. Weak schema: handle new attributes with ease
3. Upcasters: solve complex cases without long delays
4. Copy-transform:
1. Remove upcasters
2. Only those streams that need it
5. Only rebuilt projections when necessary
Slide 58
Slide 58 text
Copyright Nasa Goddard
Michiel Overeem
@michielovereem
Event Sourced Systems after launch:
1. Prevention
2. Techniques for upgrading
3. Prune your store
4. Think about privacy
5. Rebuilt your read models