Drukwerkdeal.nl Developer Meetup - Event Sourcing After Launch

Slide 1

Slide 1 text

Event Sourcing after launch how to evolve your event store along with your application Michiel Overeem, AFAS, @michielovereem

Slide 2

Slide 2 text

400+ employees (4 locations) 20 years of ERP experience 10.500 customers (companies) AFAS Software

Slide 3

Slide 3 text

HRM, CRM, finance, order management, project management, workflow, ... AFAS Profit

Slide 4

Slide 4 text

Web Scalable CQRS Event sourcing

Slide 5

Slide 5 text

Adaptable Model-based User-specific Software Ecosystems

Slide 6

Slide 6 text

Event sourcing 101 Current state Event sourcing

Slide 7

Slide 7 text

Event sourcing 101 accountId balance owner 1234567 0 Michiel Current state Event sourcing Event BankAccountCreated (accountId: 1234567, balance: 0, owner: Michiel)

Slide 8

Slide 8 text

Event sourcing 101 accountId balance owner 1234567 100 Michiel Current state Event sourcing Event BankAccountCreated (accountId: 1234567, balance: 0, owner: Michiel) DepositPerformed (accountId : 1234567, amount: 100, balance: 100)

Slide 9

Slide 9 text

Event sourcing 101 Event BankAccountCreated (accountId: 1234567, balance: 0, owner: Michiel) DepositPerformed (accountId : 1234567, amount: 100, balance: 100) WithdrawalPerformed (accountId: 1234567, amount: 50, balance: 50) accountId balance owner 1234567 50 Michiel Current state Event sourcing

Slide 10

Slide 10 text

Event sourcing 101 Event BankAccountCreated (accountId: 1234567, balance: 0, owner: Michiel) DepositPerformed (accountId : 1234567, amount: 100, balance: 100) WithdrawalPerformed (accountId: 1234567, amount: 50, balance: 50) OwnerChanged (accountId: 1234567, newOwner: Marten) accountId balance owner 1234567 50 Marten Current state Event sourcing

Slide 11

Slide 11 text

Sure, but this is normal for transactions …

Slide 12

Slide 12 text

Event sourcing 101 Event ShoppingCartCreated (cartid: 1234567, user: Michiel) cartid articles user 1234567 0 Michiel Current state Event sourcing

Slide 13

Slide 13 text

Event sourcing 101 Event ShoppingCartCreated (cartid: 1234567, user: Michiel) ArticleAddedToCart(cartid: 1234567, article: Lumia 950) cartid articles user 1234567 1 Michiel Current state Event sourcing cartid article 1234567 Lumia 950

Slide 14

Slide 14 text

Event sourcing 101 cartid articles user 1234567 0 Michiel Current state Event sourcing cartid article Event ShoppingCartCreated (cartid: 1234567, user: Michiel) ArticleAddedToCart(cartid: 1234567, article: Lumia 950) ArticleRemovedFromCart(cartid: 1234567, article: Lumia 950)

Slide 15

Slide 15 text

Event sourcing 101 cartid articles user 1234567 1 Michiel Current state Event sourcing cartid article 1234567 iPhone X Event ShoppingCartCreated (cartid: 1234567, user: Michiel) ArticleAddedToCart(cartid: 1234567, article: Lumia 950) ArticleRemovedFromCart(cartid: 1234567, article: Lumia 950) ArticleAddedToCart(cartid: 1234567, article: iPhone X)

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Why do people use this?

Slide 18

Slide 18 text

“Why was this changed?”

Slide 19

Slide 19 text

“Multiple datamodels”

Slide 20

Slide 20 text

“I know that I don't know what the future will bring”.

Slide 21

Slide 21 text

Audit, flexibility, temporal, debugging, …

Slide 22

Slide 22 text

“Versioning in event sourced systems is a big problem”

Slide 23

Slide 23 text

“We dreaded the upgrading. We had some fear in advance.”

Slide 24

Slide 24 text

“There is not much literature about it”

Slide 25

Slide 25 text

How to evolve your event store along with your application?

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

24 interviews on event sourcing and upgrades

Slide 28

Slide 28 text

Branches, domains

Slide 29

Slide 29 text

Prevention Techniques Pruning Privacy Read models

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

Which streams are in the store? Which event types are in which stream? Which attributes does an event have?

Slide 32

Slide 32 text

Applications that do not evolve in response to changing requirements or changing technology become less useful -Lehman, 1974

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

Prevention is better than a cure

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

State State’ Event

Slide 37

Slide 37 text

Events = Transactions Or not?

Slide 38

Slide 38 text

Domain-driven design Events = real world events

Slide 39

Slide 39 text

“You align the events with real world events. You are dealing with changes that have a native equivalence. Doing DDD leads to less fragile design.”

Slide 40

Slide 40 text

See what sticks Life before and after release

Slide 41

Slide 41 text

How implicit is your schema?

Slide 42

Slide 42 text

What if prevention is no longer possible?

Slide 43

Slide 43 text

Versioned Events only introduce new events and streams 2/24

Slide 44

Slide 44 text

“I try to keep my domain abstraction pure."

Slide 45

Slide 45 text

Weak Schema aka tolerant reader 11/24

Slide 46

Slide 46 text

Easy, but incomplete

Slide 47

Slide 47 text

Upcasters Transform at runtime 12/24

Slide 48

Slide 48 text

“The approach I took, and which is my impression that everyone takes …”

Slide 49

Slide 49 text

“Then we made our upcasters permanent…”

Slide 50

Slide 50 text

Immutability, not a binary option

Slide 51

Slide 51 text

“So one of the things you should never ever do is change existing events. An event written persisted to datastore is not like a hand wavy thing. It is a fact.”

Slide 52

Slide 52 text

“Historians rewrite the past too…”

Slide 53

Slide 53 text

“These rewrites aren't destructive first of all.”

Slide 54

Slide 54 text

Data Information

Slide 55

Slide 55 text

“When your events form a contract with the outside, you have to keep those.”

Slide 56

Slide 56 text

In-place transformation Just as in the old days 5/24

Slide 57

Slide 57 text

“Now that would pretty much break the whole premise of having a proper audit trail.”

Slide 58

Slide 58 text

Copy-transform When you just need to start over 14/24

Slide 59

Slide 59 text

Variations 1. Create a new store from the old one. 2. Create a new stream inside of the store, based on old streams. 3. Create new events in an existing stream.

Slide 60

Slide 60 text

“if you have like 10 billion events you just can't do it”

Slide 61

Slide 61 text

Versioned Events Weak Schema Upcasters In-Place Copy- Transform 14/24 5/24 12/24 11/24 2/24

Slide 62

Slide 62 text

Does my event store look big?

Slide 63

Slide 63 text

Year ending Classified ads

Slide 64

Slide 64 text

Privacy… Privacy

Slide 65

Slide 65 text

“I really don’t know”

Slide 66

Slide 66 text

Custom event store structure Encryption Destructive technique

Slide 67

Slide 67 text

What about all those read models?

Slide 68

Slide 68 text

Rebuild Sync

Slide 69

Slide 69 text

Prevention Techniques Pruning Privacy Read models 24 interviews on event sourcing and upgrades Event sourced systems after launch are not scary

Slide 70

Slide 70 text

It depends. Your context. Your solution.

Slide 71

Slide 71 text

So, what is our solution? 1. Copy-transform of the whole store 1. 100.000 events in 4.5 minute (around 360 per sec) 2. Rebuild query side 1. All projectors run in parallel 3. We deploy in blue-green

Slide 72

Slide 72 text

What do we (think we) want? 1. No renames 2. Weak schema: handle new attributes with ease 3. Upcasters: solve complex cases without long delays 4. Copy-transform: 1. Remove upcasters 2. Only those streams that need it 5. Only rebuild projections when necessary

Slide 73

Slide 73 text

Copyright Nasa Goddard Michiel Overeem @michielovereem Event Sourced Systems after launch: 1. Prevention 2. Techniques for upgrading 3. Prune your store 4. Think about privacy 5. Rebuild your read models