EVENT SOURCING
ARCHITECTURE
USING
ELIXIR
Bruno Macabeus
Slide 2
Slide 2 text
HELLO!
͜Μʹͪʂ
Slide 3
Slide 3 text
Bruno Macabeus
github.com/macabeus
macalogs.com.br
I’m a developer
at Pagar.me ❤
Slide 4
Slide 4 text
WHAT IS EVENT
SOURCING?
Slide 5
Slide 5 text
“The core idea of event
sourcing is that whenever
we make a change to the
state of a system, we record
that state change as an
event, and we can
confidently rebuild the
system state by
reprocessing the events at
any time in the future.”
- Martin Fowler
youtu.be/STKCRSUsyP0
bit.ly/martin-event-driven
Slide 6
Slide 6 text
This architecture has many interesting benefits!
Slide 7
Slide 7 text
This architecture has many interesting benefits!
Event log provides a strong audit capability
Slide 8
Slide 8 text
This architecture has many interesting benefits!
Event log provides a strong audit capability
We can recreate historic states by replaying the event
log up to a point
Slide 9
Slide 9 text
This architecture has many interesting benefits!
We can explore alternative histories by injecting
hypothetical events when replaying
Event log provides a strong audit capability
We can recreate historic states by replaying the event
log up to a point
Slide 10
Slide 10 text
This architecture has many interesting benefits!
Event sourcing makes it plausible to have non-durable
working copies, such as a Memory Image.
We can explore alternative histories by injecting
hypothetical events when replaying
Event log provides a strong audit capability
We can recreate historic states by replaying the event
log up to a point
Slide 11
Slide 11 text
This architecture has many interesting benefits!
Event sourcing makes it plausible to have non-durable
working copies, such as a Memory Image.
We can explore alternative histories by injecting
hypothetical events when replaying
Event log provides a strong audit capability
We can recreate historic states by replaying the event
log up to a point
Slide 12
Slide 12 text
Did you know?
Database journal file and Git are examples of using event
sourcing!
Slide 13
Slide 13 text
Did you know?
Database journal file and Git are examples of using event
sourcing!
Slide 14
Slide 14 text
THE (VERY) SIMPLE
FLUX
Slide 15
Slide 15 text
User Server
Current state:
Events log:
0
Slide 16
Slide 16 text
User Server
Current state:
Events log:
0
User creates an event
Slide 17
Slide 17 text
User Server
Current state:
Events log:
0
In this example, the
user “creates an
event”. But,
on the others
implementations
of this architecture,
another part could
create the events
Slide 18
Slide 18 text
User Server
Current state:
Events log:
0
Slide 19
Slide 19 text
User Server
Current state:
Events log:
0
Event “arrives" on
the server.
On the others
implementations,
the server could “create"
the event
Slide 20
Slide 20 text
User Server
Current state:
Events log:
+1
1
Slide 21
Slide 21 text
User Server
Current state:
Events log:
+1
1
Slide 22
Slide 22 text
User Server
Current state:
Events log:
+1
1
Slide 23
Slide 23 text
User Server
Current state:
Events log:
+1
2
+1
Slide 24
Slide 24 text
User Server
Current state:
Events log:
+1
2
+1
Slide 25
Slide 25 text
User Server
Current state:
Events log:
+1
2
+1
Slide 26
Slide 26 text
User Server
Current state:
Events log:
-1
1
+1
+1
Slide 27
Slide 27 text
User Server
Current state:
Events log:
-1
1
+1
+1
It’s a non-deterministic
event
Slide 28
Slide 28 text
User Server
Current state:
Events log:
-1
1
+1
+1
Slide 29
Slide 29 text
User Server
Current state:
Events log:
-1
1
+1
+1
Slide 30
Slide 30 text
User Server
Current state:
Events log:
-1
5
+1
+1
+4
Slide 31
Slide 31 text
User Server
Current state:
Events log:
-1
5
+1
+1
+4
Now the events
log has too many
events
Slide 32
Slide 32 text
User Server
Current state:
Events log:
-1
5
+1
+1
+4
Snapshot: state 5
Slide 33
Slide 33 text
User Server
Current state:
Events log:
-1
5
+1
+1
+4
Snapshot: state 5
This is an aggregate. That
is, a stream of events
Slide 34
Slide 34 text
User Server
Current state:
Events log:
-1
5
+1
+1
+4
Snapshot: state 5
Slide 35
Slide 35 text
User Server
Current state:
Events log:
-1
5
+1
+1
+4
Snapshot: state 5
Slide 36
Slide 36 text
User Server
Current state:
Events log:
-1
6
+1
+1
+4
+1
Snapshot: state 5
Slide 37
Slide 37 text
User Server
Current state:
Events log:
-1
??
+1
+1
+4
+1
Snapshot: state 5
What if we lose
the state?
Slide 38
Slide 38 text
User Server
Current state:
Events log:
-1
??
+1
+1
+4
+1
Snapshot: state 5
We could run again the
events to restore the
previous state!
Using the events, we
could convert the
stream of events into a
structural
representation. This
process is called
“projection".
Slide 39
Slide 39 text
GIVE ME AN
EXAMPLE!
Slide 40
Slide 40 text
Let’s see an Elixir project that uses event sourcing architecture!
github.com/macabeus/event-sourcing-example
Slide 41
Slide 41 text
GIVE ME AN EXAMPLE!
HOW EVENT SOURCING WAS
APPLIED IN THIS PROJECT?
Slide 42
Slide 42 text
No content
Slide 43
Slide 43 text
No content
Slide 44
Slide 44 text
No content
Slide 45
Slide 45 text
No content
Slide 46
Slide 46 text
GIVE ME AN EXAMPLE!
API
Slide 47
Slide 47 text
No content
Slide 48
Slide 48 text
No content
Slide 49
Slide 49 text
No content
Slide 50
Slide 50 text
GIVE ME AN EXAMPLE!
EVENTS
Slide 51
Slide 51 text
No content
Slide 52
Slide 52 text
No content
Slide 53
Slide 53 text
Hey! It’s important the events be agnostic of a language,
then you’ll can work using another languages on same
project.
Flux Standard Action is an example of solution about it.
I didn't do it in my example.
My example works only on Elixir
Slide 54
Slide 54 text
GIVE ME AN EXAMPLE!
PROCESS AND BUS PIPELINE
Slide 55
Slide 55 text
No content
Slide 56
Slide 56 text
No content
Slide 57
Slide 57 text
No content
Slide 58
Slide 58 text
No content
Slide 59
Slide 59 text
No content
Slide 60
Slide 60 text
No content
Slide 61
Slide 61 text
No content
Slide 62
Slide 62 text
No content
Slide 63
Slide 63 text
No content
Slide 64
Slide 64 text
No content
Slide 65
Slide 65 text
No content
Slide 66
Slide 66 text
No content
Slide 67
Slide 67 text
GIVE ME AN EXAMPLE!
DATABASE
Slide 68
Slide 68 text
In this project, I used two databases:
Slide 69
Slide 69 text
- An in-memory database, Mnesia, using the wrapper
Amnesia
In this project, I used two databases:
Slide 70
Slide 70 text
- An in-memory database, Mnesia, using the wrapper
Amnesia
In this project, I used two databases:
- A storage file database, DETS
Slide 71
Slide 71 text
- An in-memory database, Mnesia, using the wrapper
Amnesia
In this project, I used two databases:
- A storage file database, DETS
Both, Mnesia and DETS, are in OTP!
That is, built-in on Erlang/Elixir!
Slide 72
Slide 72 text
BUT THERE ARE
MANY MANY OTHERS
WAYS TO BUILD AN
EVENT SOURCING
APPLICATION!
Slide 73
Slide 73 text
MY SOLUTION
SUCKS!
Slide 74
Slide 74 text
?
?
?
?
!
Slide 75
Slide 75 text
CQRS
REPORTING DATABASE
?
?
!
Slide 76
Slide 76 text
CQRS: split the responsibility of "read" and “write" on the
database.
Slide 77
Slide 77 text
CQRS: split the responsibility of "read" and “write" on the
database.
Reporting database: a database optimized for read.
If a database is optimized for read, then could has
redundancy and multiples databases to query.
Slide 78
Slide 78 text
CQRS: split the responsibility of "read" and “write" on the
database.
Reporting database: a database optimized for read.
If a database is optimized for read, then could has
redundancy and multiples databases to query.
And, applying it with event sourcing: we could has a
database that centralize all events received by the system
(“event storage”), and many small databases optimized to
read. This both reads the same bus and be updates when a
new event is forwarded on the bus.
Slide 79
Slide 79 text
CQRS and Reporting database aren’t exclusive for event
sourcing.
But, it’s useful when we are using event sourcing.
!
Slide 80
Slide 80 text
CQRS
REPORTING DATABASE
PROJECTIONS
?
!
Slide 81
Slide 81 text
A projection is the transformation of a set of data from one
form into another.
Slide 82
Slide 82 text
A projection is the transformation of a set of data from one
form into another.
In the case of event sourcing architecture, we convert a
stream of events into any structural representation.
Slide 83
Slide 83 text
A projection is the transformation of a set of data from one
form into another.
In the case of event sourcing architecture, we convert a
stream of events into any structural representation.
-1
+1
+1
+4
Slide 84
Slide 84 text
Projection isn’t exclusive for event sourcing.
But, it’s useful for event sourcing.
!
Life cycle of an event:
VALIDATION
Validations: checks that input makes sense and objects are
properly suited for further actions. (this event is to me?)
Slide 88
Slide 88 text
Life cycle of an event:
VALIDATION CONSEQUENCE
Validations: checks that input makes sense and objects are
properly suited for further actions. (this event is to me?)
Consequences: initiating some action that will change the state of
the world (I need forward a new event?)
Slide 89
Slide 89 text
Life cycle of an event:
VALIDATION CONSEQUENCE DERIVATION
Validations: checks that input makes sense and objects are
properly suited for further actions. (this event is to me?)
Consequences: initiating some action that will change the state of
the world (I need forward a new event?)
Derivations: figuring out some information based on information
we already have (I created a token and I’ll send it to user)
Slide 90
Slide 90 text
Life cycle of an event:
VALIDATION CONSEQUENCE DERIVATION
Validations: checks that input makes sense and objects are
properly suited for further actions. (this event is to me?)
Consequences: initiating some action that will change the state of
the world (I need forward a new event?)
Derivations: figuring out some information based on information
we already have (I created a token and I’ll send it to user)
Using Eager Read Derivation, this is done async between events.
It’s useful because it otimize the system, but we’ll have eventual
inconsistencies.
Slide 91
Slide 91 text
Eager Read Derivation isn’t exclusive for event sourcing.
But, it’s useful for event sourcing.
!
Slide 92
Slide 92 text
CQRS
REPORTING DATABASE
PROJECTIONS
EAGER READ DERIVATION
THE BIG FLUX!
Slide 93
Slide 93 text
User
Event storage
User can send an event on bus…
Slide 94
Slide 94 text
User
Event storage
Create an account
List accounts
Send a validate e-mail
Validate an account
User can send an event on bus, and many process manager
(PM) reads this bus…
Slide 95
Slide 95 text
User
Event storage
Create an account
List accounts
Send a validate e-mail
Validate an account
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus.
Slide 96
Slide 96 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
Slide 97
Slide 97 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 98
Slide 98 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 99
Slide 99 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 100
Slide 100 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 101
Slide 101 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 102
Slide 102 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 103
Slide 103 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 104
Slide 104 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 105
Slide 105 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 106
Slide 106 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 107
Slide 107 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 108
Slide 108 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 109
Slide 109 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 110
Slide 110 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 111
Slide 111 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 112
Slide 112 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 113
Slide 113 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!