EVENT SOURCING
ARCHITECTURE
USING
ELIXIR
Bruno Macabeus
Slide 2
Slide 2 text
HELLO!
͜Μʹͪʂ
Slide 3
Slide 3 text
Bruno Macabeus
github.com/macabeus
macalogs.com.br
I’m a developer
at Pagar.me ❤
Slide 4
Slide 4 text
WHAT IS EVENT
SOURCING?
Slide 5
Slide 5 text
“The core idea of event
sourcing is that whenever
we make a change to the
state of a system, we record
that state change as an
event, and we can
confidently rebuild the
system state by
reprocessing the events at
any time in the future.”
- Martin Fowler
youtu.be/STKCRSUsyP0
bit.ly/martin-event-driven
Slide 6
Slide 6 text
This architecture has many interesting benefits!
Slide 7
Slide 7 text
This architecture has many interesting benefits!
Event log provides a strong audit capability
Slide 8
Slide 8 text
This architecture has many interesting benefits!
Event log provides a strong audit capability
We can recreate historic states by replaying the event
log up to a point
Slide 9
Slide 9 text
This architecture has many interesting benefits!
We can explore alternative histories by injecting
hypothetical events when replaying
Event log provides a strong audit capability
We can recreate historic states by replaying the event
log up to a point
Slide 10
Slide 10 text
This architecture has many interesting benefits!
Event sourcing makes it plausible to have non-durable
working copies, such as a Memory Image.
We can explore alternative histories by injecting
hypothetical events when replaying
Event log provides a strong audit capability
We can recreate historic states by replaying the event
log up to a point
Slide 11
Slide 11 text
This architecture has many interesting benefits!
Event sourcing makes it plausible to have non-durable
working copies, such as a Memory Image.
We can explore alternative histories by injecting
hypothetical events when replaying
Event log provides a strong audit capability
We can recreate historic states by replaying the event
log up to a point
Slide 12
Slide 12 text
This architecture has many interesting benefits!
Event sourcing makes it plausible to have non-durable
working copies, such as a Memory Image.
We can explore alternative histories by injecting
hypothetical events when replaying
Event log provides a strong audit capability
We can recreate historic states by replaying the event
log up to a point
Slide 13
Slide 13 text
Did you know?
Database journal file and Git are examples of using event
sourcing!
Slide 14
Slide 14 text
Did you know?
Database journal file and Git are examples of using event
sourcing!
Slide 15
Slide 15 text
THE (VERY) SIMPLE
FLUX
Slide 16
Slide 16 text
A SIMPLE COUNTER
Slide 17
Slide 17 text
User Server
Current state:
Events log:
0
Slide 18
Slide 18 text
User Server
Current state:
Events log:
0
in-memory
persistent
Slide 19
Slide 19 text
User Server
Current state:
Events log:
0
User “creates" an
event.
An event represents
something that
happened, a fact
Slide 20
Slide 20 text
User Server
Current state:
Events log:
0
In this example, the
user “creates an
event”. But,
on the others
implementations
of this architecture,
another part could
create the events
Slide 21
Slide 21 text
User Server
Current state:
Events log:
0
Slide 22
Slide 22 text
User Server
Current state:
Events log:
0
Event “arrives" on
the server.
On the others
implementations,
the server could “create"
the event
Slide 23
Slide 23 text
User Server
Current state:
Events log:
+1
1
Slide 24
Slide 24 text
User Server
Current state:
Events log:
+1
1
Slide 25
Slide 25 text
User Server
Current state:
Events log:
+1
1
Slide 26
Slide 26 text
User Server
Current state:
Events log:
+1
2
+1
Slide 27
Slide 27 text
User Server
Current state:
Events log:
+1
2
+1
Note that the events log
aren’t the represent the
final state
Slide 28
Slide 28 text
User Server
Current state:
Events log:
+1
2
+1
Slide 29
Slide 29 text
User Server
Current state:
Events log:
+1
2
+1
Slide 30
Slide 30 text
User Server
Current state:
Events log:
-1
1
+1
+1
Slide 31
Slide 31 text
User Server
Current state:
Events log:
-1
1
+1
+1
It’s a non-deterministic
event
Slide 32
Slide 32 text
User Server
Current state:
Events log:
-1
1
+1
+1
Slide 33
Slide 33 text
User Server
Current state:
Events log:
-1
1
+1
+1
Slide 34
Slide 34 text
User Server
Current state:
Events log:
-1
5
+1
+1
+4
Slide 35
Slide 35 text
User Server
Current state:
Events log:
-1
5
+1
+1
+4
Now the events
log has too many
events
Slide 36
Slide 36 text
User Server
Current state:
Events log:
-1
5
+1
+1
+4
Snapshot: state 5
Slide 37
Slide 37 text
User Server
Current state:
Events log:
-1
5
+1
+1
+4
Snapshot: state 5
Slide 38
Slide 38 text
User Server
Current state:
Events log:
-1
5
+1
+1
+4
Snapshot: state 5
Slide 39
Slide 39 text
User Server
Current state:
Events log:
-1
6
+1
+1
+4
+1
Snapshot: state 5
Slide 40
Slide 40 text
User Server
Current state:
Events log:
-1
??
+1
+1
+4
+1
Snapshot: state 5
What if we lose
the state?
Slide 41
Slide 41 text
User Server
Current state:
Events log:
-1
??
+1
+1
+4
+1
Snapshot: state 5
We could run again the
events to restore the
previous state!
Using the events, we
could convert the
stream of events into a
structural
representation. This
process is called
“projection".
Slide 42
Slide 42 text
GIVE ME A CODE
EXAMPLE!
Slide 43
Slide 43 text
A SIMPLE BANKING
SYSTEM
Slide 44
Slide 44 text
Let’s see an Elixir project that uses event sourcing architecture!
github.com/macabeus/event-sourcing-example
Slide 45
Slide 45 text
GIVE ME AN EXAMPLE!
HOW EVENT SOURCING WAS
APPLIED IN THIS PROJECT?
Slide 46
Slide 46 text
No content
Slide 47
Slide 47 text
No content
Slide 48
Slide 48 text
No content
Slide 49
Slide 49 text
No content
Slide 50
Slide 50 text
GIVE ME AN EXAMPLE!
API
Slide 51
Slide 51 text
No content
Slide 52
Slide 52 text
No content
Slide 53
Slide 53 text
No content
Slide 54
Slide 54 text
GIVE ME AN EXAMPLE!
EVENTS
Slide 55
Slide 55 text
No content
Slide 56
Slide 56 text
No content
Slide 57
Slide 57 text
Hey! It’s important the events be agnostic of a language,
then you’ll can work using another languages on same
project.
Flux Standard Action is an example of solution about it.
I didn't do it in my example.
My example works only on Elixir
Slide 58
Slide 58 text
GIVE ME AN EXAMPLE!
PROCESS AND BUS PIPELINE
Slide 59
Slide 59 text
No content
Slide 60
Slide 60 text
No content
Slide 61
Slide 61 text
No content
Slide 62
Slide 62 text
No content
Slide 63
Slide 63 text
No content
Slide 64
Slide 64 text
No content
Slide 65
Slide 65 text
No content
Slide 66
Slide 66 text
No content
Slide 67
Slide 67 text
No content
Slide 68
Slide 68 text
No content
Slide 69
Slide 69 text
No content
Slide 70
Slide 70 text
No content
Slide 71
Slide 71 text
GIVE ME AN EXAMPLE!
DATABASE
Slide 72
Slide 72 text
In this project, I used two databases:
Slide 73
Slide 73 text
- An in-memory database, Mnesia, using the wrapper
Amnesia
In this project, I used two databases:
Mnesia
Slide 74
Slide 74 text
- An in-memory database, Mnesia, using the wrapper
Amnesia
In this project, I used two databases:
- A storage file database, DETS
Mnesia DETS
Slide 75
Slide 75 text
- An in-memory database, Mnesia, using the wrapper
Amnesia
In this project, I used two databases:
- A storage file database, DETS
Both, Mnesia and DETS, are in OTP!
That is, built-in on Erlang/Elixir!
Mnesia DETS
OTP
Slide 76
Slide 76 text
BUT THERE ARE
MANY MANY OTHERS
WAYS TO BUILD AN
EVENT SOURCING
APPLICATION!
Slide 77
Slide 77 text
MY SOLUTION
SUCKS!
Slide 78
Slide 78 text
?
?
?
?
!
Slide 79
Slide 79 text
?
!
PROJECTIONS
?
?
Slide 80
Slide 80 text
A projection is the transformation of a set of data from one
form into another.
Slide 81
Slide 81 text
A projection is the transformation of a set of data from one
form into another.
In the case of event sourcing architecture, we convert a
stream of events into any structural representation.
Slide 82
Slide 82 text
A projection is the transformation of a set of data from one
form into another.
In the case of event sourcing architecture, we convert a
stream of events into any structural representation.
-1
+1
+1
+4
Slide 83
Slide 83 text
Projection isn’t exclusive for event sourcing.
But, it’s useful for event sourcing.
!
Slide 84
Slide 84 text
CQRS
REPORTING DATABASE
PROJECTIONS
?
!
Slide 85
Slide 85 text
CQRS: split the responsibility of "read" and “write" on the
database.
Slide 86
Slide 86 text
CQRS: split the responsibility of "read" and “write" on the
database.
Reporting database: a database optimized for read.
If a database is optimized for read, then could have
redundancy and multiples databases to query.
Slide 87
Slide 87 text
CQRS: split the responsibility of "read" and “write" on the
database.
Reporting database: a database optimized for read.
If a database is optimized for read, then could have
redundancy and multiples databases to query.
And, applying it with event sourcing: we could have a
database that centralize all events received by the system
(“event storage”), and many small databases optimized to
read. This both reads the same bus and be updates when a
new event is forwarded on the bus.
Slide 88
Slide 88 text
CQRS and Reporting Database aren’t exclusive for event
sourcing.
But, it’s useful when we are using event sourcing.
!
Slide 89
Slide 89 text
COMMANDS
!
CQRS
REPORTING DATABASE
PROJECTIONS
Slide 90
Slide 90 text
Commands are de requests issued by the user, and maybe
fail.
E.g: AddFunds
Slide 91
Slide 91 text
Commands are de requests issued by the user, and maybe
fail.
E.g: AddFunds
When a command are performed, maybe it generates
events - that is, facts about the application state changes
E.g: FundsAdded
Slide 92
Slide 92 text
Life cycle:
Slide 93
Slide 93 text
VALIDATION
Validations: checks if the command makes sense and objects are
properly suited for further actions (is this event is to me?)
Life cycle:
Slide 94
Slide 94 text
VALIDATION CONSEQUENCE
Validations: checks if the command makes sense and objects are
properly suited for further actions (is this event is to me?)
Consequences: initiating some action that will change the state of
the world (do I need to forward a new event?)
Life cycle:
Slide 95
Slide 95 text
VALIDATION CONSEQUENCE DERIVATION
Validations: checks if the command makes sense and objects are
properly suited for further actions (is this event is to me?)
Consequences: initiating some action that will change the state of
the world (do I need to forward a new event?)
Derivations: figuring out some information based on information
we already have (I created a token and I’ll send it to user)
Life cycle:
Slide 96
Slide 96 text
CQRS
REPORTING DATABASE
PROJECTIONS
COMMANDS
THE BIG FLUX!
Slide 97
Slide 97 text
User
Event storage
User can send a command on bus…
Slide 98
Slide 98 text
User
Event storage
Create an account
List accounts
Send a validate e-mail
Validate an account
User can send a command on bus, and many process
manager (PM) reads this bus…
Slide 99
Slide 99 text
User
Event storage
Create an account
List accounts
Send a validate e-mail
Validate an account
User can send a command on bus, and many process
manager (PM) reads this bus. Maybe a PM could send a
new event on the bus.
Slide 100
Slide 100 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
User can send a command on bus, and many process
manager (PM) reads this bus. Maybe a PM could send a
new event on the bus. And, the PM needs read events in
Event Storage, to re-build when it loses the data.
Slide 101
Slide 101 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send a command on bus, and many process
manager (PM) reads this bus. Maybe a PM could send a
new event on the bus. And, the PM needs read events in
Event Storage, to re-build when it loses the data.
And everything could be async!
Slide 102
Slide 102 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send a command on bus, and many process
manager (PM) reads this bus. Maybe a PM could send a
new event on the bus. And, the PM needs read events in
Event Storage, to re-build when it loses the data.
And everything could be async!
Slide 103
Slide 103 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 104
Slide 104 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 105
Slide 105 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 106
Slide 106 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 107
Slide 107 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 108
Slide 108 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 109
Slide 109 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 110
Slide 110 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 111
Slide 111 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 112
Slide 112 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 113
Slide 113 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 114
Slide 114 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 115
Slide 115 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 116
Slide 116 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 117
Slide 117 text
User
Event storage
Create an account
Send a validate e-mail List accounts
Validate an account
Events processing are asynchronous!
User can send an event on bus, and many process manager
(PM) reads this bus. Maybe a PM could send a new event
on the bus. And, the PM needs read events in Event
Storage, to re-build when it loses the data.
And everything could be async!
Slide 118
Slide 118 text
YOU DON’T NEED
TO DO IT YOURSELF
Slide 119
Slide 119 text
COMMANDED
FRAMEWORK
Slide 120
Slide 120 text
Commanded framework!
http://github.com/commanded
Slide 121
Slide 121 text
Commanded has more concepts that I didn’t say, (e.g:
aggregate) and a great wiki.
Slide 122
Slide 122 text
Talk at Elixir Brasil by Bernardo Amorim! %
http://bit.ly/elixir-brasil-bernado
Slide 123
Slide 123 text
Talk at Code Beam by Bernardo Amorim! %
http://bit.ly/code-beam-bernardo-amorim
Slide 124
Slide 124 text
FURTHERMORE…
Slide 125
Slide 125 text
An idle game using event sourcing!
https://github.com/telemmo