Slide 1

Slide 1 text

EVENT SOURCING ARCHITECTURE USING ELIXIR Bruno Macabeus

Slide 2

Slide 2 text

HELLO! ͜Μʹͪ͸ʂ

Slide 3

Slide 3 text

Bruno Macabeus github.com/macabeus macalogs.com.br I’m a developer at Pagar.me ❤

Slide 4

Slide 4 text

WHAT IS EVENT SOURCING?

Slide 5

Slide 5 text

“The core idea of event sourcing is that whenever we make a change to the state of a system, we record that state change as an event, and we can confidently rebuild the system state by reprocessing the events at any time in the future.” - Martin Fowler youtu.be/STKCRSUsyP0 bit.ly/martin-event-driven

Slide 6

Slide 6 text

This architecture has many interesting benefits!

Slide 7

Slide 7 text

This architecture has many interesting benefits! Event log provides a strong audit capability

Slide 8

Slide 8 text

This architecture has many interesting benefits! Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point

Slide 9

Slide 9 text

This architecture has many interesting benefits! We can explore alternative histories by injecting hypothetical events when replaying Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point

Slide 10

Slide 10 text

This architecture has many interesting benefits! Event sourcing makes it plausible to have non-durable working copies, such as a Memory Image. We can explore alternative histories by injecting hypothetical events when replaying Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point

Slide 11

Slide 11 text

This architecture has many interesting benefits! Event sourcing makes it plausible to have non-durable working copies, such as a Memory Image. We can explore alternative histories by injecting hypothetical events when replaying Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point

Slide 12

Slide 12 text

This architecture has many interesting benefits! Event sourcing makes it plausible to have non-durable working copies, such as a Memory Image. We can explore alternative histories by injecting hypothetical events when replaying Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point

Slide 13

Slide 13 text

Did you know? Database journal file and Git are examples of using event sourcing!

Slide 14

Slide 14 text

Did you know? Database journal file and Git are examples of using event sourcing!

Slide 15

Slide 15 text

THE (VERY) SIMPLE FLUX

Slide 16

Slide 16 text

A SIMPLE COUNTER

Slide 17

Slide 17 text

User Server Current state: Events log: 0

Slide 18

Slide 18 text

User Server Current state: Events log: 0 in-memory persistent

Slide 19

Slide 19 text

User Server Current state: Events log: 0 User “creates" an event. An event represents something that happened, a fact

Slide 20

Slide 20 text

User Server Current state: Events log: 0 In this example, the user “creates an event”. But, on the others implementations of this architecture, another part could create the events

Slide 21

Slide 21 text

User Server Current state: Events log: 0

Slide 22

Slide 22 text

User Server Current state: Events log: 0 Event “arrives" on the server. On the others implementations, the server could “create" the event

Slide 23

Slide 23 text

User Server Current state: Events log: +1 1

Slide 24

Slide 24 text

User Server Current state: Events log: +1 1

Slide 25

Slide 25 text

User Server Current state: Events log: +1 1

Slide 26

Slide 26 text

User Server Current state: Events log: +1 2 +1

Slide 27

Slide 27 text

User Server Current state: Events log: +1 2 +1 Note that the events log aren’t the represent the final state

Slide 28

Slide 28 text

User Server Current state: Events log: +1 2 +1

Slide 29

Slide 29 text

User Server Current state: Events log: +1 2 +1

Slide 30

Slide 30 text

User Server Current state: Events log: -1 1 +1 +1

Slide 31

Slide 31 text

User Server Current state: Events log: -1 1 +1 +1 It’s a non-deterministic event

Slide 32

Slide 32 text

User Server Current state: Events log: -1 1 +1 +1

Slide 33

Slide 33 text

User Server Current state: Events log: -1 1 +1 +1

Slide 34

Slide 34 text

User Server Current state: Events log: -1 5 +1 +1 +4

Slide 35

Slide 35 text

User Server Current state: Events log: -1 5 +1 +1 +4 Now the events log has too many events

Slide 36

Slide 36 text

User Server Current state: Events log: -1 5 +1 +1 +4 Snapshot: state 5

Slide 37

Slide 37 text

User Server Current state: Events log: -1 5 +1 +1 +4 Snapshot: state 5

Slide 38

Slide 38 text

User Server Current state: Events log: -1 5 +1 +1 +4 Snapshot: state 5

Slide 39

Slide 39 text

User Server Current state: Events log: -1 6 +1 +1 +4 +1 Snapshot: state 5

Slide 40

Slide 40 text

User Server Current state: Events log: -1 ?? +1 +1 +4 +1 Snapshot: state 5 What if we lose the state?

Slide 41

Slide 41 text

User Server Current state: Events log: -1 ?? +1 +1 +4 +1 Snapshot: state 5 We could run again the events to restore the previous state! Using the events, we could convert the stream of events into a structural representation. This process is called “projection".

Slide 42

Slide 42 text

GIVE ME A CODE EXAMPLE!

Slide 43

Slide 43 text

A SIMPLE BANKING SYSTEM

Slide 44

Slide 44 text

Let’s see an Elixir project that uses event sourcing architecture! github.com/macabeus/event-sourcing-example

Slide 45

Slide 45 text

GIVE ME AN EXAMPLE! HOW EVENT SOURCING WAS APPLIED IN THIS PROJECT?

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

GIVE ME AN EXAMPLE! API

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

GIVE ME AN EXAMPLE! EVENTS

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

Hey! It’s important the events be agnostic of a language, then you’ll can work using another languages on same project. Flux Standard Action is an example of solution about it. I didn't do it in my example. My example works only on Elixir

Slide 58

Slide 58 text

GIVE ME AN EXAMPLE! PROCESS AND BUS PIPELINE

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

No content

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

No content

Slide 65

Slide 65 text

No content

Slide 66

Slide 66 text

No content

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

No content

Slide 69

Slide 69 text

No content

Slide 70

Slide 70 text

No content

Slide 71

Slide 71 text

GIVE ME AN EXAMPLE! DATABASE

Slide 72

Slide 72 text

In this project, I used two databases:

Slide 73

Slide 73 text

- An in-memory database, Mnesia, using the wrapper Amnesia In this project, I used two databases: Mnesia

Slide 74

Slide 74 text

- An in-memory database, Mnesia, using the wrapper Amnesia In this project, I used two databases: - A storage file database, DETS Mnesia DETS

Slide 75

Slide 75 text

- An in-memory database, Mnesia, using the wrapper Amnesia In this project, I used two databases: - A storage file database, DETS Both, Mnesia and DETS, are in OTP! That is, built-in on Erlang/Elixir! Mnesia DETS OTP

Slide 76

Slide 76 text

BUT THERE ARE MANY MANY OTHERS WAYS TO BUILD AN EVENT SOURCING APPLICATION!

Slide 77

Slide 77 text

MY SOLUTION SUCKS!

Slide 78

Slide 78 text

? ? ? ? !

Slide 79

Slide 79 text

? ! PROJECTIONS ? ?

Slide 80

Slide 80 text

A projection is the transformation of a set of data from one form into another.

Slide 81

Slide 81 text

A projection is the transformation of a set of data from one form into another. In the case of event sourcing architecture, we convert a stream of events into any structural representation.

Slide 82

Slide 82 text

A projection is the transformation of a set of data from one form into another. In the case of event sourcing architecture, we convert a stream of events into any structural representation. -1 +1 +1 +4

Slide 83

Slide 83 text

Projection isn’t exclusive for event sourcing. But, it’s useful for event sourcing. !

Slide 84

Slide 84 text

CQRS REPORTING DATABASE PROJECTIONS ? !

Slide 85

Slide 85 text

CQRS: split the responsibility of "read" and “write" on the database.

Slide 86

Slide 86 text

CQRS: split the responsibility of "read" and “write" on the database. Reporting database: a database optimized for read. If a database is optimized for read, then could have redundancy and multiples databases to query.

Slide 87

Slide 87 text

CQRS: split the responsibility of "read" and “write" on the database. Reporting database: a database optimized for read. If a database is optimized for read, then could have redundancy and multiples databases to query. And, applying it with event sourcing: we could have a database that centralize all events received by the system (“event storage”), and many small databases optimized to read. This both reads the same bus and be updates when a new event is forwarded on the bus.

Slide 88

Slide 88 text

CQRS and Reporting Database aren’t exclusive for event sourcing. But, it’s useful when we are using event sourcing. !

Slide 89

Slide 89 text

COMMANDS ! CQRS REPORTING DATABASE PROJECTIONS

Slide 90

Slide 90 text

Commands are de requests issued by the user, and maybe fail. E.g: AddFunds

Slide 91

Slide 91 text

Commands are de requests issued by the user, and maybe fail. E.g: AddFunds When a command are performed, maybe it generates events - that is, facts about the application state changes E.g: FundsAdded

Slide 92

Slide 92 text

Life cycle:

Slide 93

Slide 93 text

VALIDATION Validations: checks if the command makes sense and objects are properly suited for further actions (is this event is to me?) Life cycle:

Slide 94

Slide 94 text

VALIDATION CONSEQUENCE Validations: checks if the command makes sense and objects are properly suited for further actions (is this event is to me?) Consequences: initiating some action that will change the state of the world (do I need to forward a new event?) Life cycle:

Slide 95

Slide 95 text

VALIDATION CONSEQUENCE DERIVATION Validations: checks if the command makes sense and objects are properly suited for further actions (is this event is to me?) Consequences: initiating some action that will change the state of the world (do I need to forward a new event?) Derivations: figuring out some information based on information we already have (I created a token and I’ll send it to user) Life cycle:

Slide 96

Slide 96 text

CQRS REPORTING DATABASE PROJECTIONS COMMANDS THE BIG FLUX!

Slide 97

Slide 97 text

User Event storage User can send a command on bus…

Slide 98

Slide 98 text

User Event storage Create an account List accounts Send a validate e-mail Validate an account User can send a command on bus, and many process manager (PM) reads this bus…

Slide 99

Slide 99 text

User Event storage Create an account List accounts Send a validate e-mail Validate an account User can send a command on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus.

Slide 100

Slide 100 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account User can send a command on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data.

Slide 101

Slide 101 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send a command on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 102

Slide 102 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send a command on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 103

Slide 103 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 104

Slide 104 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 105

Slide 105 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 106

Slide 106 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 107

Slide 107 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 108

Slide 108 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 109

Slide 109 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 110

Slide 110 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 111

Slide 111 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 112

Slide 112 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 113

Slide 113 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 114

Slide 114 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 115

Slide 115 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 116

Slide 116 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 117

Slide 117 text

User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!

Slide 118

Slide 118 text

YOU DON’T NEED TO DO IT YOURSELF

Slide 119

Slide 119 text

COMMANDED FRAMEWORK

Slide 120

Slide 120 text

Commanded framework! http://github.com/commanded

Slide 121

Slide 121 text

Commanded has more concepts that I didn’t say, (e.g: aggregate) and a great wiki.

Slide 122

Slide 122 text

Talk at Elixir Brasil by Bernardo Amorim! % http://bit.ly/elixir-brasil-bernado

Slide 123

Slide 123 text

Talk at Code Beam by Bernardo Amorim! % http://bit.ly/code-beam-bernardo-amorim

Slide 124

Slide 124 text

FURTHERMORE…

Slide 125

Slide 125 text

An idle game using event sourcing! https://github.com/telemmo

Slide 126

Slide 126 text

Visit my blog! ❤ http://macalogs.com.br/