Event Sourcing architecture using Elixir

Talk at "Nerdzão #37 - Back-End".

Video: https://youtu.be/tixWGkkZbXE
Projects used on this talk: https://github.com/macabeus/event-sourcing-example
My blog: http://macalogs.com.br/

Transcript

1. EVENT SOURCING ARCHITECTURE USING ELIXIR Bruno Macabeus
2. HELLO! ͜Μʹͪ͸ʂ
3. Bruno Macabeus github.com/macabeus macalogs.com.br I’m a developer at Pagar.me ❤
4. WHAT IS EVENT SOURCING?
5. “The core idea of event sourcing is that whenever we make a change to the state of a system, we record that state change as an event, and we can confidently rebuild the system state by reprocessing the events at any time in the future.” - Martin Fowler youtu.be/STKCRSUsyP0 bit.ly/martin-event-driven
6. This architecture has many interesting benefits!
7. This architecture has many interesting benefits! Event log provides a strong audit capability
8. This architecture has many interesting benefits! Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point
9. This architecture has many interesting benefits! We can explore alternative histories by injecting hypothetical events when replaying Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point
10. This architecture has many interesting benefits! Event sourcing makes it plausible to have non-durable working copies, such as a Memory Image. We can explore alternative histories by injecting hypothetical events when replaying Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point
11. This architecture has many interesting benefits! Event sourcing makes it plausible to have non-durable working copies, such as a Memory Image. We can explore alternative histories by injecting hypothetical events when replaying Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point
12. Did you know? Database journal file and Git are examples of using event sourcing!
13. Did you know? Database journal file and Git are examples of using event sourcing!
14. THE (VERY) SIMPLE FLUX
15. User Server Current state: Events log: 0
16. User Server Current state: Events log: 0 User creates an event
17. User Server Current state: Events log: 0 In this example, the user “creates an event”. But, on the others implementations of this architecture, another part could create the events
18. User Server Current state: Events log: 0
19. User Server Current state: Events log: 0 Event “arrives" on the server. On the others implementations, the server could “create" the event
20. User Server Current state: Events log: +1 1
21. User Server Current state: Events log: +1 1
22. User Server Current state: Events log: +1 1
23. User Server Current state: Events log: +1 2 +1
24. User Server Current state: Events log: +1 2 +1
25. User Server Current state: Events log: +1 2 +1
26. User Server Current state: Events log: -1 1 +1 +1
27. User Server Current state: Events log: -1 1 +1 +1 It’s a non-deterministic event
28. User Server Current state: Events log: -1 1 +1 +1
29. User Server Current state: Events log: -1 1 +1 +1
30. User Server Current state: Events log: -1 5 +1 +1 +4
31. User Server Current state: Events log: -1 5 +1 +1 +4 Now the events log has too many events
32. User Server Current state: Events log: -1 5 +1 +1 +4 Snapshot: state 5
33. User Server Current state: Events log: -1 5 +1 +1 +4 Snapshot: state 5 This is an aggregate. That is, a stream of events
34. User Server Current state: Events log: -1 5 +1 +1 +4 Snapshot: state 5
35. User Server Current state: Events log: -1 5 +1 +1 +4 Snapshot: state 5
36. User Server Current state: Events log: -1 6 +1 +1 +4 +1 Snapshot: state 5
37. User Server Current state: Events log: -1 ?? +1 +1 +4 +1 Snapshot: state 5 What if we lose the state?
38. User Server Current state: Events log: -1 ?? +1 +1 +4 +1 Snapshot: state 5 We could run again the events to restore the previous state! Using the events, we could convert the stream of events into a structural representation. This process is called “projection".
39. GIVE ME AN EXAMPLE!
40. Let’s see an Elixir project that uses event sourcing architecture! github.com/macabeus/event-sourcing-example
41. GIVE ME AN EXAMPLE! HOW EVENT SOURCING WAS APPLIED IN THIS PROJECT?
46. GIVE ME AN EXAMPLE! API
50. GIVE ME AN EXAMPLE! EVENTS
53. Hey! It’s important the events be agnostic of a language, then you’ll can work using another languages on same project. Flux Standard Action is an example of solution about it. I didn't do it in my example. My example works only on Elixir
54. GIVE ME AN EXAMPLE! PROCESS AND BUS PIPELINE
67. GIVE ME AN EXAMPLE! DATABASE
68. In this project, I used two databases:
69. - An in-memory database, Mnesia, using the wrapper Amnesia In this project, I used two databases:
70. - An in-memory database, Mnesia, using the wrapper Amnesia In this project, I used two databases: - A storage file database, DETS
71. - An in-memory database, Mnesia, using the wrapper Amnesia In this project, I used two databases: - A storage file database, DETS Both, Mnesia and DETS, are in OTP! That is, built-in on Erlang/Elixir!
72. BUT THERE ARE MANY MANY OTHERS WAYS TO BUILD AN EVENT SOURCING APPLICATION!
73. MY SOLUTION SUCKS!
74. ? ? ? ? !
75. CQRS REPORTING DATABASE ? ? !
76. CQRS: split the responsibility of "read" and “write" on the database.
77. CQRS: split the responsibility of "read" and “write" on the database. Reporting database: a database optimized for read. If a database is optimized for read, then could has redundancy and multiples databases to query.
78. CQRS: split the responsibility of "read" and “write" on the database. Reporting database: a database optimized for read. If a database is optimized for read, then could has redundancy and multiples databases to query. And, applying it with event sourcing: we could has a database that centralize all events received by the system (“event storage”), and many small databases optimized to read. This both reads the same bus and be updates when a new event is forwarded on the bus.
79. CQRS and Reporting database aren’t exclusive for event sourcing. But, it’s useful when we are using event sourcing. !
80. CQRS REPORTING DATABASE PROJECTIONS ? !
81. A projection is the transformation of a set of data from one form into another.
82. A projection is the transformation of a set of data from one form into another. In the case of event sourcing architecture, we convert a stream of events into any structural representation.
83. A projection is the transformation of a set of data from one form into another. In the case of event sourcing architecture, we convert a stream of events into any structural representation. -1 +1 +1 +4
84. Projection isn’t exclusive for event sourcing. But, it’s useful for event sourcing. !
85. CQRS REPORTING DATABASE PROJECTIONS EAGER READ DERIVATION !
86. Life cycle of an event:
87. Life cycle of an event: VALIDATION Validations: checks that input makes sense and objects are properly suited for further actions. (this event is to me?)
88. Life cycle of an event: VALIDATION CONSEQUENCE Validations: checks that input makes sense and objects are properly suited for further actions. (this event is to me?) Consequences: initiating some action that will change the state of the world (I need forward a new event?)
89. Life cycle of an event: VALIDATION CONSEQUENCE DERIVATION Validations: checks that input makes sense and objects are properly suited for further actions. (this event is to me?) Consequences: initiating some action that will change the state of the world (I need forward a new event?) Derivations: figuring out some information based on information we already have (I created a token and I’ll send it to user)
90. Life cycle of an event: VALIDATION CONSEQUENCE DERIVATION Validations: checks that input makes sense and objects are properly suited for further actions. (this event is to me?) Consequences: initiating some action that will change the state of the world (I need forward a new event?) Derivations: figuring out some information based on information we already have (I created a token and I’ll send it to user) Using Eager Read Derivation, this is done async between events. It’s useful because it otimize the system, but we’ll have eventual inconsistencies.
91. Eager Read Derivation isn’t exclusive for event sourcing. But, it’s useful for event sourcing. !
92. CQRS REPORTING DATABASE PROJECTIONS EAGER READ DERIVATION THE BIG FLUX!
93. User Event storage User can send an event on bus…
94. User Event storage Create an account List accounts Send a validate e-mail Validate an account User can send an event on bus, and many process manager (PM) reads this bus…
95. User Event storage Create an account List accounts Send a validate e-mail Validate an account User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus.
96. User Event storage Create an account Send a validate e-mail List accounts Validate an account User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data.
97. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
98. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
99. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
100. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
101. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
102. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
103. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
104. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
105. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
106. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
107. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
108. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
109. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
110. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
111. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
112. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
113. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
114. YOU DON’T NEED DO IT YOURSELF
115. COMMANDED FRAMEWORK
116. Commanded framework! http://bit.ly/elixir-brasil-bernado
117. Talk at Elixir Brasil by Bernado Amorim! % http://bit.ly/elixir-brasil-bernado
118. FURTHERMORE…
119. An idle game using event sourcing! https://github.com/telemmo
120. Visit my blog! ❤ http://macalogs.com.br/