Event Sourcing architecture using Elixir (old version)

NEW SLIDE VERSION: https://speakerdeck.com/macabeus/event-sourcing-architecture-using-elixir

Talk at "Elug Meetups #13".

Video (in other meetup, using old slides version): https://youtu.be/tixWGkkZbXE
Projects used on this talk: https://github.com/macabeus/event-sourcing-example
My blog: http://macalogs.com.br/

Transcript

1. EVENT SOURCING ARCHITECTURE USING ELIXIR Bruno Macabeus
2. HELLO! ͜Μʹͪ͸ʂ
3. Bruno Macabeus github.com/macabeus macalogs.com.br I’m a developer at Pagar.me ❤
4. WHAT IS EVENT SOURCING?
5. “The core idea of event sourcing is that whenever we make a change to the state of a system, we record that state change as an event, and we can confidently rebuild the system state by reprocessing the events at any time in the future.” - Martin Fowler youtu.be/STKCRSUsyP0 bit.ly/martin-event-driven
6. This architecture has many interesting benefits!
7. This architecture has many interesting benefits! Event log provides a strong audit capability
8. This architecture has many interesting benefits! Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point
9. This architecture has many interesting benefits! We can explore alternative histories by injecting hypothetical events when replaying Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point
10. This architecture has many interesting benefits! Event sourcing makes it plausible to have non-durable working copies, such as a Memory Image. We can explore alternative histories by injecting hypothetical events when replaying Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point
11. This architecture has many interesting benefits! Event sourcing makes it plausible to have non-durable working copies, such as a Memory Image. We can explore alternative histories by injecting hypothetical events when replaying Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point
12. This architecture has many interesting benefits! Event sourcing makes it plausible to have non-durable working copies, such as a Memory Image. We can explore alternative histories by injecting hypothetical events when replaying Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point
13. Did you know? Database journal file and Git are examples of using event sourcing!
14. Did you know? Database journal file and Git are examples of using event sourcing!
15. THE (VERY) SIMPLE FLUX
16. A SIMPLE COUNTER
17. User Server Current state: Events log: 0
18. User Server Current state: Events log: 0 in-memory persistent
19. User Server Current state: Events log: 0 User “creates" an event. An event represents something that happened, a fact
20. User Server Current state: Events log: 0 In this example, the user “creates an event”. But, on the others implementations of this architecture, another part could create the events
21. User Server Current state: Events log: 0
22. User Server Current state: Events log: 0 Event “arrives" on the server. On the others implementations, the server could “create" the event
23. User Server Current state: Events log: +1 1
24. User Server Current state: Events log: +1 1
25. User Server Current state: Events log: +1 1
26. User Server Current state: Events log: +1 2 +1
27. User Server Current state: Events log: +1 2 +1 Note that the events log aren’t the represent the final state
28. User Server Current state: Events log: +1 2 +1
29. User Server Current state: Events log: +1 2 +1
30. User Server Current state: Events log: -1 1 +1 +1
31. User Server Current state: Events log: -1 1 +1 +1 It’s a non-deterministic event
32. User Server Current state: Events log: -1 1 +1 +1
33. User Server Current state: Events log: -1 1 +1 +1
34. User Server Current state: Events log: -1 5 +1 +1 +4
35. User Server Current state: Events log: -1 5 +1 +1 +4 Now the events log has too many events
36. User Server Current state: Events log: -1 5 +1 +1 +4 Snapshot: state 5
37. User Server Current state: Events log: -1 5 +1 +1 +4 Snapshot: state 5
38. User Server Current state: Events log: -1 5 +1 +1 +4 Snapshot: state 5
39. User Server Current state: Events log: -1 6 +1 +1 +4 +1 Snapshot: state 5
40. User Server Current state: Events log: -1 ?? +1 +1 +4 +1 Snapshot: state 5 What if we lose the state?
41. User Server Current state: Events log: -1 ?? +1 +1 +4 +1 Snapshot: state 5 We could run again the events to restore the previous state! Using the events, we could convert the stream of events into a structural representation. This process is called “projection".
42. GIVE ME A CODE EXAMPLE!
43. A SIMPLE BANKING SYSTEM
44. Let’s see an Elixir project that uses event sourcing architecture! github.com/macabeus/event-sourcing-example
45. GIVE ME AN EXAMPLE! HOW EVENT SOURCING WAS APPLIED IN THIS PROJECT?
50. GIVE ME AN EXAMPLE! API
54. GIVE ME AN EXAMPLE! EVENTS
57. Hey! It’s important the events be agnostic of a language, then you’ll can work using another languages on same project. Flux Standard Action is an example of solution about it. I didn't do it in my example. My example works only on Elixir
58. GIVE ME AN EXAMPLE! PROCESS AND BUS PIPELINE
71. GIVE ME AN EXAMPLE! DATABASE
72. In this project, I used two databases:
73. - An in-memory database, Mnesia, using the wrapper Amnesia In this project, I used two databases: Mnesia
74. - An in-memory database, Mnesia, using the wrapper Amnesia In this project, I used two databases: - A storage file database, DETS Mnesia DETS
75. - An in-memory database, Mnesia, using the wrapper Amnesia In this project, I used two databases: - A storage file database, DETS Both, Mnesia and DETS, are in OTP! That is, built-in on Erlang/Elixir! Mnesia DETS OTP
76. BUT THERE ARE MANY MANY OTHERS WAYS TO BUILD AN EVENT SOURCING APPLICATION!
77. MY SOLUTION SUCKS!
78. ? ? ? ? !
79. ? ! PROJECTIONS ? ?
80. A projection is the transformation of a set of data from one form into another.
81. A projection is the transformation of a set of data from one form into another. In the case of event sourcing architecture, we convert a stream of events into any structural representation.
82. A projection is the transformation of a set of data from one form into another. In the case of event sourcing architecture, we convert a stream of events into any structural representation. -1 +1 +1 +4
83. Projection isn’t exclusive for event sourcing. But, it’s useful for event sourcing. !
84. CQRS REPORTING DATABASE PROJECTIONS ? !
85. CQRS: split the responsibility of "read" and “write" on the database.
86. CQRS: split the responsibility of "read" and “write" on the database. Reporting database: a database optimized for read. If a database is optimized for read, then could have redundancy and multiples databases to query.
87. CQRS: split the responsibility of "read" and “write" on the database. Reporting database: a database optimized for read. If a database is optimized for read, then could have redundancy and multiples databases to query. And, applying it with event sourcing: we could have a database that centralize all events received by the system (“event storage”), and many small databases optimized to read. This both reads the same bus and be updates when a new event is forwarded on the bus.
88. CQRS and Reporting Database aren’t exclusive for event sourcing. But, it’s useful when we are using event sourcing. !
89. COMMANDS ! CQRS REPORTING DATABASE PROJECTIONS
90. Commands are de requests issued by the user, and maybe fail. E.g: AddFunds
91. Commands are de requests issued by the user, and maybe fail. E.g: AddFunds When a command are performed, maybe it generates events - that is, facts about the application state changes E.g: FundsAdded
92. Life cycle:
93. VALIDATION Validations: checks if the command makes sense and objects are properly suited for further actions (is this event is to me?) Life cycle:
94. VALIDATION CONSEQUENCE Validations: checks if the command makes sense and objects are properly suited for further actions (is this event is to me?) Consequences: initiating some action that will change the state of the world (do I need to forward a new event?) Life cycle:
95. VALIDATION CONSEQUENCE DERIVATION Validations: checks if the command makes sense and objects are properly suited for further actions (is this event is to me?) Consequences: initiating some action that will change the state of the world (do I need to forward a new event?) Derivations: figuring out some information based on information we already have (I created a token and I’ll send it to user) Life cycle:
96. CQRS REPORTING DATABASE PROJECTIONS COMMANDS THE BIG FLUX!
97. User Event storage User can send a command on bus…
98. User Event storage Create an account List accounts Send a validate e-mail Validate an account User can send a command on bus, and many process manager (PM) reads this bus…
99. User Event storage Create an account List accounts Send a validate e-mail Validate an account User can send a command on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus.
100. User Event storage Create an account Send a validate e-mail List accounts Validate an account User can send a command on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data.
101. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send a command on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
102. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send a command on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
103. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
104. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
105. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
106. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
107. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
108. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
109. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
110. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
111. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
112. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
113. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
114. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
115. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
116. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
117. User Event storage Create an account Send a validate e-mail List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
118. YOU DON’T NEED TO DO IT YOURSELF
119. COMMANDED FRAMEWORK
120. Commanded framework! http://github.com/commanded
121. Commanded has more concepts that I didn’t say, (e.g: aggregate) and a great wiki.
122. Talk at Elixir Brasil by Bernardo Amorim! % http://bit.ly/elixir-brasil-bernado
123. Talk at Code Beam by Bernardo Amorim! % http://bit.ly/code-beam-bernardo-amorim
124. FURTHERMORE…
125. An idle game using event sourcing! https://github.com/telemmo
126. Visit my blog! ❤ http://macalogs.com.br/