Event Sourcing architecture using Elixir

Event Sourcing architecture using Elixir

Talk at "Nerdzão #37 - Back-End".

Video: https://youtu.be/tixWGkkZbXE
Projects used on this talk: https://github.com/macabeus/event-sourcing-example
My blog: http://macalogs.com.br/

F380535da59d6cdd5754e2e31bda8a0e?s=128

Bruno Macabeus

March 14, 2018
Tweet

Transcript

  1. EVENT SOURCING ARCHITECTURE USING ELIXIR Bruno Macabeus

  2. HELLO! ͜Μʹͪ͸ʂ

  3. Bruno Macabeus github.com/macabeus macalogs.com.br I’m a developer at Pagar.me ❤

  4. WHAT IS EVENT SOURCING?

  5. “The core idea of event sourcing is that whenever we

    make a change to the state of a system, we record that state change as an event, and we can confidently rebuild the system state by reprocessing the events at any time in the future.” - Martin Fowler youtu.be/STKCRSUsyP0 bit.ly/martin-event-driven
  6. This architecture has many interesting benefits!

  7. This architecture has many interesting benefits! Event log provides a

    strong audit capability
  8. This architecture has many interesting benefits! Event log provides a

    strong audit capability We can recreate historic states by replaying the event log up to a point
  9. This architecture has many interesting benefits! We can explore alternative

    histories by injecting hypothetical events when replaying Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point
  10. This architecture has many interesting benefits! Event sourcing makes it

    plausible to have non-durable working copies, such as a Memory Image. We can explore alternative histories by injecting hypothetical events when replaying Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point
  11. This architecture has many interesting benefits! Event sourcing makes it

    plausible to have non-durable working copies, such as a Memory Image. We can explore alternative histories by injecting hypothetical events when replaying Event log provides a strong audit capability We can recreate historic states by replaying the event log up to a point
  12. Did you know? Database journal file and Git are examples

    of using event sourcing!
  13. Did you know? Database journal file and Git are examples

    of using event sourcing!
  14. THE (VERY) SIMPLE FLUX

  15. User Server Current state: Events log: 0

  16. User Server Current state: Events log: 0 User creates an

    event
  17. User Server Current state: Events log: 0 In this example,

    the user “creates an event”. But, on the others implementations of this architecture, another part could create the events
  18. User Server Current state: Events log: 0

  19. User Server Current state: Events log: 0 Event “arrives" on

    the server. On the others implementations, the server could “create" the event
  20. User Server Current state: Events log: +1 1

  21. User Server Current state: Events log: +1 1

  22. User Server Current state: Events log: +1 1

  23. User Server Current state: Events log: +1 2 +1

  24. User Server Current state: Events log: +1 2 +1

  25. User Server Current state: Events log: +1 2 +1

  26. User Server Current state: Events log: -1 1 +1 +1

  27. User Server Current state: Events log: -1 1 +1 +1

    It’s a non-deterministic event
  28. User Server Current state: Events log: -1 1 +1 +1

  29. User Server Current state: Events log: -1 1 +1 +1

  30. User Server Current state: Events log: -1 5 +1 +1

    +4
  31. User Server Current state: Events log: -1 5 +1 +1

    +4 Now the events log has too many events
  32. User Server Current state: Events log: -1 5 +1 +1

    +4 Snapshot: state 5
  33. User Server Current state: Events log: -1 5 +1 +1

    +4 Snapshot: state 5 This is an aggregate. That is, a stream of events
  34. User Server Current state: Events log: -1 5 +1 +1

    +4 Snapshot: state 5
  35. User Server Current state: Events log: -1 5 +1 +1

    +4 Snapshot: state 5
  36. User Server Current state: Events log: -1 6 +1 +1

    +4 +1 Snapshot: state 5
  37. User Server Current state: Events log: -1 ?? +1 +1

    +4 +1 Snapshot: state 5 What if we lose the state?
  38. User Server Current state: Events log: -1 ?? +1 +1

    +4 +1 Snapshot: state 5 We could run again the events to restore the previous state! Using the events, we could convert the stream of events into a structural representation. This process is called “projection".
  39. GIVE ME AN EXAMPLE!

  40. Let’s see an Elixir project that uses event sourcing architecture!

    github.com/macabeus/event-sourcing-example
  41. GIVE ME AN EXAMPLE! HOW EVENT SOURCING WAS APPLIED IN

    THIS PROJECT?
  42. None
  43. None
  44. None
  45. None
  46. GIVE ME AN EXAMPLE! API

  47. None
  48. None
  49. None
  50. GIVE ME AN EXAMPLE! EVENTS

  51. None
  52. None
  53. Hey! It’s important the events be agnostic of a language,

    then you’ll can work using another languages on same project. Flux Standard Action is an example of solution about it. I didn't do it in my example. My example works only on Elixir
  54. GIVE ME AN EXAMPLE! PROCESS AND BUS PIPELINE

  55. None
  56. None
  57. None
  58. None
  59. None
  60. None
  61. None
  62. None
  63. None
  64. None
  65. None
  66. None
  67. GIVE ME AN EXAMPLE! DATABASE

  68. In this project, I used two databases:

  69. - An in-memory database, Mnesia, using the wrapper Amnesia In

    this project, I used two databases:
  70. - An in-memory database, Mnesia, using the wrapper Amnesia In

    this project, I used two databases: - A storage file database, DETS
  71. - An in-memory database, Mnesia, using the wrapper Amnesia In

    this project, I used two databases: - A storage file database, DETS Both, Mnesia and DETS, are in OTP! That is, built-in on Erlang/Elixir!
  72. BUT THERE ARE MANY MANY OTHERS WAYS TO BUILD AN

    EVENT SOURCING APPLICATION!
  73. MY SOLUTION SUCKS!

  74. ? ? ? ? !

  75. CQRS REPORTING DATABASE ? ? !

  76. CQRS: split the responsibility of "read" and “write" on the

    database.
  77. CQRS: split the responsibility of "read" and “write" on the

    database. Reporting database: a database optimized for read. If a database is optimized for read, then could has redundancy and multiples databases to query.
  78. CQRS: split the responsibility of "read" and “write" on the

    database. Reporting database: a database optimized for read. If a database is optimized for read, then could has redundancy and multiples databases to query. And, applying it with event sourcing: we could has a database that centralize all events received by the system (“event storage”), and many small databases optimized to read. This both reads the same bus and be updates when a new event is forwarded on the bus.
  79. CQRS and Reporting database aren’t exclusive for event sourcing. But,

    it’s useful when we are using event sourcing. !
  80. CQRS REPORTING DATABASE PROJECTIONS ? !

  81. A projection is the transformation of a set of data

    from one form into another.
  82. A projection is the transformation of a set of data

    from one form into another. In the case of event sourcing architecture, we convert a stream of events into any structural representation.
  83. A projection is the transformation of a set of data

    from one form into another. In the case of event sourcing architecture, we convert a stream of events into any structural representation. -1 +1 +1 +4
  84. Projection isn’t exclusive for event sourcing. But, it’s useful for

    event sourcing. !
  85. CQRS REPORTING DATABASE PROJECTIONS EAGER READ DERIVATION !

  86. Life cycle of an event:

  87. Life cycle of an event: VALIDATION Validations: checks that input

    makes sense and objects are properly suited for further actions. (this event is to me?)
  88. Life cycle of an event: VALIDATION CONSEQUENCE Validations: checks that

    input makes sense and objects are properly suited for further actions. (this event is to me?) Consequences: initiating some action that will change the state of the world (I need forward a new event?)
  89. Life cycle of an event: VALIDATION CONSEQUENCE DERIVATION Validations: checks

    that input makes sense and objects are properly suited for further actions. (this event is to me?) Consequences: initiating some action that will change the state of the world (I need forward a new event?) Derivations: figuring out some information based on information we already have (I created a token and I’ll send it to user)
  90. Life cycle of an event: VALIDATION CONSEQUENCE DERIVATION Validations: checks

    that input makes sense and objects are properly suited for further actions. (this event is to me?) Consequences: initiating some action that will change the state of the world (I need forward a new event?) Derivations: figuring out some information based on information we already have (I created a token and I’ll send it to user) Using Eager Read Derivation, this is done async between events. It’s useful because it otimize the system, but we’ll have eventual inconsistencies.
  91. Eager Read Derivation isn’t exclusive for event sourcing. But, it’s

    useful for event sourcing. !
  92. CQRS REPORTING DATABASE PROJECTIONS EAGER READ DERIVATION THE BIG FLUX!

  93. User Event storage User can send an event on bus…

  94. User Event storage Create an account List accounts Send a

    validate e-mail Validate an account User can send an event on bus, and many process manager (PM) reads this bus…
  95. User Event storage Create an account List accounts Send a

    validate e-mail Validate an account User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus.
  96. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data.
  97. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  98. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  99. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  100. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  101. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  102. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  103. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  104. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  105. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  106. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  107. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  108. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  109. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  110. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  111. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  112. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  113. User Event storage Create an account Send a validate e-mail

    List accounts Validate an account Events processing are asynchronous! User can send an event on bus, and many process manager (PM) reads this bus. Maybe a PM could send a new event on the bus. And, the PM needs read events in Event Storage, to re-build when it loses the data. And everything could be async!
  114. YOU DON’T NEED DO IT YOURSELF

  115. COMMANDED FRAMEWORK

  116. Commanded framework! http://bit.ly/elixir-brasil-bernado

  117. Talk at Elixir Brasil by Bernado Amorim! % http://bit.ly/elixir-brasil-bernado

  118. FURTHERMORE…

  119. An idle game using event sourcing! https://github.com/telemmo

  120. Visit my blog! ❤ http://macalogs.com.br/