Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Events. Events. Events! - krk.rb

Events. Events. Events! - krk.rb

Basic conceptions of event sourcing and how to use it in ruby.

Talk from https://krk-rb.pl

Anton Davydov

May 14, 2019
Tweet

More Decks by Anton Davydov

Other Decks in Programming

Transcript

  1. View Slide

  2. View Slide

  3. View Slide

  4. Hello krk.rb

    View Slide

  5. Hard day

    View Slide

  6. View Slide

  7. View Slide

  8. Second time in Poland

    View Slide

  9. View Slide

  10. View Slide

  11. What I know now

    View Slide

  12. View Slide

  13. View Slide

  14. Pierogi

    View Slide

  15. Pelmeni -> Pierogi z mięsem
    Varenyky -> Pierogi ruskie

    View Slide

  16. Awesome people and community

    View Slide

  17. Anton Davydov
    github.com/davydovanton

    twitter.com/anton_davydov
    davydovanton.com

    View Slide

  18. View Slide

  19. View Slide

  20. • Architect/software developer
    • OpenSource evangelist
    • Hanami core developer
    • Writer/programming streamer
    • Technical consulting

    View Slide

  21. Stickers

    View Slide

  22. • Coffee
    • Beer
    • Psychology
    • SW-2039-0208-4228 / 3DS also welcome
    • How to draw images for presentations
    • Bad stories

    View Slide

  23. • Coffee
    • Beer
    • Psychology
    • SW-2039-0208-4228 / 3DS also welcome
    • How to draw images for presentations
    • Bad stories

    View Slide

  24. Tried to be a whiter and make a story

    View Slide

  25. expectations

    View Slide

  26. reality

    View Slide

  27. But I got help from my friend

    View Slide

  28. Meet George

    View Slide

  29. He is quiet and love to ask questions

    View Slide

  30. View Slide

  31. View Slide

  32. Events

    View Slide

  33. View Slide

  34. View Slide

  35. View Slide

  36. Git

    View Slide

  37. Each commit - something
    that happened

    View Slide

  38. View Slide

  39. Each commit contains
    description and changes
    (payload)

    View Slide

  40. View Slide

  41. What happened

    View Slide

  42. Payload

    View Slide

  43. List of cart items which
    user deleted and added

    View Slide

  44. Activity audit

    View Slide

  45. Don’t delete deleted data

    View Slide

  46. deleted_at

    View Slide

  47. But we usually work with the
    current state of application

    View Slide

  48. events -> actions

    View Slide

  49. events -> actions
    current state -> data in DB

    View Slide

  50. Knok knok

    George

    View Slide



  51. George

    View Slide

  52. What will happen if we
    change this situation?

    George

    View Slide

  53. And start to store
    events

    George

    View Slide

  54. And calculate current
    state when we need it

    George

    View Slide

  55. Event Sourcing

    View Slide

  56. Store that happened instead
    of current state

    View Slide

  57. And get the specific state
    when you need it

    View Slide

  58. Where to store?

    View Slide

  59. Event storage

    View Slide

  60. It can be anything

    View Slide

  61. View Slide

  62. Rules

    View Slide

  63. Event happened

    View Slide

  64. Immutable events

    View Slide

  65. Can’t delete or update event

    View Slide

  66. View Slide

  67. View Slide

  68. View Slide

  69. Only a valid data in the
    events

    View Slide

  70. View Slide

  71. View Slide

  72. View Slide

  73. View Slide

  74. View Slide

  75. Data evolution

    View Slide

  76. View Slide

  77. View Slide

  78. View Slide

  79. View Slide

  80. View Slide

  81. Stop

    George

    View Slide

  82. We can break the app!

    George

    View Slide

  83. >_<

    George

    View Slide

  84. Pro tips: optional fields,
    binary data structures

    View Slide

  85. more in this book

    View Slide

  86. How to get the state
    from the event store?

    View Slide

  87. We have to calculate it
    using current event list

    View Slide

  88. Projections

    View Slide

  89. project(projection, events, init_state) -> state

    View Slide

  90. project(projection, events, init_state) -> state
    same as a block in ruby’s enumerator

    View Slide

  91. project(projection, events, init_state) -> state
    pure function

    View Slide

  92. Similar to #each_with_object

    View Slide

  93. events.project(init_state, &projection) -> state

    View Slide

  94. events.project(init_state, &projection) -> state
    [].each_with_object({}, &projection)

    View Slide

  95. View Slide

  96. View Slide

  97. View Slide

  98. View Slide

  99. View Slide

  100. View Slide

  101. View Slide

  102. View Slide

  103. View Slide

  104. View Slide

  105. View Slide

  106. View Slide

  107. We recalculate current state
    for all events each time

    View Slide



  108. George

    View Slide

  109. We can’t store all data for
    last 10 years

    George

    View Slide

  110. But we will have a
    performance issues in
    getting current state ▼
    George

    View Slide

  111. How to fix it?

    View Slide

  112. Streams!

    View Slide

  113. View Slide

  114. Event bus#1
    Event bus#1
    Event bus#2
    Event bus#1
    Event bus#2
    Event bus#2

    View Slide

  115. Event stream: bus#1
    Event stream: bus#1
    Event stream: bus#2
    Event stream: bus#1
    Event stream: bus#2
    Event stream: bus#2

    View Slide

  116. Stream: bus#1

    Event stream: bus#1
    Event stream: bus#1
    Event stream: bus#1

    View Slide

  117. Stream: bus#2

    Event stream: bus#2
    Event stream: bus#2
    Event stream: bus#2

    View Slide

  118. Pros: less events for processing

    View Slide

  119. Cons: one more abstraction

    View Slide

  120. Add snapshot for the current
    data state!

    View Slide

  121. project(function, events, state) -> state

    View Slide

  122. project(function, events, state) -> state
    {}

    View Slide

  123. project(function, events, state) -> state
    { users: […] }

    View Slide

  124. View Slide

  125. View Slide

  126. View Slide

  127. View Slide

  128. View Slide

  129. View Slide

  130. You can use any DB as a cache

    View Slide

  131. View Slide

  132. Idea: use more that one DB
    for cache

    View Slide

  133. Relation DB for search and index
    page

    View Slide

  134. Document oriented DB for show
    page

    View Slide

  135. Real world

    View Slide

  136. Where it can be useful

    View Slide

  137. E-comerse systems

    View Slide

  138. Order and checkout flow

    View Slide

  139. Something where you need a
    version control
    (wiki, docs, group editing)

    View Slide

  140. Domain depends on events
    (tracking systems)

    View Slide

  141. redux?

    George

    View Slide

  142. Event sourcing and event
    driven architecture are
    different things

    View Slide

  143. And they use events in
    different ways

    View Slide

  144. ES -> event happened and I store it
    ED -> event happened and I notify
    everyone about it

    View Slide

  145. Blockchain!

    George

    View Slide

  146. pros

    View Slide

  147. Easier to communicate with
    domain experts

    View Slide

  148. • Logging out from box
    • Time traveling
    • Restore system’s state

    View Slide

  149. • You don’t work with tables you work with
    events and changes
    • Experemental data structures
    • Easy to change database implementations

    View Slide

  150. cons

    View Slide

  151. • Hard to understand idea and
    complicated abstraction
    • Not popular in ruby
    • Developers need deprogramming

    View Slide

  152. • Harder to get state than CRUD
    • Hard to understand the whole
    chain of events
    • Another architecture type

    with different DB structure

    View Slide

  153. • Versions and versions
    compatibility
    • Updating or deleting
    events
    • Eventual Consistency

    View Slide

  154. Advanced topics

    View Slide

  155. How to display created data ASAP?

    View Slide

  156. Solution 1

    View Slide

  157. cheat user
    display real
    data

    View Slide

  158. View Slide

  159. View Slide

  160. View Slide

  161. View Slide

  162. View Slide

  163. solution 2

    View Slide

  164. WebSockets and Long Polling

    View Slide

  165. How to make right events order?

    View Slide

  166. View Slide

  167. View Slide

  168. View Slide

  169. View Slide

  170. distributed systems

    View Slide

  171. more here

    View Slide

  172. And here
    https://youtu.be/ArTS_AJ-smQ

    View Slide

  173. How to delete a user related data

    View Slide

  174. Retroactive Event

    View Slide

  175. View Slide

  176. Lose encrypted data key

    View Slide

  177. How to use ES or something like
    this for searching in the event
    store?

    View Slide

  178. How to use ES in ruby

    View Slide

  179. Simple way

    View Slide

  180. gems

    View Slide

  181. View Slide

  182. github.com/zilverline/sequent

    View Slide

  183. View Slide

  184. eventide-project.org

    View Slide

  185. View Slide

  186. Brave way

    View Slide

  187. Build all stuff by yourself

    View Slide

  188. View Slide

  189. View Slide

  190. Or pure kafka

    View Slide

  191. View Slide

  192. Or pure SQL

    View Slide

  193. github.com/davydovanton/ivento
    * Repository with pure sql implementation

    View Slide

  194. github.com/davydovanton/ivento
    * And without tests

    View Slide

  195. hanami-events

    View Slide

  196. Event driven transport layer
    with event versions
    and types

    View Slide

  197. Connection without global state

    View Slide

  198. Hanami::Events.new(:memory_sync)
    Hanami::Events.new(:memory_async)
    Hanami::Events.new(:redis)

    View Slide

  199. Hanami::Events::Adapter.register(:sql) do
    Adapter::SQL
    end
    Hanami::Events.new(:sql)
    # => event instance with your sql adapter

    View Slide

  200. events = Hanami::Events.new(:memory_sync)
    events.broadcast('user.created', user: user)
    Broadcaster

    View Slide

  201. events.subscribe('user.created') do |payload|
    puts payload
    end
    events.subscribe('user.*') do |payload|
    puts payload
    end
    events.subscribe(/\Auser\..*/) do |payload|
    puts payload
    end
    Subscriber

    View Slide

  202. Event sourcing looks
    cool.

    George

    View Slide

  203. But I have legacy app.
    How can I migrate?

    George

    View Slide

  204. User adds something to cart

    View Slide

  205. module Web::Controllers::CartItem
    class Create
    include Web::Action
    def call(params)
    halt(400) unless params.valid?
    if cart_repo.item_exist?(params[:item])
    item = CartItemRepository.new.create_copy(params[:item])
    else
    item = CartItemRepository.new.create(params[:item])
    end
    analytics.call('Item Added')
    redirect_to routes.order_path(params[:order_id])
    end
    end
    end

    View Slide

  206. Step 1: Event Storming

    View Slide

  207. Why?
    Detect important business
    events in the system

    View Slide

  208. Orders::Events::ItemAdded
    Orders::Events::CoppyItemAdded

    View Slide

  209. Step 2: store (apply) event

    View Slide

  210. module Web::Controllers::CartItem
    class Create
    include Web::Action
    def call(params)
    halt(400) unless params.valid?
    event_store.apply(Order::Events::ItemAdded.new(payload))
    if cart_repo.item_exist?(params[:item])
    item = CartItemRepository.new.create_copy(params[:item])
    else
    item = CartItemRepository.new.create(params[:item])
    end
    analytics.call('Item Added')
    redirect_to routes.order_path(params[:order_id])
    end
    end
    end

    View Slide

  211. How to validate item?

    View Slide

  212. module Web::Controllers::CartItem
    class Create
    include Web::Action
    def call(params)
    halt(400) unless params.valid?
    event_store.apply(Orders::Events::ItemAdded.new(payload))
    if cart_repo.item_exist?(params[:item])
    item = CartItemRepository.new.create_copy(params[:item])
    else
    item = CartItemRepository.new.create(params[:item])
    end
    analytics.call('Item Added')
    redirect_to routes.order_path(params[:order_id])
    end
    end
    end

    View Slide

  213. Step 3: calculate state

    View Slide

  214. module Web::Controllers::CartItem
    class Create
    include Web::Action
    def call(params)
    halt(400) unless params.valid?
    events = event_store.get_stream(params[:order_id])
    items = @project.call(Order::Projections::ItemList, events)
    if items.include?(params[:item])
    event_store.apply(Orders::Events::ItemAdded.new(payload))
    else
    event_store.apply(Orders::Events::CoppyItemAdded.new(payload))
    end
    analytics.call('Item Added')
    redirect_to routes.order_path(params[:order_id])
    end
    end
    end

    View Slide

  215. module Web::Controllers::CartItem
    class Create
    include Web::Action
    def call(params)
    halt(400) unless params.valid?
    events = event_store.get_stream(params[:order_id])
    items = @project.call(Order::Projections::ItemList, events, {})
    if items.include?(params[:item])
    event_store.apply(Orders::Events::ItemAdded.new(payload))
    else
    event_store.apply(Orders::Events::CoppyItemAdded.new(payload))
    end
    analytics.call('Item Added')
    redirect_to routes.order_path(params[:order_id])
    end
    end
    end

    View Slide

  216. module Projections
    class AllTask
    def call(event, state)
    case event
    when Orders::Events::ItemAdded
    state[:orders] ||= []
    state[:orders] << event.payload
    when Orders::Events::ItemRemoved
    # ...
    end
    state
    end
    end
    end

    View Slide

  217. module Projections
    class AllTask
    def call(event, state)
    case event
    when Orders::Events::ItemAdded
    state[:orders] ||= []
    state[:orders] << event.payload
    when Orders::Events::ItemRemoved
    # ...
    end
    state
    end
    end
    end

    View Slide

  218. module Projections
    class AllTask
    def call(event, state)
    case event
    when Orders::Events::ItemAdded
    state[:orders] ||= []
    state[:orders] << event.payload
    when Orders::Events::ItemRemoved
    # ...
    end
    state
    end
    end
    end

    View Slide

  219. module Web::Controllers::CartItem
    class Create
    include Web::Action
    def call(params)
    halt(400) unless params.valid?
    events = event_store.get_stream(params[:order_id])
    items = @project.call(Order::Projections::ItemList, events, {})
    if items.include?(params[:item])
    event_store.apply(Orders::Events::ItemAdded.new(payload))
    else
    event_store.apply(Orders::Events::CoppyItemAdded.new(payload))
    end
    analytics.call('Item Added')
    redirect_to routes.order_path(params[:order_id])
    end
    end
    end

    View Slide

  220. Step 4: subscribers

    View Slide

  221. # analytics subscribers
    event_store.subscribe(Orders::Events::ItemAdded) do |event|
    analytics.call('Item removed', event.payload)
    end
    event_store.subscribe(Orders::Events::CoppyItemAdded) do |event|
    analytics.call(‘Copy of item added’, event.payload)
    end
    event_store.subscribe(Orders::Events::ItemRemoved) do |event|
    analytics.call(‘Item removed', event.payload)
    end

    View Slide

  222. module Web::Controllers::CartItem
    class Create
    include Web::Action
    def call(params)
    halt(400) unless params.valid?
    events = event_store.get_stream(params[:order_id])
    items = @project.call(Order::Projections::ItemList, events, {})
    if items.include?(params[:item])
    event_store.apply(Orders::Events::ItemAdded.new(payload))
    else
    event_store.apply(Orders::Events::CoppyItemAdded.new(payload))
    end
    redirect_to routes.order_path(params[:order_id])
    end
    end
    end

    View Slide

  223. Step 4: move logic to domain

    View Slide

  224. PROFIT

    View Slide

  225. pro tip: you can use ES only
    as a part of your system

    View Slide

  226. What next?

    View Slide

  227. View Slide

  228. View Slide

  229. SAGA PATTERN

    View Slide

  230. Long business transactions

    View Slide

  231. View Slide

  232. View Slide

  233. CQRS


    (Command Query
    Responsibility Segregation)

    View Slide

  234. – Martin Fowler
    “You can use a different
    model to update information
    than the model you use to
    read information”

    View Slide

  235. Looking outside
    ruby world

    View Slide

  236. github.com/heynickc/
    awesome-ddd

    View Slide

  237. github.com/davydovanton/event_sourcing_ruby

    View Slide

  238. DDD

    View Slide

  239. Again?

    George

    View Slide

  240. Detect and store business events

    View Slide

  241. Split system by domains

    View Slide

  242. View Slide

  243. View Slide

  244. View Slide

  245. Distributed systems

    View Slide

  246. more here

    View Slide

  247. Conclusions

    View Slide

  248. Sometimes event base
    architecture wins

    View Slide

  249. Sometimes not

    View Slide

  250. EventSourcing is simple as
    conception

    View Slide

  251. But it introduces a lot of
    practical problems

    (data evolution, event order, etc)

    View Slide

  252. ES expensive

    View Slide

  253. It’s not about distributed
    systems and communication
    across services

    View Slide

  254. ES will be a trend in the
    future

    View Slide

  255. Don’t you trust me?

    View Slide

  256. – Andrzej Krzywda
    “ES will be a trend in
    the future”

    View Slide

  257. twitter.com/anton_davydov
    github.com/davydovanton
    [email protected]
    davydovanton.com
    Thank you ❤

    View Slide