An Introduction to Event Sourcing for Rubyists

Event sourcing is a design pattern to build applications that are domain centric and easy to extend. The pattern is based on the usage of a persistent event log which substitutes the more classical relational database model for Rails applications.

Such event log can then be used for extending your application in all sort of creative ways. For example, by synchronizing data between your microservices, trigger side effects without cluttering your models or controllers, or build data views optimized for your query needs. In this talk, I'll present the basic ideas, some of the tradeoffs and challenges you might find and how you could start experimenting with it.

Transcript

1. AN INTRODUCTION TO EVENT SOURCING Alfredo Motta @mottalrd London Ruby User Group, May 9th 2018
2. CREDITS @tadejm @dncrht @creditspringhq @krule @gregyoung #DDDx @krisleech
3. MY JOURNEY SO FAR • PhD in Software Engineering (UML & formal methods) • Startup launching a second-hand market for daily deals (4 ppl) • Startup growing a holiday rentals website (100+ ppl) • P2P lender launching new products for car finance (200+ ppl) • Startup building a safer way to borrow (8 ppl)
4. WE HAVE A CRUD PROBLEM (and event sourcing might be your cure)
5. CRUD WEB APPLICATIONS
7. WHY WE OVERWRITE THE DATA?
8. WHY WE OVERWRITE THE DATA?
9. WHY WE READ AND WRITE FROM THE SAME TABLE?
10. WHY WE READ AND WRITE FROM THE SAME TABLE? • Because it’s the simplest thing that makes sense? • Because it is the Rails convention? • Because we want to read our own writes? • Because we want to take advantages of database locks?
11. WHY WE READ AND WRITE FROM THE SAME TABLE? time window with stale data
12. A LITTLE SECRET (it is ok for data to be stale)
13. A LITTLE SECRET • Business has been dealing with stale data forever • Overbooking of Hotel/Flights still happens today • What if some, or even all, of your data doesn't need to be consistent all the time?
14. LET’S CHALLENGE THESE ASSUMPTIONS
15. EVENT SOURCING PERSPECTIVE • Track changes, not state • Write events, read from eventually consistent projections of these events • Focus on immutability and idempotency
16. • GIT store your current state as a change-log • i.e. a list of commits that represent your current state • it is ok if you and your co- worker does not have the same db at the same time GIT • Accountants never overwrite data • they add a new entry with the latest change in your account • they can always explain the reason behind your account balance Accountants
17. You take the blue pill, the story ends. You wake up in your bed and believe whatever you want to believe. You take the red pill, you stay in Wonderland and I show you how deep the rabbit hole goes
18. –Martin Fowler “Capture all changes to an application state as a sequence of events.” –Greg Young “A system where an Event Store is used to rebuild an object of the domain as opposed to something storing the current state.” EVENT SOURCING
19. BEYOND CRUD
20. BUILDING BLOCKS Events happen in the past email_updated email_updated at 2018-05-04 08:39:34 +0100 email_updated at 2018-05-04 08:39:34 +0100 for user with ID 99 with a@a.com email_updated at 2018-05-04 08:39:34 +0100 for user with ID 99 Events are immutable Events belongs to models *** Events have a payload storing what happened *** more precisely events belong to aggregates, aggregates define consistency boundaries for groups of related entities.
21. BENEFITS • Read schemas are decoupled from event log • Auditability / why state is the way it is • Simple way to synchronise data between micro services
22. EXAMPLE USE CASE A meeting scheduling applications
23. A SIMPLE MEETING MODEL
24. SCHEDULE A MEETING a command is a request for an event creation a successful command generates an event entity_id entity_type event_type data 1 Meeting scheduled { organizer_id: 1, invitee_ids: [2, 3], time: 2018-05-05 09:07:18 +0100 }
25. SCHEDULE A MEETING (REJECTED)
26. –Linus Torvalds “Talk is cheap, show me the code.” https://goo.gl/zREBmb
27. EVENT SCHEMA (SIMPLE) create_table 'events', id: :serial do |t| t.uuid 'entity_id', null: false, index: true t.string 'event_type', null: false t.jsonb 'data', null: false t.datetime 'created_at', null: false end email_updated id of the user { new_email: a@a.com } . .
28. EVENT SCHEMA (COMPLETE) create_table 'events', id: :serial do |t| t.uuid 'entity_id', null: false, index: true t.string 'entity_type', null: false t.string 'event_type', null: false t.jsonb 'meta', null: false t.jsonb 'data', null: false t.datetime 'created_at', null: false t.int 'entity_version', null: false t.index ['entity_id', 'entity_version'], name: 'event_version_constraint', unique: true end for example, User . . protects from multiple processes sending conflicting updates for all your meta-data needs
29. class MeetingsController < ApplicationController def create response = EventSourced::Meeting.new(SecureRandom.uuid).schedule(schedule_meeting_params) if response.success? # ... else # ... end end private def schedule_meeting_params params.require(:meeting_form).permit(:time, :organizer_id, :invitee_ids) end [ … ] end 1. create meeting instance 2. invoke schedule command
30. module EventSourced class Meeting attr_reader :id, :version, :status, :attendees_count, :time def initialize(id) @id = id @attendees_count = 0 @version = 0 end def schedule(time:, organizer_id:, invitee_ids:) Commands::Meeting::Schedule.call( self, time: meeting_time, organizer_id: organizer_id, invitee_ids: invitee_ids ) end [ … ] end end delegate to command for detailed validation
31. module Commands module Meeting ScheduleSchema = Dry::Validation.Schema do required(:time).value(gt?: Time.now + 5.minutes) required(:organizer_id).filled required(:invitee_ids).filled { each(:int?) } end class Schedule < Command def call return validate if validate.failure? event = publish_event( event_type: 'scheduled', entity_type: 'Meeting', entity_id: @entity.id, entity_version: @entity.version, data: { time: @params[:time], organizer_id: @params[:organizer_id], invitee_ids: @params[:invitee_ids] } ) @entity.handle(event) Response.new success: true, data: { new_meeting: @entity } end end end end dry-validation gem to check the event data store the event with the appropriate data update the entity
32. module EventSourced class Meeting [ … ] def scheduled_handler(event) params = event.data.symbolize_keys @time = params[:time] end def handle(event) handler = "#{event.event_type}_handler".to_sym if entity.respond_to? handler send(handler, event) @version += 1 end end [ … ] end end the event handler is responsible for updating the in-memory domain object
34. SCHEDULE A MEETING (RECAP)
35. ACCEPT A MEETING By loading a domain object from the event store
36. ACCEPT A MEETING bring domain object to the latest version
37. class MeetingsController < ApplicationController [ … ] def accept meeting_id = accept_meeting_params[:meeting_id] response = EventSourced::Meeting .get(meeting_id) .accept(user_id: accept_meeting_params[:user_id]) flash[:alert] = "Failed to accept the meeting"if response.failure? redirect_to meeting_path(meeting_id) end private [ … ] def accept_meeting_params params.require(:accept_form).permit(:meeting_id, :user_id) end end get the entity and issue the command
38. module EventSourced class Meeting [ … ] def handle(event) handler = "#{event.event_type}_handler".to_sym if entity.respond_to? handler send(handler, event) @version += 1 end end class << self def get(id) entity = new(id) load_events_for(entity) entity end def load_events_for(entity) events = Event.where(entity_type: 'meeting', entity_id: entity.id) events.each { |event| entity.handle(event) } end end end end send event to handler and increment version load events from the store and applies them to the given entity
39. WHERE ARE MY ACTIVE RECORD MODELS? (we are de-railing from the conventions)
40. WHERE ARE MY ACTIVE RECORD MODELS? • They become read-optimised views of your data • The single source of truth is the event log and your relational tables are a view on that log • Pro tip: you can have as many views as you want (think of reporting, search, analytics)
41. WHERE ARE MY ACTIVE RECORD MODELS?
42. class Meeting < ApplicationRecord has_one :organizer has_many :invitees class << self def scheduled_handler(event) params = event.data.symbolize_keys create!( entity_id: event.entity_id, time: params.fetch(:time), organizer_id: params.fetch(:organizer_id), created_at: event.created_at ) end def rescheduled_handler(event) meeting = find(event.entity_id) params = event.data.symbolize_keys meeting.update_attribute(time: params.fetch(:new_time)) end [ … ] end end create a row in a relational table when a scheduled event is received update an existing entry when a rescheduled event is received
43. A NICE CONSEQUENCE
44. IN SHORT • Event Sourcing is GIT applied to your data • You invoke commands to write events • Events become the single source of truth of your data • Your old active record models become your read- optimised views
45. BENEFITS • Read schemas are decoupled from event log • Auditability / Why state is the way it is • Simple way to synchronise data between micro services
46. DRAWBACKS • More complex • You will have to think about eventual consistency • You will have to think about idempotency • You de-rail from the Rails conventions
48. EVENT SOURCING IN RUBY • Beyond the current state: Time travel to the rescue!, Armin Pašalić, wroc_love.rb 2018 • Event Sourcing: A Rails Case Study, Francis Hwang, NYC.rb 2015 • https://github.com/fhwang/event_sourced_record • https://github.com/RailsEventStore/rails_event_store • https://github.com/Sandthorn/sandthorn • https://github.com/zilverline/sequent
49. EVENT SOURCING REFERENCES • The Many Meanings of Event-Driven Architecture, Martin Fowler, GOTO 2017 • Event Sourcing, Martin Fowler, YOW! Nights 2016 • Implementing Domain Driven Design, Vaugh Vernon, 2013 • Greg Young Event DDD, CQRS, Event sourcing course • http://subscriptions.viddler.com/GregYoung/ • Promo code: LondonRuby for a $150 off • And a few limited 99% discount (grab me after the presentation)
50. THANK YOU! QUESTIONS?
51. APPENDIX
52. EXTERNAL SYSTEMS • When issuing a command that requires a change to an external system • First call the external system (ideally inside the command) • If successful, then record the event
53. IDEMPOTENCY • Any handler with side-effects should be turned off when rebuilding your projections • Emails • Analytics tools (Google Analytics / Mixpanel / Kissmetrics) • In general, any API call to an external system that is done as a result of an event
54. SCHEMA VERSIONING • You can’t easily change event names • You can’t easily change the event schema • Depending on your tolerance you can use JSON / Protocol buffers / AVRO
55. module EventSourced class Meeting def accept(user_id:) Commands::Meeting::Accept.call(self, user_id: user_id, status: @status) end def accepted_handler(event) @attendees_count += 1 end def reschedule(new_time:) Commands::Meeting::Reschedule.call(self, new_time: new_time, status: @status) end def rescheduled_handler(event) params = event.data.symbolize_keys @time = params[:new_time] end def finish Commands::Meeting::Finish.call(self, status: @status) end def finished_handler(event) @status = :finished end def cancel Commands::Meeting::Cancel.call(self, status: @status) end def cancelled_handler(event) @status = :cancelled end end end command command command event handler event handler event handler command event handler
56. module Commands class Command def self.call(entity, params) new(entity, params).call end def initialize(entity, params) @entity = entity @params = params @validator = schema.call(params) end def publish_event(params) Event.create_and_broadcast(params) end def validate @validator.success? ? Response.new(success: true) : Response.new(success: false, errors: @validator.errors) end def schema "#{self.class.name}Schema".constantize end end end
57. module Commands module Meeting AcceptSchema = Dry::Validation.Schema do required(:user_id).filled required(:status).value(eql?: :scheduled) end class Accept < Command def call return validate if validate.failure? publish_event( event_type: 'accepted', entity_type: 'Meeting', entity_id: @entity.id, entity_version: @entity.version, data: { user_id: @params[:user_id] } ) @entity.handle(event) Response.new success: true end end end end