AN INTRODUCTION TO EVENT SOURCING Alfredo Motta @mottalrd London Ruby User Group, May 9th 2018 

CREDITS @tadejm @dncrht @creditspringhq @krule @gregyoung #DDDx @krisleech 

MY JOURNEY SO FAR • PhD in Software Engineering (UML & formal methods) • Startup launching a second-hand market for daily deals (4 ppl) • Startup growing a holiday rentals website (100+ ppl) • P2P lender launching new products for car finance (200+ ppl) • Startup building a safer way to borrow (8 ppl) 

WE HAVE A CRUD PROBLEM (and event sourcing might be your cure) 

CRUD WEB APPLICATIONS 

No content

WHY WE OVERWRITE THE DATA? 

WHY WE OVERWRITE THE DATA? 

WHY WE READ AND WRITE FROM THE SAME TABLE? 

WHY WE READ AND WRITE FROM THE SAME TABLE? • Because it’s the simplest thing that makes sense? • Because it is the Rails convention? • Because we want to read our own writes? • Because we want to take advantages of database locks? 

WHY WE READ AND WRITE FROM THE SAME TABLE? time window with stale data 

A LITTLE SECRET (it is ok for data to be stale) 

A LITTLE SECRET • Business has been dealing with stale data forever • Overbooking of Hotel/Flights still happens today • What if some, or even all, of your data doesn't need to be consistent all the time? 

LET’S CHALLENGE THESE ASSUMPTIONS 

EVENT SOURCING PERSPECTIVE • Track changes, not state • Write events, read from eventually consistent projections of these events • Focus on immutability and idempotency 

• GIT store your current state as a change-log • i.e. a list of commits that represent your current state • it is ok if you and your co- worker does not have the same db at the same time GIT • Accountants never overwrite data • they add a new entry with the latest change in your account • they can always explain the reason behind your account balance Accountants 

You take the blue pill, the story ends. You wake up in your bed and believe whatever you want to believe. You take the red pill, you stay in Wonderland and I show you how deep the rabbit hole goes 

–Martin Fowler “Capture all changes to an application state as a sequence of events.” –Greg Young “A system where an Event Store is used to rebuild an object of the domain as opposed to something storing the current state.” EVENT SOURCING 

BEYOND CRUD 

BUILDING BLOCKS Events happen in the past email_updated email_updated at 2018-05-04 08:39:34 +0100 email_updated at 2018-05-04 08:39:34 +0100 for user with ID 99 with [email protected] email_updated at 2018-05-04 08:39:34 +0100 for user with ID 99 Events are immutable Events belongs to models *** Events have a payload storing what happened *** more precisely events belong to aggregates, aggregates define consistency boundaries for groups of related entities. 

BENEFITS • Read schemas are decoupled from event log • Auditability / why state is the way it is • Simple way to synchronise data between micro services 

EXAMPLE USE CASE A meeting scheduling applications 

A SIMPLE MEETING MODEL 

SCHEDULE A MEETING a command is a request for an event creation a successful command generates an event entity_id entity_type event_type data 1 Meeting scheduled { organizer_id: 1, invitee_ids: [2, 3], time: 2018-05-05 09:07:18 +0100 } 

SCHEDULE A MEETING (REJECTED) 

–Linus Torvalds “Talk is cheap, show me the code.” https://goo.gl/zREBmb 

EVENT SCHEMA (SIMPLE) create_table 'events', id: :serial do |t| t.uuid 'entity_id', null: false, index: true t.string 'event_type', null: false t.jsonb 'data', null: false t.datetime 'created_at', null: false end email_updated id of the user { new_email: [email protected] } . . 

EVENT SCHEMA (COMPLETE) create_table 'events', id: :serial do |t| t.uuid 'entity_id', null: false, index: true t.string 'entity_type', null: false t.string 'event_type', null: false t.jsonb 'meta', null: false t.jsonb 'data', null: false t.datetime 'created_at', null: false t.int 'entity_version', null: false t.index ['entity_id', 'entity_version'], name: 'event_version_constraint', unique: true end for example, User . . protects from multiple processes sending conflicting updates for all your meta-data needs 

class MeetingsController < ApplicationController def create response = EventSourced::Meeting.new(SecureRandom.uuid).schedule(schedule_meeting_params) if response.success? # ... else # ... end end private def schedule_meeting_params params.require(:meeting_form).permit(:time, :organizer_id, :invitee_ids) end [ … ] end 1. create meeting instance 2. invoke schedule command 

module EventSourced class Meeting attr_reader :id, :version, :status, :attendees_count, :time def initialize(id) @id = id @attendees_count = 0 @version = 0 end def schedule(time:, organizer_id:, invitee_ids:) Commands::Meeting::Schedule.call( self, time: meeting_time, organizer_id: organizer_id, invitee_ids: invitee_ids ) end [ … ] end end delegate to command for detailed validation 

module Commands module Meeting ScheduleSchema = Dry::Validation.Schema do required(:time).value(gt?: Time.now + 5.minutes) required(:organizer_id).filled required(:invitee_ids).filled { each(:int?) } end class Schedule < Command def call return validate if validate.failure? event = publish_event( event_type: 'scheduled', entity_type: 'Meeting', entity_id: @entity.id, entity_version: @entity.version, data: { time: @params[:time], organizer_id: @params[:organizer_id], invitee_ids: @params[:invitee_ids] } ) @entity.handle(event) Response.new success: true, data: { new_meeting: @entity } end end end end dry-validation gem to check the event data store the event with the appropriate data update the entity 

module EventSourced class Meeting [ … ] def scheduled_handler(event) params = event.data.symbolize_keys @time = params[:time] end def handle(event) handler = "#{event.event_type}_handler".to_sym if entity.respond_to? handler send(handler, event) @version += 1 end end [ … ] end end the event handler is responsible for updating the in-memory domain object 

No content

SCHEDULE A MEETING (RECAP) 

ACCEPT A MEETING By loading a domain object from the event store 

ACCEPT A MEETING bring domain object to the latest version 

class MeetingsController < ApplicationController [ … ] def accept meeting_id = accept_meeting_params[:meeting_id] response = EventSourced::Meeting .get(meeting_id) .accept(user_id: accept_meeting_params[:user_id]) flash[:alert] = "Failed to accept the meeting"if response.failure? redirect_to meeting_path(meeting_id) end private [ … ] def accept_meeting_params params.require(:accept_form).permit(:meeting_id, :user_id) end end get the entity and issue the command 

module EventSourced class Meeting [ … ] def handle(event) handler = "#{event.event_type}_handler".to_sym if entity.respond_to? handler send(handler, event) @version += 1 end end class << self def get(id) entity = new(id) load_events_for(entity) entity end def load_events_for(entity) events = Event.where(entity_type: 'meeting', entity_id: entity.id) events.each { |event| entity.handle(event) } end end end end send event to handler and increment version load events from the store and applies them to the given entity 

WHERE ARE MY ACTIVE RECORD MODELS? (we are de-railing from the conventions) 

WHERE ARE MY ACTIVE RECORD MODELS? • They become read-optimised views of your data • The single source of truth is the event log and your relational tables are a view on that log • Pro tip: you can have as many views as you want (think of reporting, search, analytics) 

WHERE ARE MY ACTIVE RECORD MODELS? 

class Meeting < ApplicationRecord has_one :organizer has_many :invitees class << self def scheduled_handler(event) params = event.data.symbolize_keys create!( entity_id: event.entity_id, time: params.fetch(:time), organizer_id: params.fetch(:organizer_id), created_at: event.created_at ) end def rescheduled_handler(event) meeting = find(event.entity_id) params = event.data.symbolize_keys meeting.update_attribute(time: params.fetch(:new_time)) end [ … ] end end create a row in a relational table when a scheduled event is received update an existing entry when a rescheduled event is received 

A NICE CONSEQUENCE 

IN SHORT • Event Sourcing is GIT applied to your data • You invoke commands to write events • Events become the single source of truth of your data • Your old active record models become your read- optimised views 

BENEFITS • Read schemas are decoupled from event log • Auditability / Why state is the way it is • Simple way to synchronise data between micro services 

DRAWBACKS • More complex • You will have to think about eventual consistency • You will have to think about idempotency • You de-rail from the Rails conventions 

No content

EVENT SOURCING IN RUBY • Beyond the current state: Time travel to the rescue!, Armin Pašalić, wroc_love.rb 2018 • Event Sourcing: A Rails Case Study, Francis Hwang, NYC.rb 2015 • https://github.com/fhwang/event_sourced_record • https://github.com/RailsEventStore/rails_event_store • https://github.com/Sandthorn/sandthorn • https://github.com/zilverline/sequent 

EVENT SOURCING REFERENCES • The Many Meanings of Event-Driven Architecture, Martin Fowler, GOTO 2017 • Event Sourcing, Martin Fowler, YOW! Nights 2016 • Implementing Domain Driven Design, Vaugh Vernon, 2013 • Greg Young Event DDD, CQRS, Event sourcing course • http://subscriptions.viddler.com/GregYoung/ • Promo code: LondonRuby for a $150 off • And a few limited 99% discount (grab me after the presentation) 

THANK YOU! QUESTIONS? 

APPENDIX 

EXTERNAL SYSTEMS • When issuing a command that requires a change to an external system • First call the external system (ideally inside the command) • If successful, then record the event 

IDEMPOTENCY • Any handler with side-effects should be turned off when rebuilding your projections • Emails • Analytics tools (Google Analytics / Mixpanel / Kissmetrics) • In general, any API call to an external system that is done as a result of an event 

SCHEMA VERSIONING • You can’t easily change event names • You can’t easily change the event schema • Depending on your tolerance you can use JSON / Protocol buffers / AVRO 

module EventSourced class Meeting def accept(user_id:) Commands::Meeting::Accept.call(self, user_id: user_id, status: @status) end def accepted_handler(event) @attendees_count += 1 end def reschedule(new_time:) Commands::Meeting::Reschedule.call(self, new_time: new_time, status: @status) end def rescheduled_handler(event) params = event.data.symbolize_keys @time = params[:new_time] end def finish Commands::Meeting::Finish.call(self, status: @status) end def finished_handler(event) @status = :finished end def cancel Commands::Meeting::Cancel.call(self, status: @status) end def cancelled_handler(event) @status = :cancelled end end end command command command event handler event handler event handler command event handler 

module Commands class Command def self.call(entity, params) new(entity, params).call end def initialize(entity, params) @entity = entity @params = params @validator = schema.call(params) end def publish_event(params) Event.create_and_broadcast(params) end def validate @validator.success? ? Response.new(success: true) : Response.new(success: false, errors: @validator.errors) end def schema "#{self.class.name}Schema".constantize end end end 

module Commands module Meeting AcceptSchema = Dry::Validation.Schema do required(:user_id).filled required(:status).value(eql?: :scheduled) end class Accept < Command def call return validate if validate.failure? publish_event( event_type: 'accepted', entity_type: 'Meeting', entity_id: @entity.id, entity_version: @entity.version, data: { user_id: @params[:user_id] } ) @entity.handle(event) Response.new success: true end end end end