Upgrade to Pro — share decks privately, control downloads, hide ads and more …

An Introduction to Event Sourcing for Rubyists

An Introduction to Event Sourcing for Rubyists

Event sourcing is a design pattern to build applications that are domain centric and easy to extend. The pattern is based on the usage of a persistent event log which substitutes the more classical relational database model for Rails applications.

Such event log can then be used for extending your application in all sort of creative ways. For example, by synchronizing data between your microservices, trigger side effects without cluttering your models or controllers, or build data views optimized for your query needs. In this talk, I'll present the basic ideas, some of the tradeoffs and challenges you might find and how you could start experimenting with it.

384bde12832c12deef6e9a75b08bafc4?s=128

Alfredo Motta

May 09, 2018
Tweet

Transcript

  1. AN INTRODUCTION TO EVENT SOURCING Alfredo Motta @mottalrd London Ruby

    User Group, May 9th 2018
  2. CREDITS @tadejm @dncrht @creditspringhq @krule @gregyoung #DDDx @krisleech

  3. MY JOURNEY SO FAR • PhD in Software Engineering (UML

    & formal methods) • Startup launching a second-hand market for daily deals (4 ppl) • Startup growing a holiday rentals website (100+ ppl) • P2P lender launching new products for car finance (200+ ppl) • Startup building a safer way to borrow (8 ppl)
  4. WE HAVE A CRUD PROBLEM (and event sourcing might be

    your cure)
  5. CRUD WEB APPLICATIONS

  6. None
  7. WHY WE OVERWRITE THE DATA?

  8. WHY WE OVERWRITE THE DATA?

  9. WHY WE READ AND WRITE FROM THE SAME TABLE?

  10. WHY WE READ AND WRITE FROM THE SAME TABLE? •

    Because it’s the simplest thing that makes sense? • Because it is the Rails convention? • Because we want to read our own writes? • Because we want to take advantages of database locks?
  11. WHY WE READ AND WRITE FROM THE SAME TABLE? time

    window with stale data
  12. A LITTLE SECRET (it is ok for data to be

    stale)
  13. A LITTLE SECRET • Business has been dealing with stale

    data forever • Overbooking of Hotel/Flights still happens today • What if some, or even all, of your data doesn't need to be consistent all the time?
  14. LET’S CHALLENGE THESE ASSUMPTIONS

  15. EVENT SOURCING PERSPECTIVE • Track changes, not state • Write

    events, read from eventually consistent projections of these events • Focus on immutability and idempotency
  16. • GIT store your current state as a change-log •

    i.e. a list of commits that represent your current state • it is ok if you and your co- worker does not have the same db at the same time GIT • Accountants never overwrite data • they add a new entry with the latest change in your account • they can always explain the reason behind your account balance Accountants
  17. You take the blue pill, the story ends. You wake

    up in your bed and believe whatever you want to believe. You take the red pill, you stay in Wonderland and I show you how deep the rabbit hole goes
  18. –Martin Fowler “Capture all changes to an application state as

    a sequence of events.” –Greg Young “A system where an Event Store is used to rebuild an object of the domain as opposed to something storing the current state.” EVENT SOURCING
  19. BEYOND CRUD

  20. BUILDING BLOCKS Events happen in the past email_updated email_updated at

    2018-05-04 08:39:34 +0100 email_updated at 2018-05-04 08:39:34 +0100 for user with ID 99 with a@a.com email_updated at 2018-05-04 08:39:34 +0100 for user with ID 99 Events are immutable Events belongs to models *** Events have a payload storing what happened *** more precisely events belong to aggregates, aggregates define consistency boundaries for groups of related entities.
  21. BENEFITS • Read schemas are decoupled from event log •

    Auditability / why state is the way it is • Simple way to synchronise data between micro services
  22. EXAMPLE USE CASE A meeting scheduling applications

  23. A SIMPLE MEETING MODEL

  24. SCHEDULE A MEETING a command is a request for an

    event creation a successful command generates an event entity_id entity_type event_type data 1 Meeting scheduled { organizer_id: 1, invitee_ids: [2, 3], time: 2018-05-05 09:07:18 +0100 }
  25. SCHEDULE A MEETING (REJECTED)

  26. –Linus Torvalds “Talk is cheap, show me the code.” https://goo.gl/zREBmb

  27. EVENT SCHEMA (SIMPLE) create_table 'events', id: :serial do |t| t.uuid

    'entity_id', null: false, index: true t.string 'event_type', null: false t.jsonb 'data', null: false t.datetime 'created_at', null: false end email_updated id of the user { new_email: a@a.com } . .
  28. EVENT SCHEMA (COMPLETE) create_table 'events', id: :serial do |t| t.uuid

    'entity_id', null: false, index: true t.string 'entity_type', null: false t.string 'event_type', null: false t.jsonb 'meta', null: false t.jsonb 'data', null: false t.datetime 'created_at', null: false t.int 'entity_version', null: false t.index ['entity_id', 'entity_version'], name: 'event_version_constraint', unique: true end for example, User . . protects from multiple processes sending conflicting updates for all your meta-data needs
  29. class MeetingsController < ApplicationController def create response = EventSourced::Meeting.new(SecureRandom.uuid).schedule(schedule_meeting_params) if

    response.success? # ... else # ... end end private def schedule_meeting_params params.require(:meeting_form).permit(:time, :organizer_id, :invitee_ids) end [ … ] end 1. create meeting instance 2. invoke schedule command
  30. module EventSourced class Meeting attr_reader :id, :version, :status, :attendees_count, :time

    def initialize(id) @id = id @attendees_count = 0 @version = 0 end def schedule(time:, organizer_id:, invitee_ids:) Commands::Meeting::Schedule.call( self, time: meeting_time, organizer_id: organizer_id, invitee_ids: invitee_ids ) end [ … ] end end delegate to command for detailed validation
  31. module Commands module Meeting ScheduleSchema = Dry::Validation.Schema do required(:time).value(gt?: Time.now

    + 5.minutes) required(:organizer_id).filled required(:invitee_ids).filled { each(:int?) } end class Schedule < Command def call return validate if validate.failure? event = publish_event( event_type: 'scheduled', entity_type: 'Meeting', entity_id: @entity.id, entity_version: @entity.version, data: { time: @params[:time], organizer_id: @params[:organizer_id], invitee_ids: @params[:invitee_ids] } ) @entity.handle(event) Response.new success: true, data: { new_meeting: @entity } end end end end dry-validation gem to check the event data store the event with the appropriate data update the entity
  32. module EventSourced class Meeting [ … ] def scheduled_handler(event) params

    = event.data.symbolize_keys @time = params[:time] end def handle(event) handler = "#{event.event_type}_handler".to_sym if entity.respond_to? handler send(handler, event) @version += 1 end end [ … ] end end the event handler is responsible for updating the in-memory domain object
  33. None
  34. SCHEDULE A MEETING (RECAP)

  35. ACCEPT A MEETING By loading a domain object from the

    event store
  36. ACCEPT A MEETING bring domain object to the latest version

  37. class MeetingsController < ApplicationController [ … ] def accept meeting_id

    = accept_meeting_params[:meeting_id] response = EventSourced::Meeting .get(meeting_id) .accept(user_id: accept_meeting_params[:user_id]) flash[:alert] = "Failed to accept the meeting"if response.failure? redirect_to meeting_path(meeting_id) end private [ … ] def accept_meeting_params params.require(:accept_form).permit(:meeting_id, :user_id) end end get the entity and issue the command
  38. module EventSourced class Meeting [ … ] def handle(event) handler

    = "#{event.event_type}_handler".to_sym if entity.respond_to? handler send(handler, event) @version += 1 end end class << self def get(id) entity = new(id) load_events_for(entity) entity end def load_events_for(entity) events = Event.where(entity_type: 'meeting', entity_id: entity.id) events.each { |event| entity.handle(event) } end end end end send event to handler and increment version load events from the store and applies them to the given entity
  39. WHERE ARE MY ACTIVE RECORD MODELS? (we are de-railing from

    the conventions)
  40. WHERE ARE MY ACTIVE RECORD MODELS? • They become read-optimised

    views of your data • The single source of truth is the event log and your relational tables are a view on that log • Pro tip: you can have as many views as you want (think of reporting, search, analytics)
  41. WHERE ARE MY ACTIVE RECORD MODELS?

  42. class Meeting < ApplicationRecord has_one :organizer has_many :invitees class <<

    self def scheduled_handler(event) params = event.data.symbolize_keys create!( entity_id: event.entity_id, time: params.fetch(:time), organizer_id: params.fetch(:organizer_id), created_at: event.created_at ) end def rescheduled_handler(event) meeting = find(event.entity_id) params = event.data.symbolize_keys meeting.update_attribute(time: params.fetch(:new_time)) end [ … ] end end create a row in a relational table when a scheduled event is received update an existing entry when a rescheduled event is received
  43. A NICE CONSEQUENCE

  44. IN SHORT • Event Sourcing is GIT applied to your

    data • You invoke commands to write events • Events become the single source of truth of your data • Your old active record models become your read- optimised views
  45. BENEFITS • Read schemas are decoupled from event log •

    Auditability / Why state is the way it is • Simple way to synchronise data between micro services
  46. DRAWBACKS • More complex • You will have to think

    about eventual consistency • You will have to think about idempotency • You de-rail from the Rails conventions
  47. None
  48. EVENT SOURCING IN RUBY • Beyond the current state: Time

    travel to the rescue!, Armin Pašalić, wroc_love.rb 2018 • Event Sourcing: A Rails Case Study, Francis Hwang, NYC.rb 2015 • https://github.com/fhwang/event_sourced_record • https://github.com/RailsEventStore/rails_event_store • https://github.com/Sandthorn/sandthorn • https://github.com/zilverline/sequent
  49. EVENT SOURCING REFERENCES • The Many Meanings of Event-Driven Architecture,

    Martin Fowler, GOTO 2017 • Event Sourcing, Martin Fowler, YOW! Nights 2016 • Implementing Domain Driven Design, Vaugh Vernon, 2013 • Greg Young Event DDD, CQRS, Event sourcing course • http://subscriptions.viddler.com/GregYoung/ • Promo code: LondonRuby for a $150 off • And a few limited 99% discount (grab me after the presentation)
  50. THANK YOU! QUESTIONS?

  51. APPENDIX

  52. EXTERNAL SYSTEMS • When issuing a command that requires a

    change to an external system • First call the external system (ideally inside the command) • If successful, then record the event
  53. IDEMPOTENCY • Any handler with side-effects should be turned off

    when rebuilding your projections • Emails • Analytics tools (Google Analytics / Mixpanel / Kissmetrics) • In general, any API call to an external system that is done as a result of an event
  54. SCHEMA VERSIONING • You can’t easily change event names •

    You can’t easily change the event schema • Depending on your tolerance you can use JSON / Protocol buffers / AVRO
  55. module EventSourced class Meeting def accept(user_id:) Commands::Meeting::Accept.call(self, user_id: user_id, status:

    @status) end def accepted_handler(event) @attendees_count += 1 end def reschedule(new_time:) Commands::Meeting::Reschedule.call(self, new_time: new_time, status: @status) end def rescheduled_handler(event) params = event.data.symbolize_keys @time = params[:new_time] end def finish Commands::Meeting::Finish.call(self, status: @status) end def finished_handler(event) @status = :finished end def cancel Commands::Meeting::Cancel.call(self, status: @status) end def cancelled_handler(event) @status = :cancelled end end end command command command event handler event handler event handler command event handler
  56. module Commands class Command def self.call(entity, params) new(entity, params).call end

    def initialize(entity, params) @entity = entity @params = params @validator = schema.call(params) end def publish_event(params) Event.create_and_broadcast(params) end def validate @validator.success? ? Response.new(success: true) : Response.new(success: false, errors: @validator.errors) end def schema "#{self.class.name}Schema".constantize end end end
  57. module Commands module Meeting AcceptSchema = Dry::Validation.Schema do required(:user_id).filled required(:status).value(eql?:

    :scheduled) end class Accept < Command def call return validate if validate.failure? publish_event( event_type: 'accepted', entity_type: 'Meeting', entity_id: @entity.id, entity_version: @entity.version, data: { user_id: @params[:user_id] } ) @entity.handle(event) Response.new success: true end end end end