An Introduction to Event Sourcing for Rubyists

Event sourcing is a design pattern to build applications that are domain centric and easy to extend. The pattern is based on the usage of a persistent event log which substitutes the more classical relational database model for Rails applications.

Such event log can then be used for extending your application in all sort of creative ways. For example, by synchronizing data between your microservices, trigger side effects without cluttering your models or controllers, or build data views optimized for your query needs. In this talk, I'll present the basic ideas, some of the tradeoffs and challenges you might find and how you could start experimenting with it.

Transcript

1. AN INTRODUCTION TO
   EVENT SOURCING
   Alfredo Motta @mottalrd
   London Ruby User Group, May 9th 2018
   

   View Slide

2. CREDITS
   @tadejm
   @dncrht @creditspringhq
   @krule
   @gregyoung
   #DDDx
   @krisleech
   

   View Slide

3. MY JOURNEY SO FAR
   • PhD in Software Engineering (UML & formal methods)
   • Startup launching a second-hand market for daily deals (4 ppl)
   • Startup growing a holiday rentals website (100+ ppl)
   • P2P lender launching new products for car finance (200+ ppl)
   • Startup building a safer way to borrow (8 ppl)
   

   View Slide

4. WE HAVE A CRUD PROBLEM
   (and event sourcing might be your cure)
   

   View Slide

5. CRUD WEB APPLICATIONS
   

   View Slide

6. View Slide

7. WHY WE OVERWRITE THE
   DATA?
   

   View Slide

8. WHY WE OVERWRITE THE
   DATA?
   

   View Slide

9. WHY WE READ AND WRITE
   FROM THE SAME TABLE?
   

   View Slide

10. WHY WE READ AND WRITE
    FROM THE SAME TABLE?
    • Because it’s the simplest thing that makes sense?
    • Because it is the Rails convention?
    • Because we want to read our own writes?
    • Because we want to take advantages of database
    locks?
    

    View Slide

11. WHY WE READ AND WRITE
    FROM THE SAME TABLE?
    time window
    with stale
    data
    

    View Slide

12. A LITTLE SECRET
    (it is ok for data to be stale)
    

    View Slide

13. A LITTLE SECRET
    • Business has been dealing with stale data forever
    • Overbooking of Hotel/Flights still happens today
    • What if some, or even all, of your data doesn't
    need to be consistent all the time?
    

    View Slide

14. LET’S CHALLENGE THESE
    ASSUMPTIONS
    

    View Slide

15. EVENT SOURCING
    PERSPECTIVE
    • Track changes, not state
    • Write events, read from eventually consistent
    projections of these events
    • Focus on immutability and idempotency
    

    View Slide

16. • GIT store your current state as a
    change-log
    • i.e. a list of commits that
    represent your current state
    • it is ok if you and your co-
    worker does not have the same
    db at the same time
    GIT
    • Accountants never overwrite data
    • they add a new entry with the
    latest change in your account
    • they can always explain the
    reason behind your account
    balance
    Accountants
    

    View Slide

17. You take the blue pill, the story ends. You wake up in your bed and believe
    whatever you want to believe.
    You take the red pill, you stay in Wonderland and I show you how deep
    the rabbit hole goes
    

    View Slide

18. –Martin Fowler
    “Capture all changes to an application state as a
    sequence of events.”
    –Greg Young
    “A system where an Event Store is used to rebuild an
    object of the domain as opposed to something storing
    the current state.”
    EVENT SOURCING
    

    View Slide

19. BEYOND CRUD
    

    View Slide

20. BUILDING BLOCKS
    Events happen in
    the past
    email_updated email_updated
    at
    2018-05-04
    08:39:34 +0100
    email_updated
    at
    2018-05-04
    08:39:34 +0100
    for
    user with ID 99
    with
    a@a.com
    email_updated
    at
    2018-05-04
    08:39:34 +0100
    for
    user with ID 99
    Events are
    immutable
    Events belongs to
    models ***
    Events have a
    payload storing
    what happened
    *** more precisely events belong to aggregates, aggregates define consistency boundaries for groups of related entities.
    

    View Slide

21. BENEFITS
    • Read schemas are decoupled from event log
    • Auditability / why state is the way it is
    • Simple way to synchronise data between micro
    services
    

    View Slide

22. EXAMPLE USE CASE
    A meeting scheduling applications
    

    View Slide

23. A SIMPLE MEETING MODEL
    

    View Slide

24. SCHEDULE A MEETING
    a command is
    a request for
    an event
    creation
    a successful
    command
    generates an
    event
    entity_id entity_type event_type data
    1 Meeting scheduled
    { organizer_id: 1, invitee_ids: [2, 3],
    time: 2018-05-05 09:07:18 +0100 }
    

    View Slide

25. SCHEDULE A MEETING
    (REJECTED)
    

    View Slide

26. –Linus Torvalds
    “Talk is cheap, show me the code.”
    https://goo.gl/zREBmb
    

    View Slide

27. EVENT SCHEMA
    (SIMPLE)
    create_table 'events', id: :serial do |t|
    t.uuid 'entity_id', null: false, index: true
    t.string 'event_type', null: false
    t.jsonb 'data', null: false
    t.datetime 'created_at', null: false
    end
    email_updated
    id of the user
    { new_email: a@a.com }
    .
    .
    

    View Slide

28. EVENT SCHEMA
    (COMPLETE)
    create_table 'events', id: :serial do |t|
    t.uuid 'entity_id', null: false, index: true
    t.string 'entity_type', null: false
    t.string 'event_type', null: false
    t.jsonb 'meta', null: false
    t.jsonb 'data', null: false
    t.datetime 'created_at', null: false
    t.int 'entity_version', null: false
    t.index ['entity_id', 'entity_version'], name:
    'event_version_constraint', unique: true
    end
    for example, User
    .
    . protects from multiple
    processes sending
    conflicting updates
    for all your meta-data needs
    

    View Slide

29. class MeetingsController < ApplicationController
    def create
    response = EventSourced::Meeting.new(SecureRandom.uuid).schedule(schedule_meeting_params)
    if response.success?
    # ...
    else
    # ...
    end
    end
    private
    def schedule_meeting_params
    params.require(:meeting_form).permit(:time, :organizer_id, :invitee_ids)
    end
    [ … ]
    end
    1. create meeting
    instance
    2. invoke schedule
    command
    

    View Slide

30. module EventSourced
    class Meeting
    attr_reader :id, :version, :status, :attendees_count, :time
    def initialize(id)
    @id = id
    @attendees_count = 0
    @version = 0
    end
    def schedule(time:, organizer_id:, invitee_ids:)
    Commands::Meeting::Schedule.call(
    self,
    time: meeting_time,
    organizer_id: organizer_id,
    invitee_ids: invitee_ids
    )
    end
    [ … ]
    end
    end
    delegate to
    command for
    detailed
    validation
    

    View Slide

31. module Commands
    module Meeting
    ScheduleSchema = Dry::Validation.Schema do
    required(:time).value(gt?: Time.now + 5.minutes)
    required(:organizer_id).filled
    required(:invitee_ids).filled { each(:int?) }
    end
    class Schedule < Command
    def call
    return validate if validate.failure?
    event = publish_event(
    event_type: 'scheduled',
    entity_type: 'Meeting',
    entity_id: @entity.id,
    entity_version: @entity.version,
    data: {
    time: @params[:time],
    organizer_id: @params[:organizer_id],
    invitee_ids: @params[:invitee_ids]
    }
    )
    @entity.handle(event)
    Response.new success: true, data: { new_meeting: @entity }
    end
    end
    end
    end
    dry-validation gem to
    check the event data
    store the event with the
    appropriate data
    update the entity
    

    View Slide

32. module EventSourced
    class Meeting
    [ … ]
    def scheduled_handler(event)
    params = event.data.symbolize_keys
    @time = params[:time]
    end
    def handle(event)
    handler = "#{event.event_type}_handler".to_sym
    if entity.respond_to? handler
    send(handler, event)
    @version += 1
    end
    end
    [ … ]
    end
    end
    the event handler is responsible for
    updating the in-memory domain
    object
    

    View Slide

33. View Slide

34. SCHEDULE A MEETING
    (RECAP)
    

    View Slide

35. ACCEPT A MEETING
    By loading a domain object from the event store
    

    View Slide

36. ACCEPT A MEETING
    bring domain object
    to the latest version
    

    View Slide

37. class MeetingsController < ApplicationController
    [ … ]
    def accept
    meeting_id = accept_meeting_params[:meeting_id]
    response = EventSourced::Meeting
    .get(meeting_id)
    .accept(user_id: accept_meeting_params[:user_id])
    flash[:alert] = "Failed to accept the meeting"if response.failure?
    redirect_to meeting_path(meeting_id)
    end
    private
    [ … ]
    def accept_meeting_params
    params.require(:accept_form).permit(:meeting_id, :user_id)
    end
    end
    get the entity and
    issue the command
    

    View Slide

38. module EventSourced
    class Meeting
    [ … ]
    def handle(event)
    handler = "#{event.event_type}_handler".to_sym
    if entity.respond_to? handler
    send(handler, event)
    @version += 1
    end
    end
    class << self
    def get(id)
    entity = new(id)
    load_events_for(entity)
    entity
    end
    def load_events_for(entity)
    events = Event.where(entity_type: 'meeting', entity_id: entity.id)
    events.each { |event| entity.handle(event) }
    end
    end
    end
    end
    send event to
    handler and
    increment version
    load events from the store
    and applies them to the
    given entity
    

    View Slide

39. WHERE ARE MY ACTIVE
    RECORD MODELS?
    (we are de-railing from the conventions)
    

    View Slide

40. WHERE ARE MY ACTIVE
    RECORD MODELS?
    • They become read-optimised views of your data
    • The single source of truth is the event log and
    your relational tables are a view on that log
    • Pro tip: you can have as many views as you want
    (think of reporting, search, analytics)
    

    View Slide

41. WHERE ARE MY ACTIVE
    RECORD MODELS?
    

    View Slide

42. class Meeting < ApplicationRecord
    has_one :organizer
    has_many :invitees
    class << self
    def scheduled_handler(event)
    params = event.data.symbolize_keys
    create!(
    entity_id: event.entity_id,
    time: params.fetch(:time),
    organizer_id: params.fetch(:organizer_id),
    created_at: event.created_at
    )
    end
    def rescheduled_handler(event)
    meeting = find(event.entity_id)
    params = event.data.symbolize_keys
    meeting.update_attribute(time: params.fetch(:new_time))
    end
    [ … ]
    end
    end
    create a row in a
    relational table when
    a scheduled event is
    received
    update an existing
    entry when a
    rescheduled event is
    received
    

    View Slide

43. A NICE CONSEQUENCE
    

    View Slide

44. IN SHORT
    • Event Sourcing is GIT applied to your data
    • You invoke commands to write events
    • Events become the single source of truth of your
    data
    • Your old active record models become your read-
    optimised views
    

    View Slide

45. BENEFITS
    • Read schemas are decoupled from event log
    • Auditability / Why state is the way it is
    • Simple way to synchronise data between micro
    services
    

    View Slide

46. DRAWBACKS
    • More complex
    • You will have to think about eventual consistency
    • You will have to think about idempotency
    • You de-rail from the Rails conventions
    

    View Slide

47. View Slide

48. EVENT SOURCING IN RUBY
    • Beyond the current state: Time travel to the rescue!, Armin Pašalić,
    wroc_love.rb 2018
    • Event Sourcing: A Rails Case Study, Francis Hwang, NYC.rb 2015
    • https://github.com/fhwang/event_sourced_record
    • https://github.com/RailsEventStore/rails_event_store
    • https://github.com/Sandthorn/sandthorn
    • https://github.com/zilverline/sequent
    

    View Slide

49. EVENT SOURCING
    REFERENCES
    • The Many Meanings of Event-Driven Architecture, Martin Fowler, GOTO 2017
    • Event Sourcing, Martin Fowler, YOW! Nights 2016
    • Implementing Domain Driven Design, Vaugh Vernon, 2013
    • Greg Young Event DDD, CQRS, Event sourcing course
    • http://subscriptions.viddler.com/GregYoung/
    • Promo code: LondonRuby for a $150 off
    • And a few limited 99% discount (grab me after the presentation)
    

    View Slide

50. THANK YOU!
    QUESTIONS?
    

    View Slide

51. APPENDIX
    

    View Slide

52. EXTERNAL SYSTEMS
    • When issuing a command that requires a change
    to an external system
    • First call the external system (ideally inside the
    command)
    • If successful, then record the event
    

    View Slide

53. IDEMPOTENCY
    • Any handler with side-effects should be turned off when
    rebuilding your projections
    • Emails
    • Analytics tools (Google Analytics / Mixpanel /
    Kissmetrics)
    • In general, any API call to an external system that is done
    as a result of an event
    

    View Slide

54. SCHEMA VERSIONING
    • You can’t easily change event names
    • You can’t easily change the event schema
    • Depending on your tolerance you can use JSON /
    Protocol buffers / AVRO
    

    View Slide

55. module EventSourced
    class Meeting
    def accept(user_id:)
    Commands::Meeting::Accept.call(self, user_id: user_id, status: @status)
    end
    def accepted_handler(event)
    @attendees_count += 1
    end
    def reschedule(new_time:)
    Commands::Meeting::Reschedule.call(self, new_time: new_time, status: @status)
    end
    def rescheduled_handler(event)
    params = event.data.symbolize_keys
    @time = params[:new_time]
    end
    def finish
    Commands::Meeting::Finish.call(self, status: @status)
    end
    def finished_handler(event)
    @status = :finished
    end
    def cancel
    Commands::Meeting::Cancel.call(self, status: @status)
    end
    def cancelled_handler(event)
    @status = :cancelled
    end
    end
    end
    command
    command
    command
    event handler
    event handler
    event handler
    command
    event handler
    

    View Slide

56. module Commands
    class Command
    def self.call(entity, params)
    new(entity, params).call
    end
    def initialize(entity, params)
    @entity = entity
    @params = params
    @validator = schema.call(params)
    end
    def publish_event(params)
    Event.create_and_broadcast(params)
    end
    def validate
    @validator.success? ? Response.new(success: true) : Response.new(success: false, errors:
    @validator.errors)
    end
    def schema
    "#{self.class.name}Schema".constantize
    end
    end
    end
    

    View Slide

57. module Commands
    module Meeting
    AcceptSchema = Dry::Validation.Schema do
    required(:user_id).filled
    required(:status).value(eql?: :scheduled)
    end
    class Accept < Command
    def call
    return validate if validate.failure?
    publish_event(
    event_type: 'accepted',
    entity_type: 'Meeting',
    entity_id: @entity.id,
    entity_version: @entity.version,
    data: { user_id: @params[:user_id] }
    )
    @entity.handle(event)
    Response.new success: true
    end
    end
    end
    end
    

    View Slide