Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Life Beyond Relational Database

Life Beyond Relational Database

Introduction to event sourcing in Haskell, in the context of Capital Match

Arnaud Bailly

March 10, 2016
Tweet

More Decks by Arnaud Bailly

Other Decks in Technology

Transcript

  1. Life Beyond Relational Database
    Capital Match Team
    2016-03-10

    View full-size slide

  2. Agenda
    Introduc on
    Event‐Sourcing Model
    Implementa on & Usage
    Future works

    View full-size slide

  3. Introduction

    View full-size slide

  4. Who are we?
    Capital Match is the leading plaform in Singapore for peer‐
    to‐peer lending to SMEs
    Backend system developed in Haskell, frontend in
    Clojurescript/Om since 2014
    Core Development team of 3 + 1: Amar, Arnaud, Guo
    Liang, Zhou Yu

    View full-size slide

  5. Relational Model

    View full-size slide

  6. What's good with Relational Model?
    Really great for querying  →  SQL Rocks!
    Conceptually simple to understand: Everything is a Table
    Ubiquitous

    View full-size slide

  7. What's wrong with Relational Model?
    Writes/updates are complex
    Impedance Mismatch: Lot of data is more tree‐ish or graph‐
    ish
    One single Database for everything  →  SPOF
    Mutable State

    View full-size slide

  8. Event Sourcing

    View full-size slide

  9. State vs. Transitions

    View full-size slide

  10. State vs. Transitions
    RDBMS stores the state of the model at some point in
    me...
    ... But we are also interested in the transi ons ...
    ... And state can always be reconstructed from a sequence
    of transi ons.
    1

    View full-size slide

  11. The Event Sourcing Model
    Event Sourcing ensures that all changes to
    applica on state are stored as a sequence of
    events. Not just can we query these events,
    we can also use the event log to reconstruct
    past states, and as a founda on to
    automa cally adjust the state to cope with
    retroac ve changes.
    Mar n Fowler

    View full-size slide

  12. Events makes it easier to...
    Audit current state and what lead to it
    Implement generic undo/redo mechanism
    Run simula ons with different hypothesis over live data
    Cope with data format migra ons
    Handle poten ally conflic ng changes
    2
    3

    View full-size slide

  13. Events Drive Business
    Events are what makes a model dynamic: What affects it,
    how it reacts to outside world...
    Provide founda on for techniques
     →  Be er business models, Ubiquitous language
    Lead to technique for "requirements"
    elicita on and business domain modelling
    Domain Driven Design
    Event Storming
    4

    View full-size slide

  14. Pure Business Models
    Each model delimits a Bounded Context: It is responsible
    for a single cohesive part of the domain
    Models are pure immutable data structures
    Dis nguish Commands from Events

    View full-size slide

  15. Pure Business Models (2)
    Commands compute Event from State
    Events modify model
    a
    c
    t :
    : C
    o
    m
    m
    a
    n
    d -
    > M
    o
    d
    e
    l -
    > E
    v
    e
    n
    t
    a
    p
    p
    l
    y :
    : E
    v
    e
    n
    t -
    > M
    o
    d
    e
    l -
    > M
    o
    d
    e
    l

    View full-size slide

  16. E ectful Services
    Services are used to orchestrate interac on
    between one or more business models and
    the outside world
    Services are func ons opera ng across several contexts
    They can be synchronous or asynchronous (we use mostly
    synchronous)
    There are no distributed transac ons: Service has to cope
    with failures from each context
    5

    View full-size slide

  17. E ectful Services (2)
    We have a monad to express effects and sequencing on
    each context: W
    e
    b
    S
    t
    a
    t
    e
    M
    g is a "global" Model which can be accessed concurrently  
    →  protected in STM
    l is local data, contextual to a single service execu on
    m is underlying monad, usually I
    O
    n
    e
    w
    t
    y
    p
    e W
    e
    b
    S
    t
    a
    t
    e
    M g l m a = W
    e
    b
    S
    t
    a
    t
    e
    M { r
    u
    n
    W
    e
    b
    M :
    : T
    V
    a
    r g -
    > l -
    > m a }

    View full-size slide

  18. Events Storage
    d
    a
    t
    a S
    t
    o
    r
    e
    d
    E
    v
    e
    n
    t s = S
    t
    o
    r
    e
    d
    E
    v
    e
    n
    t { e
    v
    e
    n
    t
    V
    e
    r
    s
    i
    o
    n :
    : E
    v
    e
    n
    t
    V
    e
    r
    s
    i
    o
    n
    , e
    v
    e
    n
    t
    T
    y
    p
    e :
    : E
    v
    e
    n
    t
    T
    y
    p
    e s
    , e
    v
    e
    n
    t
    D
    a
    t
    e :
    : D
    a
    t
    e
    , e
    v
    e
    n
    t
    U
    s
    e
    r :
    : U
    s
    e
    r
    I
    d
    , e
    v
    e
    n
    t
    R
    e
    q
    u
    e
    s
    t :
    : E
    n
    c
    o
    d
    e
    d H
    e
    x
    , e
    v
    e
    n
    t
    S
    H
    A
    1 :
    : E
    n
    c
    o
    d
    e
    d H
    e
    x
    , e
    v
    e
    n
    t :
    : B
    y
    t
    e
    S
    t
    r
    i
    n
    g
    }

    View full-size slide

  19. Events Storage (2)
    We use a simple Append‐only file store, wri ng serialized
    events (mostly JSON) packed with metadata
    Each event has a (monotonically increasing) version which
    is used for proper deserializa on
    Events carry useful informa on for troubleshoo ng and
    audi ng: User who ini ated the request, request id itself,
    SHA1 represen ng version of appplica on
    Events Store serializes concurrent writes

    View full-size slide

  20. Demo
    Anatomy of a complete business model
    Web layer w/ servant
    Service layer (w/ Free monads...)
    Business model
    Migra on code
    Standalone service
    Using Haskell scripts for opera onal queries and updates

    View full-size slide

  21. Future Works

    View full-size slide

  22. Implement Better CQRS
    Separate Read Model from Write Model
    Write Model: Append‐only linear data store per context,
    very fast, minimize locking/write me
    Read model: Op mized for specific querying, may be
    rela onal if needed in order to make it more user‐friendly

    View full-size slide

  23. Make models resilient
    Resilience of models  →  Replica on
    Use to maintain strong consistency of models:
    Haskell
    Started implementa on of prac cal cluster based on Ra ,
    called
    Ra several
    implementa ons in
    raptr

    View full-size slide

  24. Make models secure
    Turn event stream into a source of truth  →  Blockchain
    and beyond...
    Juno: over Ra cluster
    Uses cryptographically signed events to ensure history
    cannot be tampered with
    Turns journal into a "legally binding ledger"?
    6
    Smart contracts

    View full-size slide

  25. 1. Assuming state is determinis c of course
    2. May require inver ble events
    3. That's the way RDBMS handle transac onal isola on:
    Record a log of all opera ons on data then reconcile when
    transac ons are commi ed
    4. I never know how many l
    s modelling takes...
    5. Synchronicity is a property of the business domain, e.g.
    depends on what client expects from the service and
    whether or not he wants to "pay" for synchronous
    confirma on
    6. Blockchain is all rage in the FinTech ecosystem those days,
    although early implementa on like Bitcoins or Dogecoins
    failed to deliver all their promises.






    View full-size slide