Life Beyond Relational Database

Introduction to event sourcing in Haskell, in the context of Capital Match

Transcript

1. Life Beyond Relational Database
   Capital Match Team
   2016-03-10
   

   View Slide

2. Agenda
   Introduc on
   Event‐Sourcing Model
   Implementa on & Usage
   Future works
   

   View Slide

3. Introduction
   

   View Slide

4. Who are we?
   

   View Slide

5. Who are we?
   Capital Match is the leading plaform in Singapore for peer‐
   to‐peer lending to SMEs
   Backend system developed in Haskell, frontend in
   Clojurescript/Om since 2014
   Core Development team of 3 + 1: Amar, Arnaud, Guo
   Liang, Zhou Yu
   

   View Slide

6. Relational Model
   

   View Slide

7. View Slide

8. What's good with Relational Model?
   Really great for querying  →  SQL Rocks!
   Conceptually simple to understand: Everything is a Table
   Ubiquitous
   

   View Slide

9. What's wrong with Relational Model?
   Writes/updates are complex
   Impedance Mismatch: Lot of data is more tree‐ish or graph‐
   ish
   One single Database for everything  →  SPOF
   Mutable State
   

   View Slide

10. View Slide

11. Event Sourcing
    

    View Slide

12. State vs. Transitions
    

    View Slide

13. State vs. Transitions
    RDBMS stores the state of the model at some point in
    me...
    ... But we are also interested in the transi ons ...
    ... And state can always be reconstructed from a sequence
    of transi ons.
    1
    

    View Slide

14. The Event Sourcing Model
    Event Sourcing ensures that all changes to
    applica on state are stored as a sequence of
    events. Not just can we query these events,
    we can also use the event log to reconstruct
    past states, and as a founda on to
    automa cally adjust the state to cope with
    retroac ve changes.
    Mar n Fowler
    

    View Slide

15. Events makes it easier to...
    Audit current state and what lead to it
    Implement generic undo/redo mechanism
    Run simula ons with different hypothesis over live data
    Cope with data format migra ons
    Handle poten ally conflic ng changes
    2
    3
    

    View Slide

16. Events Drive Business
    Events are what makes a model dynamic: What affects it,
    how it reacts to outside world...
    Provide founda on for techniques
     →  Be er business models, Ubiquitous language
    Lead to technique for "requirements"
    elicita on and business domain modelling
    Domain Driven Design
    Event Storming
    4
    

    View Slide

17. In Practice
    

    View Slide

18. Overview
    

    View Slide

19. Pure Business Models
    Each model delimits a Bounded Context: It is responsible
    for a single cohesive part of the domain
    Models are pure immutable data structures
    Dis nguish Commands from Events
    

    View Slide

20. Pure Business Models (2)
    Commands compute Event from State
    Events modify model
    a
    c
    t :
    : C
    o
    m
    m
    a
    n
    d -
    > M
    o
    d
    e
    l -
    > E
    v
    e
    n
    t
    a
    p
    p
    l
    y :
    : E
    v
    e
    n
    t -
    > M
    o
    d
    e
    l -
    > M
    o
    d
    e
    l
    

    View Slide

21. E ectful Services
    Services are used to orchestrate interac on
    between one or more business models and
    the outside world
    Services are func ons opera ng across several contexts
    They can be synchronous or asynchronous (we use mostly
    synchronous)
    There are no distributed transac ons: Service has to cope
    with failures from each context
    5
    

    View Slide

22. E ectful Services (2)
    We have a monad to express effects and sequencing on
    each context: W
    e
    b
    S
    t
    a
    t
    e
    M
    g is a "global" Model which can be accessed concurrently  
    →  protected in STM
    l is local data, contextual to a single service execu on
    m is underlying monad, usually I
    O
    n
    e
    w
    t
    y
    p
    e W
    e
    b
    S
    t
    a
    t
    e
    M g l m a = W
    e
    b
    S
    t
    a
    t
    e
    M { r
    u
    n
    W
    e
    b
    M :
    : T
    V
    a
    r g -
    > l -
    > m a }
    

    View Slide

23. Events Storage
    d
    a
    t
    a S
    t
    o
    r
    e
    d
    E
    v
    e
    n
    t s = S
    t
    o
    r
    e
    d
    E
    v
    e
    n
    t { e
    v
    e
    n
    t
    V
    e
    r
    s
    i
    o
    n :
    : E
    v
    e
    n
    t
    V
    e
    r
    s
    i
    o
    n
    , e
    v
    e
    n
    t
    T
    y
    p
    e :
    : E
    v
    e
    n
    t
    T
    y
    p
    e s
    , e
    v
    e
    n
    t
    D
    a
    t
    e :
    : D
    a
    t
    e
    , e
    v
    e
    n
    t
    U
    s
    e
    r :
    : U
    s
    e
    r
    I
    d
    , e
    v
    e
    n
    t
    R
    e
    q
    u
    e
    s
    t :
    : E
    n
    c
    o
    d
    e
    d H
    e
    x
    , e
    v
    e
    n
    t
    S
    H
    A
    1 :
    : E
    n
    c
    o
    d
    e
    d H
    e
    x
    , e
    v
    e
    n
    t :
    : B
    y
    t
    e
    S
    t
    r
    i
    n
    g
    }
    

    View Slide

24. Events Storage (2)
    We use a simple Append‐only file store, wri ng serialized
    events (mostly JSON) packed with metadata
    Each event has a (monotonically increasing) version which
    is used for proper deserializa on
    Events carry useful informa on for troubleshoo ng and
    audi ng: User who ini ated the request, request id itself,
    SHA1 represen ng version of appplica on
    Events Store serializes concurrent writes
    

    View Slide

25. Software
    

    View Slide

26. View Slide

27. In Prac ce
    

    View Slide

28. Demo
    Anatomy of a complete business model
    Web layer w/ servant
    Service layer (w/ Free monads...)
    Business model
    Migra on code
    Standalone service
    Using Haskell scripts for opera onal queries and updates
    

    View Slide

29. Future Works
    

    View Slide

30. View Slide

31. Implement Better CQRS
    Separate Read Model from Write Model
    Write Model: Append‐only linear data store per context,
    very fast, minimize locking/write me
    Read model: Op mized for specific querying, may be
    rela onal if needed in order to make it more user‐friendly
    

    View Slide

32. Make models resilient
    Resilience of models  →  Replica on
    Use to maintain strong consistency of models:
    Haskell
    Started implementa on of prac cal cluster based on Ra ,
    called
    Ra several
    implementa ons in
    raptr
    

    View Slide

33. Make models secure
    Turn event stream into a source of truth  →  Blockchain
    and beyond...
    Juno: over Ra cluster
    Uses cryptographically signed events to ensure history
    cannot be tampered with
    Turns journal into a "legally binding ledger"?
    6
    Smart contracts
    

    View Slide

34. Questions?
    

    View Slide

35. View Slide

36. Credits
    

    View Slide

37. View Slide

38. 1. Assuming state is determinis c of course
    2. May require inver ble events
    3. That's the way RDBMS handle transac onal isola on:
    Record a log of all opera ons on data then reconcile when
    transac ons are commi ed
    4. I never know how many l
    s modelling takes...
    5. Synchronicity is a property of the business domain, e.g.
    depends on what client expects from the service and
    whether or not he wants to "pay" for synchronous
    confirma on
    6. Blockchain is all rage in the FinTech ecosystem those days,
    although early implementa on like Bitcoins or Dogecoins
    failed to deliver all their promises.
    ↩
    ↩
    ↩
    ↩
    ↩
    ↩
    

    View Slide