Life Beyond Relational Database

Introduction to event sourcing in Haskell, in the context of Capital Match

Transcript

1. Life Beyond Relational Database Capital Match Team 2016-03-10
2. Agenda Introduc on Event‐Sourcing Model Implementa on & Usage Future works
3. Introduction
4. Who are we?
5. Who are we? Capital Match is the leading plaform in Singapore for peer‐ to‐peer lending to SMEs Backend system developed in Haskell, frontend in Clojurescript/Om since 2014 Core Development team of 3 + 1: Amar, Arnaud, Guo Liang, Zhou Yu
6. Relational Model
8. What's good with Relational Model? Really great for querying  →  SQL Rocks! Conceptually simple to understand: Everything is a Table Ubiquitous
9. What's wrong with Relational Model? Writes/updates are complex Impedance Mismatch: Lot of data is more tree‐ish or graph‐ ish One single Database for everything  →  SPOF Mutable State
11. Event Sourcing
12. State vs. Transitions
13. State vs. Transitions RDBMS stores the state of the model at some point in me... ... But we are also interested in the transi ons ... ... And state can always be reconstructed from a sequence of transi ons. 1
14. The Event Sourcing Model Event Sourcing ensures that all changes to applica on state are stored as a sequence of events. Not just can we query these events, we can also use the event log to reconstruct past states, and as a founda on to automa cally adjust the state to cope with retroac ve changes. Mar n Fowler
15. Events makes it easier to... Audit current state and what lead to it Implement generic undo/redo mechanism Run simula ons with different hypothesis over live data Cope with data format migra ons Handle poten ally conflic ng changes 2 3
16. Events Drive Business Events are what makes a model dynamic: What affects it, how it reacts to outside world... Provide founda on for techniques  →  Be er business models, Ubiquitous language Lead to technique for "requirements" elicita on and business domain modelling Domain Driven Design Event Storming 4
17. In Practice
18. Overview
19. Pure Business Models Each model delimits a Bounded Context: It is responsible for a single cohesive part of the domain Models are pure immutable data structures Dis nguish Commands from Events
20. Pure Business Models (2) Commands compute Event from State Events modify model a c t : : C o m m a n d - > M o d e l - > E v e n t a p p l y : : E v e n t - > M o d e l - > M o d e l
21. E ectful Services Services are used to orchestrate interac on between one or more business models and the outside world Services are func ons opera ng across several contexts They can be synchronous or asynchronous (we use mostly synchronous) There are no distributed transac ons: Service has to cope with failures from each context 5
22. E ectful Services (2) We have a monad to express effects and sequencing on each context: W e b S t a t e M g is a "global" Model which can be accessed concurrently   →  protected in STM l is local data, contextual to a single service execu on m is underlying monad, usually I O n e w t y p e W e b S t a t e M g l m a = W e b S t a t e M { r u n W e b M : : T V a r g - > l - > m a }
23. Events Storage d a t a S t o r e d E v e n t s = S t o r e d E v e n t { e v e n t V e r s i o n : : E v e n t V e r s i o n , e v e n t T y p e : : E v e n t T y p e s , e v e n t D a t e : : D a t e , e v e n t U s e r : : U s e r I d , e v e n t R e q u e s t : : E n c o d e d H e x , e v e n t S H A 1 : : E n c o d e d H e x , e v e n t : : B y t e S t r i n g }
24. Events Storage (2) We use a simple Append‐only file store, wri ng serialized events (mostly JSON) packed with metadata Each event has a (monotonically increasing) version which is used for proper deserializa on Events carry useful informa on for troubleshoo ng and audi ng: User who ini ated the request, request id itself, SHA1 represen ng version of appplica on Events Store serializes concurrent writes
25. Software
27. In Prac ce
28. Demo Anatomy of a complete business model Web layer w/ servant Service layer (w/ Free monads...) Business model Migra on code Standalone service Using Haskell scripts for opera onal queries and updates
29. Future Works
31. Implement Better CQRS Separate Read Model from Write Model Write Model: Append‐only linear data store per context, very fast, minimize locking/write me Read model: Op mized for specific querying, may be rela onal if needed in order to make it more user‐friendly
32. Make models resilient Resilience of models  →  Replica on Use to maintain strong consistency of models: Haskell Started implementa on of prac cal cluster based on Ra , called Ra several implementa ons in raptr
33. Make models secure Turn event stream into a source of truth  →  Blockchain and beyond... Juno: over Ra cluster Uses cryptographically signed events to ensure history cannot be tampered with Turns journal into a "legally binding ledger"? 6 Smart contracts
34. Questions?
36. Credits
38. 1. Assuming state is determinis c of course 2. May require inver ble events 3. That's the way RDBMS handle transac onal isola on: Record a log of all opera ons on data then reconcile when transac ons are commi ed 4. I never know how many l s modelling takes... 5. Synchronicity is a property of the business domain, e.g. depends on what client expects from the service and whether or not he wants to "pay" for synchronous confirma on 6. Blockchain is all rage in the FinTech ecosystem those days, although early implementa on like Bitcoins or Dogecoins failed to deliver all their promises. ↩ ↩ ↩ ↩ ↩ ↩