Clojure/conj 2015: From REST To CQRS With Clojure, Kafka, & Datomic

Abstract: http://clojure-conj.org/speakers#bcalderwood
Video: https://www.youtube.com/watch?v=qDNPQo9UmJA

Have you ever hit a wall with REST? Does modeling your problem domain into CRUD-able entities feel like fitting a square peg into a round hole? Have you ever tried implementing a PATCH request (without going crazy), or debated PUT vs. POST for resource updates?

An alternative to CRUD-flavored REST is CQRS (Command and Query Responsibility Segregation). In this talk, I'll discuss the advantages and trade-offs of CQRS (and its cousin, Event Sourcing), and demonstrate them using an implementation of CQRS based on Clojure microservices, Kafka, and Datomic. These advantages include:

Operational simplicity, flexibility, and extensibility
Applying Clojure's sound model of state transitions to the resources exposed by your web application
Simplified client implementation
BONUS: We'll even be able to keep the best parts of REST!

I'll also discuss how this technique dovetails nicely with other good ideas in the Clojure community like the design of Datomic and the ideas behind Om.Next.

Transcript

1. From REST to CQRS
   with Clojure, Kafka (and Samza), and Datomic
   

   View Slide

2. Who am I?
   • Member of the Tech Fellows team at
   • Formerly worked on the Product team at Cognitect,
   building and supporting Datomic with Rich, Stu, and
   team
   • Have worked at various startups and large enterprises
   in both business and technical roles
   bobby.calderwood@capitalone.com
   @bobbycalderwood
   https://github.com/bobby
   

   View Slide

3. How to proceed?
   • Define Problem
   • Essential REST
   • CRUD-flavored REST
   • CQRS and Event Sourcing
   • Design
   • Demo
   • Assess benefits and trade-offs
   

   View Slide

4. How not to proceed
   • This talk is NOT a repudiation of REST!
   • https://www.pandastrike.com/posts/20151015-
   rest-vs-relay
   • http://blog.steveklabnik.com/posts/2011-07-03-
   nobody-understands-rest-or-http
   • This talk does NOT describe the only possible
   implementation of CQRS and Event Sourcing via
   HTTP
   

   View Slide

5. Goal
   • Provide valuable informational and transactional
   services via Mobile and Web software
   • To lots of customers
   • With excellent user experience
   • Securely and in compliance with regulations
   • Can easily experiment, enhance, maintain, and
   operate
   

   View Slide

6. Problems
   

   View Slide

7. Organizational Scale
   • Large organization
   • Many product dev teams across several vertical
   LOBs
   • Horizontal dev and ops teams for shared services
   • Currently, deploying a customer-facing application
   involves development of client (mobile or single-
   page JS app) and supporting HTTP API services
   

   View Slide

8. HTTP API Proliferation
   • Some externally-facing, to power customer UIs
   • Some partner-facing
   • Some internally-facing, consuming other APIs
   • DevOps only goes so far to taming this complexity!
   

   View Slide

9. Teams Reinvent Wheels
   • API development:
   • Distributed systems problems: HTTP comms, failure
   modes, timeouts, retries, etc.
   • Auth(n|z), security
   • Audit, regulatory, compliance, metrics, monitoring
   • Database, cache
   • DevOps automation to build, test, deploy, operate
   

   View Slide

10. Teams Reinvent Wheels
    • UI development (JS, iOS, Android):
    • How to talk to various endpoints/APIs
    • Failure modes, transactional semantics,
    granularity of action
    • Optimistic vs. Pessimistic updates
    

    View Slide

11. Teams Reinvent Wheels
    • Library and code-reuse only go so far!!!
    • Internal frameworks and libraries
    • Shared practices and sample code repos
    • Communities of Practice/Excellence/Whathaveyou
    • All add layers of bloat, code generation, misdirection or
    obfuscation => complexity
    • None address the fundamental problem of HTTP API
    proliferation
    

    View Slide

12. Ideal State
    • Simplify UI development for vertical teams by reducing
    number and complexity of HTTP APIs
    • Simplify HTTP API development for vertical teams by not
    doing so much of it, and handling Business Logic
    differently
    • Horizontal teams handle common functionality via shared
    service and infrastructure re-use (i.e. not via code or
    library re-use), Vertical teams experiment on value adds
    • Record clear picture of customer intent and interactions,
    for audit/compliance, replay, etc.
    

    View Slide

13. Essential REST
    

    View Slide

14. The Good Parts
    • Clear communication semantics for big distributed
    systems in low-trust, low-coordination context
    • Proxying/caching of various types of requests over
    unknown network path
    • Clear (enough) error codes for when requests fail (most
    of the time)
    • Loose coupling and robustness principle
    • Sometimes even Conditional requests via Expiration,
    ETags, etc. (but not often)
    

    View Slide

15. The Good Parts
    • Self-describing/documenting for ease of
    consumption, discovery, navigation
    • HTML representation lets humans (possible client
    implementors) learn/discover resources quickly
    • Sometimes even HATEOS (but not often)
    

    View Slide

16. The Good Parts
    • Data-oriented
    • Requests and responses are data, in various
    representation formats, languages, and
    encodings
    • Sometimes even Content Negotiation (but not
    often)
    

    View Slide

17. The Bad Parts
    • “To the person with a hammer…” proliferation
    • Tooling and “pop-culture” around REST leads to
    bad implementations
    

    View Slide

18. CRUD-flavored REST
    

    View Slide

19. The Good Parts
    • My language has tools for that!
    • Java — Jersey/Spring(Boot)/Dropwizard
    • Ruby — Rails/Sinatra
    • JavaScript — Node.js/Express/Loopback
    • Python — Django/Tastypie/Webmachine
    • Clojure — Ring/Pedestal/Liberator/Yada
    • Scala — Play/Lift/Scalatra
    

    View Slide

20. The Good Parts
    • Widely supported and Understood
    • Lots of docs, examples, etc.
    • Easy to hire for
    • On-ramp, guardrails for less experienced
    developers
    • Tools like Swagger, etc. for specifying,
    documenting, validating
    

    View Slide

21. All the cool kids are doing it!
    • It’s Agile™
    • Easy on ramp tooling makes teams feel productive
    vs. straw-man alternatives like SOAP et. al.
    • Easy to get something (poorly specified) into
    “production”
    • Microservices! Containers! DevOps! Cloud!
    • If it’s good enough for NetflixTwitterFacebook…
    

    View Slide

22. The Bad Parts
    

    View Slide

23. All the cool kids are doing it!
    • REST is Over! http://blog.steveklabnik.com/posts/
    2012-02-23-rest-is-over
    

    View Slide

24. Back into the Tarpit
    We use tools designed to eliminate the accidental
    complexity of mutability-centric object orientation in
    the small…
    • Functional languages with referential
    transparency
    • Immutable data structures
    • Immutable databases (Datomic)
    

    View Slide

25. Back into the Tarpit
    

    View Slide

26. Back into the Tarpit
    …in order to implement a thin veneer over mutable
    objects in the large!!!
    

    View Slide

27. Back into the Tarpit
    

    View Slide

28. The S in REST
    • But whereas at least Object Oriented languages
    have clear get/set semantics for attribute updates,
    CRUD-flavored REST has:
    • POST
    • PUT
    • PATCH (?!)
    • Tricky semantics, promote in-place mutability
    

    View Slide

29. The Kingdom of Nouns for
    Distributed Systems
    • Jamming every operation into a CRUD on some resource
    is often unnatural, creating impedance mismatch
    • Complects communication semantics with those of
    application domain
    • I want to (choose one):
    A. “pay my credit-card bill”
    B. “POST a new bill payment sub-resource of the credit-
    card resource”
    

    View Slide

30. The Kingdom of Nouns for
    Distributed Systems
    • Burdens clients with resource orchestration
    • Keep UI in sync via various read endpoints,
    index vs. show granularity, when to follow
    hypermedia
    • Create a composite action comprised of
    behaviors across various write endpoints
    • What about actions that need to transactionally
    modify several different resources at one time?
    

    View Slide

31. The Kingdom of Nouns for
    Distributed Systems
    • Proliferates URL paths and POST/PUT/PATCH/
    DELETE semantics just like Mutable Objects
    proliferate data access methods!
    • Could possibly dispatch domain behavior on
    • HTTP method
    • Header value
    • Path param
    • Query param
    • Some characteristic of Body
    

    View Slide

32. Operationally rigid
    • The following criticisms apply to many out-of-the-
    box implementations and deployment patterns for
    CRUD-flavored REST…
    • Entire API, with all N endpoints often deploy,
    version, change, scale, etc. together
    • Scaling individual resources (from a single
    codebase) requires re-work
    • Scaling reads separate from writes is hard
    

    View Slide

33. Hard to extend or integrate
    /api/v1/resource
    HTTP API
    Database
    Data Storage
    Integrate here? Or here?!?!?
    

    View Slide

34. Obscures the story
    • Typical problems with mutable-state systems
    • What is the client/customer’s actual intent?
    • What happened to bring us to the current state?
    • How do we audit that sequence of events for
    security, compliance, or correctness?
    • Splunk on request logs anyone?
    

    View Slide

35. CQRS and Event
    Sourcing
    

    View Slide

36. What is CQRS?
    • Command (and) Query Responsibility Segregation
    • Separate interface and model for write (command)
    and read (query) paths in an application
    • In REST/HTTP terms, perhaps:
    • /commands for writes
    • /query (pull) and /updates (push, e.g. via WS or
    SSE) for read
    

    View Slide

37. CQRS Baggage
    • CQRS in the small, i.e. within a single codebase
    • For each Entity/Object, have a read class and a write
    class
    • Complexity! Proliferation!
    • CQRS in the large, i.e. architecture of a distributed system
    • At a whole service/enterprise level, have one read
    endpoint and one write endpoint
    • Simplicity! Reduction!
    

    View Slide

38. CQRS Baggage
    • CQRS in the small, i.e. within a single codebase
    • For each Entity/Object, have a read class and a write
    class
    • Complexity! Proliferation!
    • CQRS in the large, i.e. architecture of a distributed system
    • At a whole service/enterprise level, have one read
    endpoint and one write endpoint
    • Simplicity! Reduction!
    

    View Slide

39. CQRS Baggage
    • Object-Oriented CQRS (esp. Java and C#) got all
    tangled up with
    • Domain-Driven Design
    • Aspect Oriented Programming
    • Naked Objects
    • OO “architecture” (again, in the small)
    

    View Slide

40. CQRS Baggage
    • We functional programmers know better!
    • The Language of the System
    • Immutability
    • Values flow through system being transformed
    • Isolate and delay side-effects until the edges
    

    View Slide

41. CQRS via HTTP
    /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    /updates
    WS or SSE
    Clients
    Web Services +
    Business Logic
    Database
    Data Storage
    

    View Slide

42. Why might I use CQRS?
    • Simplicity and de-complecting
    • Operational simplicity and agility
    • Client implementation simplicity and agility
    (Om.Next!)
    

    View Slide

43. What is Event Sourcing?
    • Model and calculate a system’s state by storing all
    events that update that state
    • Often associated with CQRS: store all the events
    that come through command path
    

    View Slide

44. CQRS+ES via HTTP
    /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    /updates
    WS or SSE
    Clients
    Web Services +
    Business Logic
    Indexes
    Event Log
    Data Storage
    

    View Slide

45. Datomic was CQRS+ES
    before it was cool
    • Datomic follows a CQRS design pattern with Event
    Sourcing, whence much of the goodness
    • Reads are cacheable and distributed among peers
    reading from storage
    • Writes are consistent, atomic, sequential, and only
    performed by transactor
    • State of database at time t is (logically) the
    aggregation of all prior events
    

    View Slide

46. Why might I use
    Event Sourcing?
    • Your application models state transitions over time
    • You need to integrate with external systems, or
    extend existing functionality
    • You need a reified, immutable record of
    customer/client intent in domain terms!
    

    View Slide

47. Why might I use
    Event Sourcing?
    /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    /updates
    WS or SSE
    Clients
    Web Services +
    Business Logic
    Indexes
    Event Log
    Data Storage External Processes
    Processes!
    Processes!
    Processes!
    

    View Slide

48. Why might I prefer CRUD-
    flavored REST over CQRS+ES?
    • External requirement or existing SLA for “REST” API
    • Legacy codebase with existing clients
    • Can’t afford additional operational cost
    • Messaging/event-tracking system in addition to a database
    • Microservice cardinality (i.e. deploy, manage, scale three
    endpoints instead of one)
    • Your problem domain truly is resource-oriented and
    CRUD-y
    

    View Slide

49. Design
    

    View Slide

50. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions
    

    View Slide

51. Command
    • A two-tuple of action and data
    1 [:create-customer {:first_name "John"
    2 :last_name "Doe"
    3 :address {:street_number "1234"
    4 :street_name "Main St."
    5 :city "Anytown"
    6 :state "NH"
    7 :zip "03755"}}]
    

    View Slide

52. Query
    • A Datomic datalog query
    1 [:find (pull ?e [*])
    2 :where [?e :customer/id]]
    

    View Slide

53. Update
    • A Websocket message about updated data
    1 [:tx/accepted {:key #uuid "5644d334-c537-40de-901e-5cdb9ede4c51"
    2 :value {:offset "5"
    3 :system "local-kafka"
    4 :stream "pending-txes"
    5 :partition 0
    6 :basis-t 1039}
    7 :topic "accepted-txes"
    8 :partition 0
    9 :offset 3}]
    

    View Slide

54. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions
    

    View Slide

55. 1 (ns cqrs.client
    2 (:require-macros [cljs.core.async.macros :refer [go]])
    3 (:require [cljs-http.client :as http]
    4 [cljs.core.async :as a]
    5 [cognitect.transit :as t]
    6 [taoensso.sente :as sente :refer (cb-success?)]
    7 [taoensso.sente.packers.transit :as sente-transit]))
    8
    9 (defn- request-opts
    10 [opts]
    11 (assoc opts
    12 :channel (a/promise-chan)
    13 :headers {"accept" "application/transit+json"}))
    14
    15 (defn command!
    16 ([action data]
    17 (transact action data false))
    18 ([action data sync?]
    19 (http/post "/api/v1/commands"
    20 (cond-> {:transit-params [action data]}
    21 true request-opts
    22 sync? (assoc :query-params {:sync true})))))
    23
    24 (defn- db-arg?
    25 [o]
    26 (and (map? o)
    27 (or (contains? o :basis-t)
    28 (contains? o :as-of)
    29 (contains? o :history))))
    30
    31 (defn q
    32 [query & args]
    33 (let [[db args] (if (db-arg? (first args))
    34 [(first args) (rest args)]
    35 [nil args])
    36 {:keys [history as-of basis-t]} db
    37 as-of (or as-of basis-t)]
    38 (http/get "/api/v1/query"
    39 (cond-> {:query-params {:query (pr-str query)}}
    40 true request-opts
    41 (seq args) (assoc :args (pr-str args))
    42 as-of (assoc-in [:query-params :as-of] as-of)
    43 history (assoc-in [:query-params :history] history)))))
    44
    45 (defn updates
    46 ([]
    47 (updates
    48 (a/chan 1 (comp (map :event)
    49 (filter #(= :chs/recv (first %)))))))
    50 ([ch]
    51 (-> "/api/v1/updates"
    52 (sente/make-channel-socket!
    53 {:packer (sente-transit/->TransitPacker :json {} {})
    54 :type :auto})
    55 :ch-recv
    56 (a/pipe ch))))
    

    View Slide

56. Om.Next!
    

    View Slide

57. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions
    

    View Slide

58. 43 (defresource commands [datomic producer pending-cmd-topic accepted-cmd-pub failed-cmd-pub]
    44 :available-media-types ["text/html" "application/json" "application/transit+json" "application/edn"]
    45 :known-content-type? #(check-content-type % ["application/json"
    46 "application/transit+json"
    47 "application/edn"
    48 "application/x-www-form-urlencoded"])
    49 :allowed-methods #{:post :get}
    50 :processable? parse-and-validate-cmd
    51
    52 ...
    53
    54 :post!
    55 (fn [ctx]
    56 (let [id (d/squuid)
    57 sync (get-in ctx [:request :params :sync])
    58 result (if sync
    59 (sync-post! producer pending-cmd-topic accepted-cmd-pub failed-cmd-pub (::cmd ctx) id)
    60 (async-post! producer pending-cmd-topic (::cmd ctx) id))]
    61 {::id id
    62 ::sync sync
    63 ::result (dissoc result :success?)
    64 ::success? (:success? result)
    65 :location (str "/api/v1/commands/" id)}))
    66
    67 ...
    68
    69 )
    

    View Slide

59. 1 (defn await-cmd-submission
    2 [id ch]
    3 (a/alt!!
    4 ch ([v] (assoc v :id id))
    5
    6 (a/timeout 2000) ([v] (throw (ex-info "Timeout writing to Kafka" {:id id})))))
    7
    8 (defn async-post!
    9 [producer pending-cmd-topic cmd id]
    10 (await-cmd-submission id
    11 (kafka/send! producer {:topic pending-cmd-topic
    12 :key id
    13 :value cmd})))
    14
    15 (defn sync-post!
    16 [producer pending-cmd-topic accepted-cmd-pub failed-cmd-pub cmd id]
    17 (let [ach (a/promise-chan)
    18 _ (a/sub accepted-cmd-pub id ach)
    19 fch (a/promise-chan)
    20 _ (a/sub failed-cmd-pub id fch)
    21 _ (log/info ::sync-post! :pre-cmd)
    22 cmdr (await-cmd-submission id
    23 (kafka/send! producer {:topic pending-cmd-topic
    24 :key id
    25 :value cmd}))
    26 _ (log/info ::sync-post! :post-cmd
    27 :cmd cmd
    28 :cmdr cmdr
    29 :accepted-cmd-pub accepted-cmd-pub
    30 :failed-cmd-pub failed-cmd-pub)
    31 base {:cmd cmd
    32 :pending cmdr}
    33 ret (a/alt!!
    34 ach ([v] (assoc base :success? true :accepted v))
    35 fch ([v] (assoc base :success? false :failed v))
    36 (a/timeout 5000) ([v] (throw (ex-info "Timeout reading from Kafka" {:cmd cmd :cmdr cmdr}))))]
    37 (a/close! ach)
    38 (a/unsub accepted-cmd-pub id ach)
    39 (a/close! fch)
    40 (a/unsub failed-cmd-pub id fch)
    41 ret))
    

    View Slide

60. 1 (defresource query [datomic]
    2 :available-media-types ["text/html" "application/json" "application/transit+json" "application/edn"]
    3 :allowed-methods #{:get}
    4 :processable? (fn [ctx]
    5 (let [{:keys [query args as-of history]} (get-in ctx [:request :params])]
    6 [(boolean query) {::query (edn/read-string query)
    7 ::args (edn/read-string args)
    8 ::as-of (safe-parse-int as-of)
    9 ::history (= history "true")}]))
    10 :handle-unprocessable-entity (fn [ctx]
    11 (let [{:keys [::query ::args ::as-of ::history]} ctx
    12 media-type (get-in ctx [:representation :media-type])]
    13 (if (= media-type "text/html")
    14 (h/layout
    15 [:div.query
    16 [:h2 "Invalid Query"]
    17 (query-form-hiccup ctx)])
    18 {:query (pr-str query)
    19 :args (pr-str args)
    20 :as-of as-of
    21 :history history
    22 :error "invalid query"})))
    23 :handle-not-found (fn [ctx] [])
    24 :exists? (fn [ctx]
    25 (when-let [db (-> datomic :conn (get-database ctx))]
    26 {::db db}))
    27
    28 :handle-ok (fn [ctx]
    29 (let [{:keys [::query ::args ::as-of ::history]} ctx
    30 media-type (get-in ctx [:representation :media-type])
    31 db (::db ctx)
    32 result (apply d/q
    33 query
    34 db
    35 args)]
    36 (if (= media-type "text/html")
    37 (h/layout
    38 [:div.query
    39 [:h2 "Query"]
    40 (query-form-hiccup ctx)]
    41 [:div.results
    42 [:h2 "Results"]
    43 (if (seq result)
    44 [:table.results.pure-table.pure-table-bordered
    45 (for [i result]
    46 [:tr.result
    47 (for [j i]
    48 [:td.value (pr-str j)])])]
    49 [:p "No results."])])
    50 {:query (pr-str query)
    51 :args (pr-str args)
    52 :basis-t (d/basis-t db)
    53 :history history
    54 :result result}))))
    

    View Slide

61. It’s still REST!
    • Self documentation/description of 3 endpoints
    • HATEOS
    • GETS cacheable, proxy-able, etc.
    • Content negotiation of various representation formats,
    languages, encodings
    • Conditional requests, ETags, etc.
    • Response codes describe communication semantics and
    command status, not domain semantics
    

    View Slide

62. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions
    

    View Slide

63. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions
    

    View Slide

64. 1 (ns cqrs.processing
    2 (:require [datomic.api :as d]
    3 [cqrs.samza :as samza])
    4 (:import [org.apache.samza.task MessageCollector TaskCoordinator TaskContext]
    5 [org.apache.samza.config Config]
    6 [org.apache.samza.system IncomingMessageEnvelope OutgoingMessageEnvelope SystemStream]))
    7
    8 (gen-class
    9 :name cqrs.processing.TxTask
    10 :prefix tx-
    11 :state state
    12 :init ctor
    13 :implements [org.apache.samza.task.InitableTask
    14 org.apache.samza.task.StreamTask])
    15
    ... ;; helpers to init database connection
    ... ;; (defmulti action->tx) to handle various actions
    61 (defn tx-process
    62 [this
    63 ^IncomingMessageEnvelope envelope
    64 ^MessageCollector collector
    65 ^TaskCoordinator coordinator]
    66 (let [conn @(.state this)
    67 action (.getMessage envelope)
    68 id (.getKey envelope)
    69 offset (.getOffset envelope)
    70 ssp (.getSystemStreamPartition envelope)
    71 ss (.getSystemStream ssp)
    72 system (.getSystem ss)
    73 stream (.getStream ss)
    74 partition (-> ssp .getPartition .getPartitionId)
    75 tx (-> action
    76 action->tx
    77 (tx-metadata id system stream partition offset))
    78
    79 [success? {:keys [db-after]}]
    80 (try [true @(d/transact conn tx)]
    81 (catch Exception e
    82 [false @(d/transact conn (-> tx
    83 last
    84 (assoc :tx/success? false)
    85 vector))]))]
    86 (samza/send-message collector
    87 {:system system :stream (if success? "accepted-txes" "failed-txes")}
    88 id
    89 {:offset offset
    90 :system system
    91 :stream stream
    92 :partition partition
    93 :basis-t (d/basis-t db-after)})))
    

    View Slide

65. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions
    

    View Slide

66. Demo
    

    View Slide

67. Benefits of Design
    

    View Slide

68. Allocate Responsibility to
    Maximize Agility
    • Centralize and standardize the common, tricky,
    non-domain concerns
    • HTTP comms, distributed systems problems
    • Auth(n|z), security
    • Audit, compliance, regulatory
    • Decentralize and experiment with domain logic and
    UI
    

    View Slide

69. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions
    

    View Slide

70. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions
    

    View Slide

71. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions
    

    View Slide

72. Allocate Responsibility to
    Maximize Agility
    • Solve the hard, non-domain (cost-center) concerns
    once
    • Experiment on the domain (money-making)
    concerns N times, with low drag
    • No more web API proliferation!
    

    View Slide

73. No OLTP OLAP Split!
    • Analytics/Big Data/Fast Data are first-class
    • Perform analytics on live data
    • Improve situational awareness for leadership
    (Dashboards!)
    

    View Slide

74. Simplifies Business Logic
    • De-couples each unit of business logic from
    • the web tier and communication concerns
    • every other unit of business logic
    • Simplify development: a Samza job is a single
    method interface
    • Simplify operations: deploy a Samza job, possibly
    in parallel with existing business logic!
    

    View Slide

75. Simplifies Client
    Development
    • Abstract client communication with web tier into
    shared libraries for JS, iOS, Android
    • On read, UI components are demand-driven, a la
    Om.Next
    • On write, clients express intent in domain terms via
    commands
    

    View Slide

76. Reifies customer intent
    • Command events are front and center in development
    process
    • Events are stored and replay-able
    • You can always re-construct the story of how you got
    here
    • You could even re-play using new business logic
    while existing system is running!
    • Very useful due to regulations, audit, compliance
    

    View Slide

77. Next?
    • POC
    • Om.Next UI
    • Test at scale (load, latency)
    • auth(n|z)
    • Improve discoverability of API via hypermedia and displaying
    mapping of action => schema
    • Weigh trade-offs
    • Explore idea of company/service as a reactor to stream of events,
    vs. as database
    

    View Slide

78. References and Inspiration
    • Talks with Tim Ewald and Rich Hickey
    • Datomic — http://datomic.com
    • Martin Kleppmann — http://www.confluent.io/blog/turning-the-database-
    inside-out-with-apache-samza/
    • Om.Next — https://www.youtube.com/watch?v=ByNs9TG30E8
    • Falcor — https://netflix.github.io/falcor/
    • Relay — https://facebook.github.io/relay/
    • Various CQRS musings on the web — http://martinfowler.com/bliki/
    CQRS.html, https://msdn.microsoft.com/en-us/library/dn568103.aspx,
    https://cqrs.files.wordpress.com/2010/11/cqrs_documents.pdf
    

    View Slide

79. Thank you!
    Questions?
    

    View Slide