$30 off During Our Annual Pro Sale. View Details »

Clojure/conj 2015: From REST To CQRS With Clojure, Kafka, & Datomic

Clojure/conj 2015: From REST To CQRS With Clojure, Kafka, & Datomic

Abstract: http://clojure-conj.org/speakers#bcalderwood
Video: https://www.youtube.com/watch?v=qDNPQo9UmJA

Have you ever hit a wall with REST? Does modeling your problem domain into CRUD-able entities feel like fitting a square peg into a round hole? Have you ever tried implementing a PATCH request (without going crazy), or debated PUT vs. POST for resource updates?

An alternative to CRUD-flavored REST is CQRS (Command and Query Responsibility Segregation). In this talk, I'll discuss the advantages and trade-offs of CQRS (and its cousin, Event Sourcing), and demonstrate them using an implementation of CQRS based on Clojure microservices, Kafka, and Datomic. These advantages include:

Operational simplicity, flexibility, and extensibility
Applying Clojure's sound model of state transitions to the resources exposed by your web application
Simplified client implementation
BONUS: We'll even be able to keep the best parts of REST!

I'll also discuss how this technique dovetails nicely with other good ideas in the Clojure community like the design of Datomic and the ideas behind Om.Next.

Bobby Calderwood

November 18, 2015
Tweet

More Decks by Bobby Calderwood

Other Decks in Programming

Transcript

  1. From REST to CQRS
    with Clojure, Kafka (and Samza), and Datomic

    View Slide

  2. Who am I?
    • Member of the Tech Fellows team at
    • Formerly worked on the Product team at Cognitect,
    building and supporting Datomic with Rich, Stu, and
    team
    • Have worked at various startups and large enterprises
    in both business and technical roles
    [email protected]
    @bobbycalderwood
    https://github.com/bobby

    View Slide

  3. How to proceed?
    • Define Problem
    • Essential REST
    • CRUD-flavored REST
    • CQRS and Event Sourcing
    • Design
    • Demo
    • Assess benefits and trade-offs

    View Slide

  4. How not to proceed
    • This talk is NOT a repudiation of REST!
    • https://www.pandastrike.com/posts/20151015-
    rest-vs-relay
    • http://blog.steveklabnik.com/posts/2011-07-03-
    nobody-understands-rest-or-http
    • This talk does NOT describe the only possible
    implementation of CQRS and Event Sourcing via
    HTTP

    View Slide

  5. Goal
    • Provide valuable informational and transactional
    services via Mobile and Web software
    • To lots of customers
    • With excellent user experience
    • Securely and in compliance with regulations
    • Can easily experiment, enhance, maintain, and
    operate

    View Slide

  6. Problems

    View Slide

  7. Organizational Scale
    • Large organization
    • Many product dev teams across several vertical
    LOBs
    • Horizontal dev and ops teams for shared services
    • Currently, deploying a customer-facing application
    involves development of client (mobile or single-
    page JS app) and supporting HTTP API services

    View Slide

  8. HTTP API Proliferation
    • Some externally-facing, to power customer UIs
    • Some partner-facing
    • Some internally-facing, consuming other APIs
    • DevOps only goes so far to taming this complexity!

    View Slide

  9. Teams Reinvent Wheels
    • API development:
    • Distributed systems problems: HTTP comms, failure
    modes, timeouts, retries, etc.
    • Auth(n|z), security
    • Audit, regulatory, compliance, metrics, monitoring
    • Database, cache
    • DevOps automation to build, test, deploy, operate

    View Slide

  10. Teams Reinvent Wheels
    • UI development (JS, iOS, Android):
    • How to talk to various endpoints/APIs
    • Failure modes, transactional semantics,
    granularity of action
    • Optimistic vs. Pessimistic updates

    View Slide

  11. Teams Reinvent Wheels
    • Library and code-reuse only go so far!!!
    • Internal frameworks and libraries
    • Shared practices and sample code repos
    • Communities of Practice/Excellence/Whathaveyou
    • All add layers of bloat, code generation, misdirection or
    obfuscation => complexity
    • None address the fundamental problem of HTTP API
    proliferation

    View Slide

  12. Ideal State
    • Simplify UI development for vertical teams by reducing
    number and complexity of HTTP APIs
    • Simplify HTTP API development for vertical teams by not
    doing so much of it, and handling Business Logic
    differently
    • Horizontal teams handle common functionality via shared
    service and infrastructure re-use (i.e. not via code or
    library re-use), Vertical teams experiment on value adds
    • Record clear picture of customer intent and interactions,
    for audit/compliance, replay, etc.

    View Slide

  13. Essential REST

    View Slide

  14. The Good Parts
    • Clear communication semantics for big distributed
    systems in low-trust, low-coordination context
    • Proxying/caching of various types of requests over
    unknown network path
    • Clear (enough) error codes for when requests fail (most
    of the time)
    • Loose coupling and robustness principle
    • Sometimes even Conditional requests via Expiration,
    ETags, etc. (but not often)

    View Slide

  15. The Good Parts
    • Self-describing/documenting for ease of
    consumption, discovery, navigation
    • HTML representation lets humans (possible client
    implementors) learn/discover resources quickly
    • Sometimes even HATEOS (but not often)

    View Slide

  16. The Good Parts
    • Data-oriented
    • Requests and responses are data, in various
    representation formats, languages, and
    encodings
    • Sometimes even Content Negotiation (but not
    often)

    View Slide

  17. The Bad Parts
    • “To the person with a hammer…” proliferation
    • Tooling and “pop-culture” around REST leads to
    bad implementations

    View Slide

  18. CRUD-flavored REST

    View Slide

  19. The Good Parts
    • My language has tools for that!
    • Java — Jersey/Spring(Boot)/Dropwizard
    • Ruby — Rails/Sinatra
    • JavaScript — Node.js/Express/Loopback
    • Python — Django/Tastypie/Webmachine
    • Clojure — Ring/Pedestal/Liberator/Yada
    • Scala — Play/Lift/Scalatra

    View Slide

  20. The Good Parts
    • Widely supported and Understood
    • Lots of docs, examples, etc.
    • Easy to hire for
    • On-ramp, guardrails for less experienced
    developers
    • Tools like Swagger, etc. for specifying,
    documenting, validating

    View Slide

  21. All the cool kids are doing it!
    • It’s Agile™
    • Easy on ramp tooling makes teams feel productive
    vs. straw-man alternatives like SOAP et. al.
    • Easy to get something (poorly specified) into
    “production”
    • Microservices! Containers! DevOps! Cloud!
    • If it’s good enough for NetflixTwitterFacebook…

    View Slide

  22. The Bad Parts

    View Slide

  23. All the cool kids are doing it!
    • REST is Over! http://blog.steveklabnik.com/posts/
    2012-02-23-rest-is-over

    View Slide

  24. Back into the Tarpit
    We use tools designed to eliminate the accidental
    complexity of mutability-centric object orientation in
    the small…
    • Functional languages with referential
    transparency
    • Immutable data structures
    • Immutable databases (Datomic)

    View Slide

  25. Back into the Tarpit

    View Slide

  26. Back into the Tarpit
    …in order to implement a thin veneer over mutable
    objects in the large!!!

    View Slide

  27. Back into the Tarpit

    View Slide

  28. The S in REST
    • But whereas at least Object Oriented languages
    have clear get/set semantics for attribute updates,
    CRUD-flavored REST has:
    • POST
    • PUT
    • PATCH (?!)
    • Tricky semantics, promote in-place mutability

    View Slide

  29. The Kingdom of Nouns for
    Distributed Systems
    • Jamming every operation into a CRUD on some resource
    is often unnatural, creating impedance mismatch
    • Complects communication semantics with those of
    application domain
    • I want to (choose one):
    A. “pay my credit-card bill”
    B. “POST a new bill payment sub-resource of the credit-
    card resource”

    View Slide

  30. The Kingdom of Nouns for
    Distributed Systems
    • Burdens clients with resource orchestration
    • Keep UI in sync via various read endpoints,
    index vs. show granularity, when to follow
    hypermedia
    • Create a composite action comprised of
    behaviors across various write endpoints
    • What about actions that need to transactionally
    modify several different resources at one time?

    View Slide

  31. The Kingdom of Nouns for
    Distributed Systems
    • Proliferates URL paths and POST/PUT/PATCH/
    DELETE semantics just like Mutable Objects
    proliferate data access methods!
    • Could possibly dispatch domain behavior on
    • HTTP method
    • Header value
    • Path param
    • Query param
    • Some characteristic of Body

    View Slide

  32. Operationally rigid
    • The following criticisms apply to many out-of-the-
    box implementations and deployment patterns for
    CRUD-flavored REST…
    • Entire API, with all N endpoints often deploy,
    version, change, scale, etc. together
    • Scaling individual resources (from a single
    codebase) requires re-work
    • Scaling reads separate from writes is hard

    View Slide

  33. Hard to extend or integrate
    /api/v1/resource
    HTTP API
    Database
    Data Storage
    Integrate here? Or here?!?!?

    View Slide

  34. Obscures the story
    • Typical problems with mutable-state systems
    • What is the client/customer’s actual intent?
    • What happened to bring us to the current state?
    • How do we audit that sequence of events for
    security, compliance, or correctness?
    • Splunk on request logs anyone?

    View Slide

  35. CQRS and Event
    Sourcing

    View Slide

  36. What is CQRS?
    • Command (and) Query Responsibility Segregation
    • Separate interface and model for write (command)
    and read (query) paths in an application
    • In REST/HTTP terms, perhaps:
    • /commands for writes
    • /query (pull) and /updates (push, e.g. via WS or
    SSE) for read

    View Slide

  37. CQRS Baggage
    • CQRS in the small, i.e. within a single codebase
    • For each Entity/Object, have a read class and a write
    class
    • Complexity! Proliferation!
    • CQRS in the large, i.e. architecture of a distributed system
    • At a whole service/enterprise level, have one read
    endpoint and one write endpoint
    • Simplicity! Reduction!

    View Slide

  38. CQRS Baggage
    • CQRS in the small, i.e. within a single codebase
    • For each Entity/Object, have a read class and a write
    class
    • Complexity! Proliferation!
    • CQRS in the large, i.e. architecture of a distributed system
    • At a whole service/enterprise level, have one read
    endpoint and one write endpoint
    • Simplicity! Reduction!

    View Slide

  39. CQRS Baggage
    • Object-Oriented CQRS (esp. Java and C#) got all
    tangled up with
    • Domain-Driven Design
    • Aspect Oriented Programming
    • Naked Objects
    • OO “architecture” (again, in the small)

    View Slide

  40. CQRS Baggage
    • We functional programmers know better!
    • The Language of the System
    • Immutability
    • Values flow through system being transformed
    • Isolate and delay side-effects until the edges

    View Slide

  41. CQRS via HTTP
    /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    /updates
    WS or SSE
    Clients
    Web Services +
    Business Logic
    Database
    Data Storage

    View Slide

  42. Why might I use CQRS?
    • Simplicity and de-complecting
    • Operational simplicity and agility
    • Client implementation simplicity and agility
    (Om.Next!)

    View Slide

  43. What is Event Sourcing?
    • Model and calculate a system’s state by storing all
    events that update that state
    • Often associated with CQRS: store all the events
    that come through command path

    View Slide

  44. CQRS+ES via HTTP
    /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    /updates
    WS or SSE
    Clients
    Web Services +
    Business Logic
    Indexes
    Event Log
    Data Storage

    View Slide

  45. Datomic was CQRS+ES
    before it was cool
    • Datomic follows a CQRS design pattern with Event
    Sourcing, whence much of the goodness
    • Reads are cacheable and distributed among peers
    reading from storage
    • Writes are consistent, atomic, sequential, and only
    performed by transactor
    • State of database at time t is (logically) the
    aggregation of all prior events

    View Slide

  46. Why might I use
    Event Sourcing?
    • Your application models state transitions over time
    • You need to integrate with external systems, or
    extend existing functionality
    • You need a reified, immutable record of
    customer/client intent in domain terms!

    View Slide

  47. Why might I use
    Event Sourcing?
    /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    /updates
    WS or SSE
    Clients
    Web Services +
    Business Logic
    Indexes
    Event Log
    Data Storage External Processes
    Processes!
    Processes!
    Processes!

    View Slide

  48. Why might I prefer CRUD-
    flavored REST over CQRS+ES?
    • External requirement or existing SLA for “REST” API
    • Legacy codebase with existing clients
    • Can’t afford additional operational cost
    • Messaging/event-tracking system in addition to a database
    • Microservice cardinality (i.e. deploy, manage, scale three
    endpoints instead of one)
    • Your problem domain truly is resource-oriented and
    CRUD-y

    View Slide

  49. Design

    View Slide

  50. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions

    View Slide

  51. Command
    • A two-tuple of action and data
    1 [:create-customer {:first_name "John"
    2 :last_name "Doe"
    3 :address {:street_number "1234"
    4 :street_name "Main St."
    5 :city "Anytown"
    6 :state "NH"
    7 :zip "03755"}}]

    View Slide

  52. Query
    • A Datomic datalog query
    1 [:find (pull ?e [*])
    2 :where [?e :customer/id]]

    View Slide

  53. Update
    • A Websocket message about updated data
    1 [:tx/accepted {:key #uuid "5644d334-c537-40de-901e-5cdb9ede4c51"
    2 :value {:offset "5"
    3 :system "local-kafka"
    4 :stream "pending-txes"
    5 :partition 0
    6 :basis-t 1039}
    7 :topic "accepted-txes"
    8 :partition 0
    9 :offset 3}]

    View Slide

  54. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions

    View Slide

  55. 1 (ns cqrs.client
    2 (:require-macros [cljs.core.async.macros :refer [go]])
    3 (:require [cljs-http.client :as http]
    4 [cljs.core.async :as a]
    5 [cognitect.transit :as t]
    6 [taoensso.sente :as sente :refer (cb-success?)]
    7 [taoensso.sente.packers.transit :as sente-transit]))
    8
    9 (defn- request-opts
    10 [opts]
    11 (assoc opts
    12 :channel (a/promise-chan)
    13 :headers {"accept" "application/transit+json"}))
    14
    15 (defn command!
    16 ([action data]
    17 (transact action data false))
    18 ([action data sync?]
    19 (http/post "/api/v1/commands"
    20 (cond-> {:transit-params [action data]}
    21 true request-opts
    22 sync? (assoc :query-params {:sync true})))))
    23
    24 (defn- db-arg?
    25 [o]
    26 (and (map? o)
    27 (or (contains? o :basis-t)
    28 (contains? o :as-of)
    29 (contains? o :history))))
    30
    31 (defn q
    32 [query & args]
    33 (let [[db args] (if (db-arg? (first args))
    34 [(first args) (rest args)]
    35 [nil args])
    36 {:keys [history as-of basis-t]} db
    37 as-of (or as-of basis-t)]
    38 (http/get "/api/v1/query"
    39 (cond-> {:query-params {:query (pr-str query)}}
    40 true request-opts
    41 (seq args) (assoc :args (pr-str args))
    42 as-of (assoc-in [:query-params :as-of] as-of)
    43 history (assoc-in [:query-params :history] history)))))
    44
    45 (defn updates
    46 ([]
    47 (updates
    48 (a/chan 1 (comp (map :event)
    49 (filter #(= :chs/recv (first %)))))))
    50 ([ch]
    51 (-> "/api/v1/updates"
    52 (sente/make-channel-socket!
    53 {:packer (sente-transit/->TransitPacker :json {} {})
    54 :type :auto})
    55 :ch-recv
    56 (a/pipe ch))))

    View Slide

  56. Om.Next!

    View Slide

  57. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions

    View Slide

  58. 43 (defresource commands [datomic producer pending-cmd-topic accepted-cmd-pub failed-cmd-pub]
    44 :available-media-types ["text/html" "application/json" "application/transit+json" "application/edn"]
    45 :known-content-type? #(check-content-type % ["application/json"
    46 "application/transit+json"
    47 "application/edn"
    48 "application/x-www-form-urlencoded"])
    49 :allowed-methods #{:post :get}
    50 :processable? parse-and-validate-cmd
    51
    52 ...
    53
    54 :post!
    55 (fn [ctx]
    56 (let [id (d/squuid)
    57 sync (get-in ctx [:request :params :sync])
    58 result (if sync
    59 (sync-post! producer pending-cmd-topic accepted-cmd-pub failed-cmd-pub (::cmd ctx) id)
    60 (async-post! producer pending-cmd-topic (::cmd ctx) id))]
    61 {::id id
    62 ::sync sync
    63 ::result (dissoc result :success?)
    64 ::success? (:success? result)
    65 :location (str "/api/v1/commands/" id)}))
    66
    67 ...
    68
    69 )

    View Slide

  59. 1 (defn await-cmd-submission
    2 [id ch]
    3 (a/alt!!
    4 ch ([v] (assoc v :id id))
    5
    6 (a/timeout 2000) ([v] (throw (ex-info "Timeout writing to Kafka" {:id id})))))
    7
    8 (defn async-post!
    9 [producer pending-cmd-topic cmd id]
    10 (await-cmd-submission id
    11 (kafka/send! producer {:topic pending-cmd-topic
    12 :key id
    13 :value cmd})))
    14
    15 (defn sync-post!
    16 [producer pending-cmd-topic accepted-cmd-pub failed-cmd-pub cmd id]
    17 (let [ach (a/promise-chan)
    18 _ (a/sub accepted-cmd-pub id ach)
    19 fch (a/promise-chan)
    20 _ (a/sub failed-cmd-pub id fch)
    21 _ (log/info ::sync-post! :pre-cmd)
    22 cmdr (await-cmd-submission id
    23 (kafka/send! producer {:topic pending-cmd-topic
    24 :key id
    25 :value cmd}))
    26 _ (log/info ::sync-post! :post-cmd
    27 :cmd cmd
    28 :cmdr cmdr
    29 :accepted-cmd-pub accepted-cmd-pub
    30 :failed-cmd-pub failed-cmd-pub)
    31 base {:cmd cmd
    32 :pending cmdr}
    33 ret (a/alt!!
    34 ach ([v] (assoc base :success? true :accepted v))
    35 fch ([v] (assoc base :success? false :failed v))
    36 (a/timeout 5000) ([v] (throw (ex-info "Timeout reading from Kafka" {:cmd cmd :cmdr cmdr}))))]
    37 (a/close! ach)
    38 (a/unsub accepted-cmd-pub id ach)
    39 (a/close! fch)
    40 (a/unsub failed-cmd-pub id fch)
    41 ret))

    View Slide

  60. 1 (defresource query [datomic]
    2 :available-media-types ["text/html" "application/json" "application/transit+json" "application/edn"]
    3 :allowed-methods #{:get}
    4 :processable? (fn [ctx]
    5 (let [{:keys [query args as-of history]} (get-in ctx [:request :params])]
    6 [(boolean query) {::query (edn/read-string query)
    7 ::args (edn/read-string args)
    8 ::as-of (safe-parse-int as-of)
    9 ::history (= history "true")}]))
    10 :handle-unprocessable-entity (fn [ctx]
    11 (let [{:keys [::query ::args ::as-of ::history]} ctx
    12 media-type (get-in ctx [:representation :media-type])]
    13 (if (= media-type "text/html")
    14 (h/layout
    15 [:div.query
    16 [:h2 "Invalid Query"]
    17 (query-form-hiccup ctx)])
    18 {:query (pr-str query)
    19 :args (pr-str args)
    20 :as-of as-of
    21 :history history
    22 :error "invalid query"})))
    23 :handle-not-found (fn [ctx] [])
    24 :exists? (fn [ctx]
    25 (when-let [db (-> datomic :conn (get-database ctx))]
    26 {::db db}))
    27
    28 :handle-ok (fn [ctx]
    29 (let [{:keys [::query ::args ::as-of ::history]} ctx
    30 media-type (get-in ctx [:representation :media-type])
    31 db (::db ctx)
    32 result (apply d/q
    33 query
    34 db
    35 args)]
    36 (if (= media-type "text/html")
    37 (h/layout
    38 [:div.query
    39 [:h2 "Query"]
    40 (query-form-hiccup ctx)]
    41 [:div.results
    42 [:h2 "Results"]
    43 (if (seq result)
    44 [:table.results.pure-table.pure-table-bordered
    45 (for [i result]
    46 [:tr.result
    47 (for [j i]
    48 [:td.value (pr-str j)])])]
    49 [:p "No results."])])
    50 {:query (pr-str query)
    51 :args (pr-str args)
    52 :basis-t (d/basis-t db)
    53 :history history
    54 :result result}))))

    View Slide

  61. It’s still REST!
    • Self documentation/description of 3 endpoints
    • HATEOS
    • GETS cacheable, proxy-able, etc.
    • Content negotiation of various representation formats,
    languages, encodings
    • Conditional requests, ETags, etc.
    • Response codes describe communication semantics and
    command status, not domain semantics

    View Slide

  62. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions

    View Slide

  63. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions

    View Slide

  64. 1 (ns cqrs.processing
    2 (:require [datomic.api :as d]
    3 [cqrs.samza :as samza])
    4 (:import [org.apache.samza.task MessageCollector TaskCoordinator TaskContext]
    5 [org.apache.samza.config Config]
    6 [org.apache.samza.system IncomingMessageEnvelope OutgoingMessageEnvelope SystemStream]))
    7
    8 (gen-class
    9 :name cqrs.processing.TxTask
    10 :prefix tx-
    11 :state state
    12 :init ctor
    13 :implements [org.apache.samza.task.InitableTask
    14 org.apache.samza.task.StreamTask])
    15
    ... ;; helpers to init database connection
    ... ;; (defmulti action->tx) to handle various actions
    61 (defn tx-process
    62 [this
    63 ^IncomingMessageEnvelope envelope
    64 ^MessageCollector collector
    65 ^TaskCoordinator coordinator]
    66 (let [conn @(.state this)
    67 action (.getMessage envelope)
    68 id (.getKey envelope)
    69 offset (.getOffset envelope)
    70 ssp (.getSystemStreamPartition envelope)
    71 ss (.getSystemStream ssp)
    72 system (.getSystem ss)
    73 stream (.getStream ss)
    74 partition (-> ssp .getPartition .getPartitionId)
    75 tx (-> action
    76 action->tx
    77 (tx-metadata id system stream partition offset))
    78
    79 [success? {:keys [db-after]}]
    80 (try [true @(d/transact conn tx)]
    81 (catch Exception e
    82 [false @(d/transact conn (-> tx
    83 last
    84 (assoc :tx/success? false)
    85 vector))]))]
    86 (samza/send-message collector
    87 {:system system :stream (if success? "accepted-txes" "failed-txes")}
    88 id
    89 {:offset offset
    90 :system system
    91 :stream stream
    92 :partition partition
    93 :basis-t (d/basis-t db-after)})))

    View Slide

  65. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions

    View Slide

  66. Demo

    View Slide

  67. Benefits of Design

    View Slide

  68. Allocate Responsibility to
    Maximize Agility
    • Centralize and standardize the common, tricky,
    non-domain concerns
    • HTTP comms, distributed systems problems
    • Auth(n|z), security
    • Audit, compliance, regulatory
    • Decentralize and experiment with domain logic and
    UI

    View Slide

  69. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions

    View Slide

  70. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions

    View Slide

  71. /commands(/:id)
    /query
    POST
    GET (single
    or collection)
    GET
    pending-cmds
    /updates
    WS or SSE
    Canonical
    Datastore
    Cache, Index
    or Materialized
    View
    Transaction
    Processing
    accepted-cmds
    1
    2a
    sync
    Index/cache/
    view update
    API/Data lake/
    Process
    arbitrary topics
    Arbitrary
    command
    action
    Arbitrary
    command
    action
    Arbitrary action
    Clients Web Services
    Load balancing
    Auth(n|z)
    Input validation
    REST minus CRUD
    failed-cmds
    2b
    Business Logic
    Command Log Datastores and
    Materialized Views
    sync
    cmd-status
    Remembering customer
    intent
    1. Write command down, whole
    command on success, just metadata of
    command on failure/error.
    2a. On successfully applied cmd, write to
    accepted-cmds topic
    2b. On failed application of cmd, write to
    failed-cmds topic
    Query for
    arbitrarily
    nested data
    structures
    Subscribe to
    server-side
    updates to stay
    current
    Create and
    view
    commands/
    transactions

    View Slide

  72. Allocate Responsibility to
    Maximize Agility
    • Solve the hard, non-domain (cost-center) concerns
    once
    • Experiment on the domain (money-making)
    concerns N times, with low drag
    • No more web API proliferation!

    View Slide

  73. No OLTP OLAP Split!
    • Analytics/Big Data/Fast Data are first-class
    • Perform analytics on live data
    • Improve situational awareness for leadership
    (Dashboards!)

    View Slide

  74. Simplifies Business Logic
    • De-couples each unit of business logic from
    • the web tier and communication concerns
    • every other unit of business logic
    • Simplify development: a Samza job is a single
    method interface
    • Simplify operations: deploy a Samza job, possibly
    in parallel with existing business logic!

    View Slide

  75. Simplifies Client
    Development
    • Abstract client communication with web tier into
    shared libraries for JS, iOS, Android
    • On read, UI components are demand-driven, a la
    Om.Next
    • On write, clients express intent in domain terms via
    commands

    View Slide

  76. Reifies customer intent
    • Command events are front and center in development
    process
    • Events are stored and replay-able
    • You can always re-construct the story of how you got
    here
    • You could even re-play using new business logic
    while existing system is running!
    • Very useful due to regulations, audit, compliance

    View Slide

  77. Next?
    • POC
    • Om.Next UI
    • Test at scale (load, latency)
    • auth(n|z)
    • Improve discoverability of API via hypermedia and displaying
    mapping of action => schema
    • Weigh trade-offs
    • Explore idea of company/service as a reactor to stream of events,
    vs. as database

    View Slide

  78. References and Inspiration
    • Talks with Tim Ewald and Rich Hickey
    • Datomic — http://datomic.com
    • Martin Kleppmann — http://www.confluent.io/blog/turning-the-database-
    inside-out-with-apache-samza/
    • Om.Next — https://www.youtube.com/watch?v=ByNs9TG30E8
    • Falcor — https://netflix.github.io/falcor/
    • Relay — https://facebook.github.io/relay/
    • Various CQRS musings on the web — http://martinfowler.com/bliki/
    CQRS.html, https://msdn.microsoft.com/en-us/library/dn568103.aspx,
    https://cqrs.files.wordpress.com/2010/11/cqrs_documents.pdf

    View Slide

  79. Thank you!
    Questions?

    View Slide