Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Practical Generative Testing Patterns

Practical Generative Testing Patterns

Talk slides as presented at Dutch Clojure Days 2018

Srihari Sriraman

April 21, 2018
Tweet

More Decks by Srihari Sriraman

Other Decks in Technology

Transcript

  1. why

  2. practical system specimen user signup/create authenticate add/update-card get cart add/update-item

    remove-item get payment pay refund/cancel apply-offer receipt a generic e-commerce system
  3. problems inertia, learning curve thinking in properties slow, and ineffective

    high maintenance difficult to diagnose benefits better than humans at input generation more tests, more coverage better chance at finding bugs before users do
  4. problems inertia, learning curve thinking in properties slow, and ineffective

    high maintenance difficult to diagnose benefits better than humans at input generation more tests, more coverage better chance at finding bugs before users do
  5. problems inertia, learning curve thinking in properties slow, and ineffective

    high maintenance difficult to diagnose inertia John Hughes – Testing the Hard Stuff and Staying Sane
  6. learning curve Gary Fredericks – Building test check Generators problems

    inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
  7. thinking in properties purely functional data algorithms or data structures

    are relatively easy to test most practical systems involve making a series of stateful api calls, most of which are CRUD problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
  8. slow and ineffective CI duration grows exponentially bugs found in

    test failures aren’t necessarily relevant more tests ≠ more coverage problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
  9. high maintenance generally more complex to write and read arrange,

    and act portions of test are still dependent on source problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
  10. difficult to diagnose automatic shrinking really helps, but only until

    a point complex, stateful tests aren’t necessarily deterministic, and are difficult to reproduce finding root-cause of failure without the right tools can be very difficult problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
  11. system under test fn input fn argument request state output

    return value response state state one fn many fns entire system
  12. action under test fn input fn argument request state output

    return value response state state one fn many fns entire system
  13. pattern /ˈpat(ə)n/ a regular and intelligible form or sequence discernible

    in the way in which something happens or is done
  14. 1. derive parameter specification 2. declare action dependencies 3. probability

    matrices for flows 4. deterministic seed data 5. model external domain patterns in generating input
  15. derive param specification g1. (ns system.schema) (defschema CreateAccountParam {:email (s/maybe

    valid-email) :username non-empty-string :password non-empty-string}) (defschema UserAccount {:id String :email String :username String}) (ns system.account) (defaction user-create :name ::user/create :param-schema s/CreateAccountParam :value-schema s/UserAccount :fn ([{:as param}] ...)) param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
  16. derive param specification g1. (ns system.schema) (defschema CreateAccountParam {:email (s/maybe

    valid-email) :username non-empty-string :password non-empty-string}) (defschema UserAccount {:id String :email String :username String}) (ns system.account) (defaction user-create :name ::user/create :param-schema s/CreateAccountParam :value-schema s/UserAccount :fn ([{:as param}] ...)) param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
  17. g1. (ns system.test (:require [clojure.test.check.generators :as gen])) (def valid-email-gen (->

    string/join (gen/fmap (gen/vector gen/char-alphanumeric 1 20)) (gen/bind #(gen/return (str % "@example.com"))))) (gen/generate (params-gen ::user/create)) {:email "[email protected]" :username "Jn*kLqvOXol:u^>XC" :password "y&aOrni}"} derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
  18. g1. (ns system.test (:require [clojure.test.check.generators :as gen])) (def valid-email-gen (->

    string/join (gen/fmap (gen/vector gen/char-alphanumeric 1 20)) (gen/bind #(gen/return (str % "@example.com"))))) (gen/generate (params-gen ::user/create)) {:email "[email protected]" :username "Jn*kLqvOXol:u^>XC" :password "y&aOrni}"} derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
  19. g1. (defn error? [response] (or (not (<= 200 (:status response)

    299)) (exception? response) (timeout? response))) (defn assert-no-error [action] (let [params (gen/generate (params-gen action)) response (exec-action action params)] (is (not (error? response))))) (assert-no-error ::user/create) derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
  20. g1. (defn error? [response] (or (not (<= 200 (:status response)

    299)) (exception? response) (timeout? response))) (defn assert-no-error [action] (let [params (gen/generate (params-gen action)) response (exec-action action params)] (is (not (error? response))))) (assert-no-error ::user/create) derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
  21. g1. (defn error? [response] (or (not (<= 200 (:status response)

    299)) (exception? response) (timeout? response))) (defn assert-no-error [action] (let [params (gen/generate (params-gen action)) response (exec-action action params)] (is (not (error? response))))) (assert-no-error ::user/create) derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
  22. g2. (defaction user-create :name ::user/create :param-schema s/CreateAccountParam :value-schema s/UserAccount :fn

    ([{:as param}] ...)) declare action dependencies what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase
  23. g2. (def checkout-dependencies {::user/create [] ::user/authenticate [::user/create] ::user/add-card [::user/authenticate] ::user/update-card

    [::user/add-card] ::user/get [::user/create] ::cart/add-item [::user/authenticate] ::cart/update-item [::cart/add-item] ::cart/remove-item [::cart/add-item] ::cart/get [::user/authenticate] ::payment/apply-offer [::cart/add-item] ::payment/pay [::user/add-card ::cart/add-item] ::payment/refund [::user/pay] ::payment/cancel [::cart/add-item]}) declare action dependencies what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase
  24. g2. declare action dependencies (def checkout-dependencies {::user/create [] ::user/authenticate [::user/create]

    ::user/add-card [::user/authenticate] ::user/update-card [::user/add-card] ::user/get [::user/create] ::cart/add-item [::user/authenticate] ::cart/update-item [::cart/add-item] ::cart/remove-item [::cart/add-item] ::cart/get [::user/authenticate] ::payment/apply-offer [::cart/add-item] ::payment/pay [::user/add-card ::cart/add-item] ::payment/refund [::user/pay] ::payment/cancel [::cart/add-item]}) what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase
  25. g2. declare action dependencies (def all-actions #{::user/create ::user/authenticate ::user/add-card ::user/update-card

    ::user/get ::cart/add-item ::cart/update-item ::cart/remove-item ::cart/get ::payment/apply-offer ::payment/pay ::payment/refund ::payment/cancel}) (for [action actions] (assert-no-error action)) what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase
  26. g3. what action might occur after? probabilistic modelling of usage

    is effective 0 and 100 are special cases automate the “act” phase probability matrices for flows (def most-users-flow {::user/create {::user/authenticate 90} ::user/authenticate {::user/add-card 20 ::cart/add-item 50} ::user/add-card {::user/update-card 10} ::cart/add-item {::cart/add-item 30 ::cart/update-item 10 ::cart/remove-item 20 ::payment/pay 20} ::payment/pay {::payment/refund 5}}) (def infrequent-users-flow …) (def rich-users-flow ...)
  27. g3. probability matrices for flows (def most-users-flow {::user/create {::user/authenticate 90}

    ::user/authenticate {::user/add-card 20 ::cart/add-item 50} ::user/add-card {::user/update-card 10} ::cart/add-item {::cart/add-item 30 ::cart/update-item 10 ::cart/remove-item 20 ::payment/pay 20} ::payment/pay {::payment/refund 5}}) (def infrequent-users-flow …) (def rich-users-flow ...) what action might occur after? probabilistic modelling of usage is effective 0 and 100 are special cases automate the “act” phase
  28. g3. what action might occur after? probabilistic modelling of usage

    is effective 0 and 100 are special cases automate the “act” phase probability matrices for flows (def most-users-flow {::user/create {::user/authenticate 90} ::user/authenticate {::user/add-card 20 ::cart/add-item 50} ::user/add-card {::user/update-card 10} ::cart/add-item {::cart/add-item 30 ::cart/update-item 10 ::cart/remove-item 20 ::payment/pay 20} ::payment/pay {::payment/refund 5}}) (def infrequent-users-flow …) (def rich-users-flow ...)
  29. g3. what action might occur after? probabilistic modelling of usage

    is effective 0 and 100 are special cases automate the “act” phase probability matrices for flows (->> (action-seq-gen most-users-flow) gen/sample (take 2)) ([::user/create ::user/authenticate ::cart/add-item ::cart/update-item ::cart/update-item ::cart/get ::cart/update-item ::cart/get ::cart/remove-item ::user/add-card ::payment/apply-offer ::payment/apply-offer ::payment/cancel] [::user/create ::user/authenticate ::cart/add-item ::payment/apply-offer ::cart/add-item ::cart/add-item ::cart/add-item ::cart/add-item ::cart/add-item ::cart/get ::cart/update-item])
  30. g3. what action might occur after? probabilistic modelling of usage

    is effective 0 and 100 are special cases automate the “act” phase probability matrices for flows [{:action ::user/create :params {:email "d3*(@example.com" :username "sdf@37" ...}} {:action ::user/authenticate :params {:username “73@fds” :password "23nv#2&" ...}} {:action ::cart/add-item :params {:sku 143 :qty 1 ...}} ...]
  31. g4. model close-to real scenarios, not completely random ones simplifies

    debugging gives deterministic deterministic seed data [{:action ::user/create :params {:email "[email protected]" :username "bill" ...}} {:action ::user/authenticate :params {:username "bill" :password "bill_password" ...}} {:action ::user/add-card :params {:card-numer "1234123412341234" :cvc "606" ...}} ...]
  32. g5. software only models subset of domain tests need to

    model domain around usage model external domain money in user’s card payment gateway latency inventory of supplier
  33. 1. store everything immutably 2. params as latest values in

    state 3. catalog known errors 4. abstract request engine patterns in executing generative tests
  34. store everything immutably e1. ability to query for anything during

    execution, assertion or diagnosis tests need to be completely independent of source timeline matters (:require [datascript.core :as d]) (defn store-action [flow-id action-id request response entities] (let [metadata {:flow-id flow-id :action-id action-id}] (d/transact! db/conn [(-> request (assoc :data-type :request) (merge metadata))]) (d/transact! db/conn [(-> response (assoc :data-type :response) (merge metadata))]) (d/transact! db/conn [(-> entities compute-state (assoc :data-type :state) (merge metadata))])))
  35. store everything immutably e1. ability to query for anything during

    execution, assertion or diagnosis tests need to be completely independent of source timeline matters (:require [datascript.core :as d]) (defn store-action [flow-id action-id request response entities] (let [metadata {:flow-id flow-id :action-id action-id}] (d/transact! db/conn [(-> request (assoc :data-type :request) (merge metadata))]) (d/transact! db/conn [(-> response (assoc :data-type :response) (merge metadata))]) (d/transact! db/conn [(-> entities compute-state (assoc :data-type :state) (merge metadata))])))
  36. store everything immutably e1. ability to query for anything during

    execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn items [flow-id] (->> (d/q '[:find ?sku :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :cart/sku ?sku]] @db/conn flow-id) (map first)))
  37. store everything immutably e1. ability to query for anything during

    execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn items [flow-id] (->> (d/q '[:find ?sku :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :cart/sku ?sku]] @db/conn flow-id) (map first)))
  38. store everything immutably e1. ability to query for anything during

    execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn params-for [flow-id action] (->> (d/q '[:find ?request :in $ ?flow-id :where [?request :flow-id ?flow-id] [?request :data-type :request] [?request :action action]] @db/conn flow-id) (map first)))
  39. store everything immutably e1. ability to query for anything during

    execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn params-for [flow-id action] (->> (d/q '[:find ?request :in $ ?flow-id :where [?request :flow-id ?flow-id] [?request :data-type :request] [?request :action action]] @db/conn flow-id) (map first)))
  40. store everything immutably e1. ability to query for anything during

    execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn user-timeline [flow-id user-id] (->> (d/q '[:find ?e ?t :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :user-id ?user-id ?t]] @db/conn flow-id) (sort-by second >)))
  41. store everything immutably e1. ability to query for anything during

    execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn user-timeline [flow-id user-id] (->> (d/q '[:find ?e ?t :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :user-id ?user-id ?t]] @db/conn flow-id) (sort-by second >)))
  42. params as latest value most recent value is most relevant

    use hooks to manipulate engine behaviour e2. (defn latest-param [flow-id attr-name] (->> (d/q '[:find ?attr-val ?t :in $ ?attr-name ?flow-id :where [?as :data-type :action-spec] [?as :flow-id ?flow-id] [?as :action-id ?aid] [?e :action-id ?aid] [?e ?attr-name ?attr-val ?t]] @db/conn attr-name flow-id) (sort-by second >) ffirst))
  43. params as latest value most recent value is most relevant

    use hooks to manipulate engine behaviour e2. (defn latest-param [flow-id attr-name] (->> (d/q '[:find ?attr-val ?t :in $ ?attr-name ?flow-id :where [?as :data-type :action-spec] [?as :flow-id ?flow-id] [?as :action-id ?aid] [?e :action-id ?aid] [?e ?attr-name ?attr-val ?t]] @db/conn attr-name flow-id) (sort-by second >) ffirst))
  44. params as latest value most recent value is most relevant

    use hooks to manipulate engine behaviour e2. {::user/create {:entity-type :user :params-fn (fn [flow-id params] (assoc params :name (str "gen-user-" (u/uuid))))} ::payment/pay {:entity-type :payment :params-fn (fn [flow-id params] (->> (p/fetch-param flow-id :default-card) (assoc params :card-number)))}}
  45. catalog known errors improve relevance reduce redundancy absorb into system

    e3. {::user/create {:entity-type :user :known-exceptions #{:duplicate-user} :params-fn (fn ...)} ::payment/pay {:entity-type :payment :known-exceptions #{:inactive-session :payment-complete} :params-fn (fn ...)}}
  46. catalog known errors improve relevance reduce redundancy absorb into system

    e3. {::user/create {:entity-type :user :known-exceptions #{:duplicate-user} :params-fn (fn ...)} ::payment/pay {:entity-type :payment :known-exceptions #{:inactive-session :payment-complete} :params-fn (fn ...)}}
  47. catalog known errors improve relevance reduce redundancy absorb into system

    e3. (def known-errors {:duplicate-user (fn [flow-id params] (some? (exec-action ::user/get (:username params)))) :payment-complete (fn [flow-id params] (some? (db/find-action flow-id ::payment/pay)))})
  48. abstract request engine quickly switch to higher or lower level

    of specificity api vs controller makes debugging simpler e4. (defn make-request [medium action params] (case medium :fn ((:fn action) params) :bus (bus/request-value action params) :http (-> action http-request (client/request params))))
  49. property unit (no order) integration / system (ordered) no-error* ✓

    ✓ egality ✗ ✓ idempotence ⍻ ✓ identity ✓ ✗ inverse ✗ ✓ commutativity ✗ ✓ associativity ✗ ✓ algebraic properties a1.
  50. algebraic properties no-error {:message "No :error" :result (not-any? #(= :error

    %) (map :status responses))} ;; status (if (or (not (<= 200 status 299)) (some? exception) (timeout?)) :error) no-exceptions no-timeouts always-available a1.
  51. algebraic properties egality (assert-equal [{:action ::user/signup}] [{:action ::user/get}]) (assert-equal [{:action

    ::cart/update-item :params {:deleted? true}}] [{:action ::cart/remove-item}]) f(a) ≡ g(b) effective equality different paths, same result a1.
  52. algebraic properties idempotence {:percent-flow 10 :adjacency {:immediate 10 :distant 90}

    :blacklist #{::payment/pay ::payment/refund ::payment/cancel ::user/signup}} f(a) · f(a) ≡ f(a) 2 kinds of adjacencies blacklist / whitelist sensitive / distributed systems a b c c d e a b c d c e immediate distant a1.
  53. algebraic properties inverse (assert-equal [{:action ::cart/add-item}] [{:action ::cart/remove-item}] :value-action ::cart/get)

    ;; normalised (assert-equal [{:action ::payment/pay} {:action ::payment/cancel} {:action ::payment/pay}] [{:action ::payment/pay}] :value-action ::payment/receipt) f(a) · f -1(a) ≡ a f -1(f(f -1(a)))=f -1(a) normalised inverse a1.
  54. algebraic properties commutativity (assert-equal [{:action ::cart/add-item :params {:sku 1}} {:action

    ::cart/add-item :params {:sku 2}}] [{:action ::cart/add-item :params {:sku 2}} {:action ::cart/add-item :params {:sku 1}}] :value-action ::cart/get) f(a) · g(a) ≡ g(a) · f(a) non-compliance is a property too a1.
  55. algebraic properties commutativity (assert-commutative [{:action ::cart/add-item :params {:sku 1}} {:action

    ::cart/add-item :params {:sku 2}}] :value-action ::cart/get) (assert-not-commutative [{:action ::cart/update-item :params {:sku 1 :qty 1}} {:action ::cart/update-item :params {:sku 1 :qty 6}}] :value-action ::cart/get) f(a) · g(a) ≡ g(a) · f(a) non-compliance is a property too a1.
  56. state machines as properties new has-items success cancelled refunded authenticate

    add-item update-item remove-item apply-offer pay refund cancel remove-item a2.
  57. (defn compute-expected-state [state action status] (match [state action status] [_

    ::user/authenticate :ok] "new" ["new" ::cart/add-item :ok] "has-items" ["has-items" ::cart/update-item :ok] "has-items" ["has-items" ::cart/remove-item :ok] "has-items" ["has-items" ::cart/remove-item :ok] "new" ["has-items" ::payment/apply-offer :ok] "has-items" ["has-items" ::payment/pay :ok] "success" ["success" ::payment/refund :ok] "refunded" ["has-items" ::payment/cancel :ok] "cancelled" [_ _ :error] "error" [_ _ :known-exception] "known-exception")) a2.
  58. 1. tests as portable data 2. domain based checkpoints 3.

    flow timeline, and walk throughs 4. automated bug reports patterns in diagnosis
  59. tests as portable data flow / action-specs are stored tests

    determinism 㱺 reproducibility store passing and failing tests version tests d1. [{:action ::user/create :params {:email "d3*(@example.com" :username "sdf@37" ...}} {:action ::user/authenticate :params {:username “73@fds” :password "23nv#2&" ...}} {:action ::cart/add-item :params {:sku 143 :qty 1 ...}} ...]
  60. tests as portable data flow / action-specs are stored tests

    determinism 㱺 reproducibility store passing and failing tests version tests d1. (-> "rc42-login-bug.edn" test/import test/run) (-> "v42-regression-test.edn" test/import test/run)
  61. domain based checkpoints think of code as a trie useful

    when there are 100s of flows as on CI d2. {:checkpoints [{:action ::user/authenticate :description "Logged in"} {:action ::cart/add-item :description "Added item to cart"} {:action ::payment/pay :description "Payment attempted"}]}
  62. flow timeline, walkthroughs d3. | action | status | action-id

    | time | |-----------------------+--------+-------------------+------------| | ::user/create | :ok | 1d8040ee-2486-... | 1524287925 | | ::user/authenticate | :ok | 92c6fb41-ad78-... | 1524287925 | | ::cart/add-item | :ok | 20c55130-85d2-... | 1524287927 | | ::payment/apply-offer | :ok | 239ebf75-a6c4-... | 1524287928 | | ::cart/add-item | :ok | d991dc62-5322-... | 1524287928 | | ::cart/add-item | :ok | f4b02a65-ac8f-... | 1524287928 | | ::cart/add-item | :ok | e0b85377-b6b7-... | 1524287931 | | ::cart/add-item | :ok | cb676220-8d6a-... | 1524287942 | | ::cart/add-item | :ok | 1d9546fb-7356-... | 1524287942 | | ::cart/get | :ok | b282a885-6e32-... | 1524287942 | | ::cart/update-item | :ex | ed062dca-f457-... | 1524287943 | timeline of actions relating to a user step-by-step diagnosis of events everything has an id, and a timestamp
  63. flow timeline, walkthroughs d3. (dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6") {:metadata {:action ::cart/update-item :flow-id

    "35cb3104f-e0b0-..." :action-id "1d9546fb-7356-..."}} [{:data-type :action-spec :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :request :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :response :response {:status :error, :error-type java.lang.NullPointerException, :report {:cause #error { :cause nil :via [{:type java.lang.NullPointerException :message nil :at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}] :trace [[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77] [demo-proj.modules.cart$extract_filename invoke "cart.clj" 74] [demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139] [demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134] [clojure.lang.Var invoke "Var.java" 379] [demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke "logging.clj" 11] [intbus.core$invoke_handler_fn invokeStatic "core.clj" 134] [intbus.core$invoke_handler_fn invoke "core.clj" 129] [intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}] timeline of actions relating to a user step-by-step diagnosis of events everything has an id, and a timestamp
  64. flow timeline, walkthroughs d3. (dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6") {:metadata {:action ::cart/update-item :flow-id

    "35cb3104f-e0b0-..." :action-id "1d9546fb-7356-..."}} [{:data-type :action-spec :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :request :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :response :response {:status :error, :error-type java.lang.NullPointerException, :report {:cause #error { :cause nil :via [{:type java.lang.NullPointerException :message nil :at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}] :trace [[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77] [demo-proj.modules.cart$extract_filename invoke "cart.clj" 74] [demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139] [demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134] [clojure.lang.Var invoke "Var.java" 379] [demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke "logging.clj" 11] [intbus.core$invoke_handler_fn invokeStatic "core.clj" 134] [intbus.core$invoke_handler_fn invoke "core.clj" 129] [intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}] timeline of actions relating to a user step-by-step diagnosis of events everything has an id, and a timestamp
  65. flow timeline, walkthroughs d3. (dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6") {:metadata {:action ::cart/update-item :flow-id

    "35cb3104f-e0b0-..." :action-id "1d9546fb-7356-..."}} [{:data-type :action-spec :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :request :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :response :response {:status :error, :error-type java.lang.NullPointerException, :report {:cause #error { :cause nil :via [{:type java.lang.NullPointerException :message nil :at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}] :trace [[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77] [demo-proj.modules.cart$extract_filename invoke "cart.clj" 74] [demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139] [demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134] [clojure.lang.Var invoke "Var.java" 379] [demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke "logging.clj" 11] [intbus.core$invoke_handler_fn invokeStatic "core.clj" 134] [intbus.core$invoke_handler_fn invoke "core.clj" 129] [intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}] timeline of actions relating to a user step-by-step diagnosis of events everything has an id, and a timestamp
  66. 1. derive parameter specification 2. declare action dependencies 3. probability

    matrices for flows 4. deterministic seed data 5. model external domain 1. store everything immutably 2. params as latest values in state 3. catalog known errors 4. abstract request engine 1. algebraic properties 2. state machines as properties 3. domain based invariants 1. tests as portable data 2. domain based checkpoints 3. flow timeline, and walk throughs 4. automated bug reports generation assertion execution diagnosis