practical generative testing patterns srihari sriraman | ø nilenso

about this talk why, what a practical system benefits and problems terminology patterns

why

tests are good why don’t we write them often enough? generative tests are better

what

a thought paradigm automating software testing patterns

a practical system

practical system specimen user signup/create authenticate add/update-card get cart add/update-item remove-item get payment pay refund/cancel apply-offer receipt a generic e-commerce system

practical system specimen a generic e-commerce system create (signup) authenticate (login) add-card add-item pay update-item

benefits and problems

problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose benefits better than humans at input generation more tests, more coverage better chance at finding bugs before users do

problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose benefits better than humans at input generation more tests, more coverage better chance at finding bugs before users do

problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose inertia John Hughes – Testing the Hard Stuff and Staying Sane

learning curve Gary Fredericks – Building test check Generators problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose

thinking in properties purely functional data algorithms or data structures are relatively easy to test most practical systems involve making a series of stateful api calls, most of which are CRUD problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose

slow and ineffective CI duration grows exponentially bugs found in test failures aren’t necessarily relevant more tests ≠ more coverage problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose

high maintenance generally more complex to write and read arrange, and act portions of test are still dependent on source problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose

difficult to diagnose automatic shrinking really helps, but only until a point complex, stateful tests aren’t necessarily deterministic, and are difficult to reproduce finding root-cause of failure without the right tools can be very difficult problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose

terminology

phase of testing level of testing focus of testing segments

assume arrange act assert phase level focus

generation execution assertion diagnosis phase level focus

System Integration Unit + Value + Speed phase level focus

System Integration Unit + Value + Speed phase level focus

System phase level focus

correctness (smoke, acceptance, verification) regression (functional, quality control) performance (load, stress, endurance, spike) phase level focus

system under test fn input fn argument request state output return value response state state one fn many fns entire system

action under test fn input fn argument request state output return value response state state one fn many fns entire system

automation QA Engineer Test Developer

patterns

pattern /ˈpat(ə)n/ a regular and intelligible form or sequence discernible in the way in which something happens or is done

generation execution assertion diagnosis patterns in generating input

1. derive parameter specification 2. declare action dependencies 3. probability matrices for flows 4. deterministic seed data 5. model external domain patterns in generating input

derive param specification g1. (ns system.schema) (defschema CreateAccountParam {:email (s/maybe valid-email) :username non-empty-string :password non-empty-string}) (defschema UserAccount {:id String :email String :username String}) (ns system.account) (defaction user-create :name ::user/create :param-schema s/CreateAccountParam :value-schema s/UserAccount :fn ([{:as param}] ...)) param spec in action metadata generate input from spec no maintenance of test params spec can become api doc

derive param specification g1. (ns system.schema) (defschema CreateAccountParam {:email (s/maybe valid-email) :username non-empty-string :password non-empty-string}) (defschema UserAccount {:id String :email String :username String}) (ns system.account) (defaction user-create :name ::user/create :param-schema s/CreateAccountParam :value-schema s/UserAccount :fn ([{:as param}] ...)) param spec in action metadata generate input from spec no maintenance of test params spec can become api doc

g1. (ns system.test (:require [clojure.test.check.generators :as gen])) (def valid-email-gen (-> string/join (gen/fmap (gen/vector gen/char-alphanumeric 1 20)) (gen/bind #(gen/return (str % "@example.com"))))) (gen/generate (params-gen ::user/create)) {:email "[email protected]" :username "Jn*kLqvOXol:u^>XC" :password "y&aOrni}"} derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc

g1. (ns system.test (:require [clojure.test.check.generators :as gen])) (def valid-email-gen (-> string/join (gen/fmap (gen/vector gen/char-alphanumeric 1 20)) (gen/bind #(gen/return (str % "@example.com"))))) (gen/generate (params-gen ::user/create)) {:email "[email protected]" :username "Jn*kLqvOXol:u^>XC" :password "y&aOrni}"} derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc

g1. (defn error? [response] (or (not (<= 200 (:status response) 299)) (exception? response) (timeout? response))) (defn assert-no-error [action] (let [params (gen/generate (params-gen action)) response (exec-action action params)] (is (not (error? response))))) (assert-no-error ::user/create) derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc

g1. (defn error? [response] (or (not (<= 200 (:status response) 299)) (exception? response) (timeout? response))) (defn assert-no-error [action] (let [params (gen/generate (params-gen action)) response (exec-action action params)] (is (not (error? response))))) (assert-no-error ::user/create) derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc

g1. (defn error? [response] (or (not (<= 200 (:status response) 299)) (exception? response) (timeout? response))) (defn assert-no-error [action] (let [params (gen/generate (params-gen action)) response (exec-action action params)] (is (not (error? response))))) (assert-no-error ::user/create) derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc

g2. (defaction user-create :name ::user/create :param-schema s/CreateAccountParam :value-schema s/UserAccount :fn ([{:as param}] ...)) declare action dependencies what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase

g2. (def checkout-dependencies {::user/create [] ::user/authenticate [::user/create] ::user/add-card [::user/authenticate] ::user/update-card [::user/add-card] ::user/get [::user/create] ::cart/add-item [::user/authenticate] ::cart/update-item [::cart/add-item] ::cart/remove-item [::cart/add-item] ::cart/get [::user/authenticate] ::payment/apply-offer [::cart/add-item] ::payment/pay [::user/add-card ::cart/add-item] ::payment/refund [::user/pay] ::payment/cancel [::cart/add-item]}) declare action dependencies what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase

g2. declare action dependencies (def checkout-dependencies {::user/create [] ::user/authenticate [::user/create] ::user/add-card [::user/authenticate] ::user/update-card [::user/add-card] ::user/get [::user/create] ::cart/add-item [::user/authenticate] ::cart/update-item [::cart/add-item] ::cart/remove-item [::cart/add-item] ::cart/get [::user/authenticate] ::payment/apply-offer [::cart/add-item] ::payment/pay [::user/add-card ::cart/add-item] ::payment/refund [::user/pay] ::payment/cancel [::cart/add-item]}) what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase

g2. declare action dependencies (def all-actions #{::user/create ::user/authenticate ::user/add-card ::user/update-card ::user/get ::cart/add-item ::cart/update-item ::cart/remove-item ::cart/get ::payment/apply-offer ::payment/pay ::payment/refund ::payment/cancel}) (for [action actions] (assert-no-error action)) what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase

g3. what action might occur after? probabilistic modelling of usage is effective 0 and 100 are special cases automate the “act” phase probability matrices for flows (def most-users-flow {::user/create {::user/authenticate 90} ::user/authenticate {::user/add-card 20 ::cart/add-item 50} ::user/add-card {::user/update-card 10} ::cart/add-item {::cart/add-item 30 ::cart/update-item 10 ::cart/remove-item 20 ::payment/pay 20} ::payment/pay {::payment/refund 5}}) (def infrequent-users-flow …) (def rich-users-flow ...)

g3. probability matrices for flows (def most-users-flow {::user/create {::user/authenticate 90} ::user/authenticate {::user/add-card 20 ::cart/add-item 50} ::user/add-card {::user/update-card 10} ::cart/add-item {::cart/add-item 30 ::cart/update-item 10 ::cart/remove-item 20 ::payment/pay 20} ::payment/pay {::payment/refund 5}}) (def infrequent-users-flow …) (def rich-users-flow ...) what action might occur after? probabilistic modelling of usage is effective 0 and 100 are special cases automate the “act” phase

g3. what action might occur after? probabilistic modelling of usage is effective 0 and 100 are special cases automate the “act” phase probability matrices for flows (def most-users-flow {::user/create {::user/authenticate 90} ::user/authenticate {::user/add-card 20 ::cart/add-item 50} ::user/add-card {::user/update-card 10} ::cart/add-item {::cart/add-item 30 ::cart/update-item 10 ::cart/remove-item 20 ::payment/pay 20} ::payment/pay {::payment/refund 5}}) (def infrequent-users-flow …) (def rich-users-flow ...)

g3. what action might occur after? probabilistic modelling of usage is effective 0 and 100 are special cases automate the “act” phase probability matrices for flows (->> (action-seq-gen most-users-flow) gen/sample (take 2)) ([::user/create ::user/authenticate ::cart/add-item ::cart/update-item ::cart/update-item ::cart/get ::cart/update-item ::cart/get ::cart/remove-item ::user/add-card ::payment/apply-offer ::payment/apply-offer ::payment/cancel] [::user/create ::user/authenticate ::cart/add-item ::payment/apply-offer ::cart/add-item ::cart/add-item ::cart/add-item ::cart/add-item ::cart/add-item ::cart/get ::cart/update-item])

g3. what action might occur after? probabilistic modelling of usage is effective 0 and 100 are special cases automate the “act” phase probability matrices for flows [{:action ::user/create :params {:email "d3*(@example.com" :username "sdf@37" ...}} {:action ::user/authenticate :params {:username “73@fds” :password "23nv#2&" ...}} {:action ::cart/add-item :params {:sku 143 :qty 1 ...}} ...]

g4. model close-to real scenarios, not completely random ones simplifies debugging gives deterministic deterministic seed data [{:action ::user/create :params {:email "[email protected]" :username "bill" ...}} {:action ::user/authenticate :params {:username "bill" :password "bill_password" ...}} {:action ::user/add-card :params {:card-numer "1234123412341234" :cvc "606" ...}} ...]

g5. software only models subset of domain tests need to model domain around usage model external domain money in user’s card payment gateway latency inventory of supplier

generation execution assertion diagnosis patterns in executing generative tests

1. store everything immutably 2. params as latest values in state 3. catalog known errors 4. abstract request engine patterns in executing generative tests

fn input output action lifecycle e1.

fn input output param-fn generated params state params continue? action lifecycle e1.

fn input output param-fn generated params state params DB continue? store everything e1.

store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (:require [datascript.core :as d]) (defn store-action [flow-id action-id request response entities] (let [metadata {:flow-id flow-id :action-id action-id}] (d/transact! db/conn [(-> request (assoc :data-type :request) (merge metadata))]) (d/transact! db/conn [(-> response (assoc :data-type :response) (merge metadata))]) (d/transact! db/conn [(-> entities compute-state (assoc :data-type :state) (merge metadata))])))

store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (:require [datascript.core :as d]) (defn store-action [flow-id action-id request response entities] (let [metadata {:flow-id flow-id :action-id action-id}] (d/transact! db/conn [(-> request (assoc :data-type :request) (merge metadata))]) (d/transact! db/conn [(-> response (assoc :data-type :response) (merge metadata))]) (d/transact! db/conn [(-> entities compute-state (assoc :data-type :state) (merge metadata))])))

store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn items [flow-id] (->> (d/q '[:find ?sku :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :cart/sku ?sku]] @db/conn flow-id) (map first)))

store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn items [flow-id] (->> (d/q '[:find ?sku :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :cart/sku ?sku]] @db/conn flow-id) (map first)))

store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn params-for [flow-id action] (->> (d/q '[:find ?request :in $ ?flow-id :where [?request :flow-id ?flow-id] [?request :data-type :request] [?request :action action]] @db/conn flow-id) (map first)))

store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn params-for [flow-id action] (->> (d/q '[:find ?request :in $ ?flow-id :where [?request :flow-id ?flow-id] [?request :data-type :request] [?request :action action]] @db/conn flow-id) (map first)))

store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn user-timeline [flow-id user-id] (->> (d/q '[:find ?e ?t :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :user-id ?user-id ?t]] @db/conn flow-id) (sort-by second >)))

store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn user-timeline [flow-id user-id] (->> (d/q '[:find ?e ?t :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :user-id ?user-id ?t]] @db/conn flow-id) (sort-by second >)))

params as latest value most recent value is most relevant use hooks to manipulate engine behaviour e2. (defn latest-param [flow-id attr-name] (->> (d/q '[:find ?attr-val ?t :in $ ?attr-name ?flow-id :where [?as :data-type :action-spec] [?as :flow-id ?flow-id] [?as :action-id ?aid] [?e :action-id ?aid] [?e ?attr-name ?attr-val ?t]] @db/conn attr-name flow-id) (sort-by second >) ffirst))

params as latest value most recent value is most relevant use hooks to manipulate engine behaviour e2. (defn latest-param [flow-id attr-name] (->> (d/q '[:find ?attr-val ?t :in $ ?attr-name ?flow-id :where [?as :data-type :action-spec] [?as :flow-id ?flow-id] [?as :action-id ?aid] [?e :action-id ?aid] [?e ?attr-name ?attr-val ?t]] @db/conn attr-name flow-id) (sort-by second >) ffirst))

fn input output param-fn generated params state params DB continue? store everything e1.

params as latest value most recent value is most relevant use hooks to manipulate engine behaviour e2. {::user/create {:entity-type :user :params-fn (fn [flow-id params] (assoc params :name (str "gen-user-" (u/uuid))))} ::payment/pay {:entity-type :payment :params-fn (fn [flow-id params] (->> (p/fetch-param flow-id :default-card) (assoc params :card-number)))}}

catalog known errors improve relevance reduce redundancy absorb into system e3. {::user/create {:entity-type :user :known-exceptions #{:duplicate-user} :params-fn (fn ...)} ::payment/pay {:entity-type :payment :known-exceptions #{:inactive-session :payment-complete} :params-fn (fn ...)}}

catalog known errors improve relevance reduce redundancy absorb into system e3. {::user/create {:entity-type :user :known-exceptions #{:duplicate-user} :params-fn (fn ...)} ::payment/pay {:entity-type :payment :known-exceptions #{:inactive-session :payment-complete} :params-fn (fn ...)}}

catalog known errors improve relevance reduce redundancy absorb into system e3. (def known-errors {:duplicate-user (fn [flow-id params] (some? (exec-action ::user/get (:username params)))) :payment-complete (fn [flow-id params] (some? (db/find-action flow-id ::payment/pay)))})

abstract request engine quickly switch to higher or lower level of specificity api vs controller makes debugging simpler e4. (defn make-request [medium action params] (case medium :fn ((:fn action) params) :bus (bus/request-value action params) :http (-> action http-request (client/request params))))

generation execution assertion diagnosis patterns in assertions

1. algebraic properties 2. state machines as properties 3. domain based invariants patterns in assertions

property unit (no order) integration / system (ordered) no-error* ✓ ✓ egality ✗ ✓ idempotence ⍻ ✓ identity ✓ ✗ inverse ✗ ✓ commutativity ✗ ✓ associativity ✗ ✓ algebraic properties a1.

algebraic properties no-error {:message "No :error" :result (not-any? #(= :error %) (map :status responses))} ;; status (if (or (not (<= 200 status 299)) (some? exception) (timeout?)) :error) no-exceptions no-timeouts always-available a1.

algebraic properties egality (assert-equal [{:action ::user/signup}] [{:action ::user/get}]) (assert-equal [{:action ::cart/update-item :params {:deleted? true}}] [{:action ::cart/remove-item}]) f(a) ≡ g(b) effective equality different paths, same result a1.

algebraic properties idempotence {:percent-flow 10 :adjacency {:immediate 10 :distant 90} :blacklist #{::payment/pay ::payment/refund ::payment/cancel ::user/signup}} f(a) · f(a) ≡ f(a) 2 kinds of adjacencies blacklist / whitelist sensitive / distributed systems a b c c d e a b c d c e immediate distant a1.

algebraic properties inverse (assert-equal [{:action ::cart/add-item}] [{:action ::cart/remove-item}] :value-action ::cart/get) ;; normalised (assert-equal [{:action ::payment/pay} {:action ::payment/cancel} {:action ::payment/pay}] [{:action ::payment/pay}] :value-action ::payment/receipt) f(a) · f -1(a) ≡ a f -1(f(f -1(a)))=f -1(a) normalised inverse a1.

algebraic properties commutativity (assert-equal [{:action ::cart/add-item :params {:sku 1}} {:action ::cart/add-item :params {:sku 2}}] [{:action ::cart/add-item :params {:sku 2}} {:action ::cart/add-item :params {:sku 1}}] :value-action ::cart/get) f(a) · g(a) ≡ g(a) · f(a) non-compliance is a property too a1.

algebraic properties commutativity (assert-commutative [{:action ::cart/add-item :params {:sku 1}} {:action ::cart/add-item :params {:sku 2}}] :value-action ::cart/get) (assert-not-commutative [{:action ::cart/update-item :params {:sku 1 :qty 1}} {:action ::cart/update-item :params {:sku 1 :qty 6}}] :value-action ::cart/get) f(a) · g(a) ≡ g(a) · f(a) non-compliance is a property too a1.

state machines as properties new has-items success cancelled refunded authenticate add-item update-item remove-item apply-offer pay refund cancel remove-item a2.

(defn compute-expected-state [state action status] (match [state action status] [_ ::user/authenticate :ok] "new" ["new" ::cart/add-item :ok] "has-items" ["has-items" ::cart/update-item :ok] "has-items" ["has-items" ::cart/remove-item :ok] "has-items" ["has-items" ::cart/remove-item :ok] "new" ["has-items" ::payment/apply-offer :ok] "has-items" ["has-items" ::payment/pay :ok] "success" ["success" ::payment/refund :ok] "refunded" ["has-items" ::payment/cancel :ok] "cancelled" [_ _ :error] "error" [_ _ :known-exception] "known-exception")) a2.

domain based invariants for completeness create-count-is-count-of-list lookups-return-the-same-thing num-items-sold-matches-inventory receipt-totals-equal-profit a3.

generation execution assertion diagnosis patterns in diagnosis

1. tests as portable data 2. domain based checkpoints 3. flow timeline, and walk throughs 4. automated bug reports patterns in diagnosis

tests as portable data flow / action-specs are stored tests determinism 㱺 reproducibility store passing and failing tests version tests d1. [{:action ::user/create :params {:email "d3*(@example.com" :username "sdf@37" ...}} {:action ::user/authenticate :params {:username “73@fds” :password "23nv#2&" ...}} {:action ::cart/add-item :params {:sku 143 :qty 1 ...}} ...]

tests as portable data flow / action-specs are stored tests determinism 㱺 reproducibility store passing and failing tests version tests d1. (-> "rc42-login-bug.edn" test/import test/run) (-> "v42-regression-test.edn" test/import test/run)

domain based checkpoints think of code as a trie useful when there are 100s of flows as on CI d2. {:checkpoints [{:action ::user/authenticate :description "Logged in"} {:action ::cart/add-item :description "Added item to cart"} {:action ::payment/pay :description "Payment attempted"}]}

flow timeline, walkthroughs d3. | action | status | action-id | time | |-----------------------+--------+-------------------+------------| | ::user/create | :ok | 1d8040ee-2486-... | 1524287925 | | ::user/authenticate | :ok | 92c6fb41-ad78-... | 1524287925 | | ::cart/add-item | :ok | 20c55130-85d2-... | 1524287927 | | ::payment/apply-offer | :ok | 239ebf75-a6c4-... | 1524287928 | | ::cart/add-item | :ok | d991dc62-5322-... | 1524287928 | | ::cart/add-item | :ok | f4b02a65-ac8f-... | 1524287928 | | ::cart/add-item | :ok | e0b85377-b6b7-... | 1524287931 | | ::cart/add-item | :ok | cb676220-8d6a-... | 1524287942 | | ::cart/add-item | :ok | 1d9546fb-7356-... | 1524287942 | | ::cart/get | :ok | b282a885-6e32-... | 1524287942 | | ::cart/update-item | :ex | ed062dca-f457-... | 1524287943 | timeline of actions relating to a user step-by-step diagnosis of events everything has an id, and a timestamp

flow timeline, walkthroughs d3. (dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6") {:metadata {:action ::cart/update-item :flow-id "35cb3104f-e0b0-..." :action-id "1d9546fb-7356-..."}} [{:data-type :action-spec :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :request :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :response :response {:status :error, :error-type java.lang.NullPointerException, :report {:cause #error { :cause nil :via [{:type java.lang.NullPointerException :message nil :at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}] :trace [[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77] [demo-proj.modules.cart$extract_filename invoke "cart.clj" 74] [demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139] [demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134] [clojure.lang.Var invoke "Var.java" 379] [demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke "logging.clj" 11] [intbus.core$invoke_handler_fn invokeStatic "core.clj" 134] [intbus.core$invoke_handler_fn invoke "core.clj" 129] [intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}] timeline of actions relating to a user step-by-step diagnosis of events everything has an id, and a timestamp

flow timeline, walkthroughs d3. (dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6") {:metadata {:action ::cart/update-item :flow-id "35cb3104f-e0b0-..." :action-id "1d9546fb-7356-..."}} [{:data-type :action-spec :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :request :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :response :response {:status :error, :error-type java.lang.NullPointerException, :report {:cause #error { :cause nil :via [{:type java.lang.NullPointerException :message nil :at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}] :trace [[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77] [demo-proj.modules.cart$extract_filename invoke "cart.clj" 74] [demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139] [demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134] [clojure.lang.Var invoke "Var.java" 379] [demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke "logging.clj" 11] [intbus.core$invoke_handler_fn invokeStatic "core.clj" 134] [intbus.core$invoke_handler_fn invoke "core.clj" 129] [intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}] timeline of actions relating to a user step-by-step diagnosis of events everything has an id, and a timestamp

flow timeline, walkthroughs d3. (dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6") {:metadata {:action ::cart/update-item :flow-id "35cb3104f-e0b0-..." :action-id "1d9546fb-7356-..."}} [{:data-type :action-spec :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :request :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :response :response {:status :error, :error-type java.lang.NullPointerException, :report {:cause #error { :cause nil :via [{:type java.lang.NullPointerException :message nil :at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}] :trace [[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77] [demo-proj.modules.cart$extract_filename invoke "cart.clj" 74] [demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139] [demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134] [clojure.lang.Var invoke "Var.java" 379] [demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke "logging.clj" 11] [intbus.core$invoke_handler_fn invokeStatic "core.clj" 134] [intbus.core$invoke_handler_fn invoke "core.clj" 129] [intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}] timeline of actions relating to a user step-by-step diagnosis of events everything has an id, and a timestamp

automated bug reports d4. write code to summarise events create issues from CI to github

automated bug reports d4. write code to summarise events create issues from CI to github

1. derive parameter specification 2. declare action dependencies 3. probability matrices for flows 4. deterministic seed data 5. model external domain 1. store everything immutably 2. params as latest values in state 3. catalog known errors 4. abstract request engine 1. algebraic properties 2. state machines as properties 3. domain based invariants 1. tests as portable data 2. domain based checkpoints 3. flow timeline, and walk throughs 4. automated bug reports generation assertion execution diagnosis

a thought paradigm automating software testing patterns

references microsoft PEX patterns datomic/simulant

practical generative testing patterns srihari sriraman | ø nilenso