Practical Generative Testing Patterns

Practical Generative Testing Patterns

Talk slides as presented at Dutch Clojure Days 2018

6389f3db059111f68daa44ab6d01a1bd?s=128

Srihari Sriraman

April 21, 2018
Tweet

Transcript

  1. practical generative testing patterns srihari sriraman | ø nilenso

  2. about this talk why, what a practical system benefits and

    problems terminology patterns
  3. why

  4. tests are good why don’t we write them often enough?

    generative tests are better
  5. what

  6. a thought paradigm automating software testing patterns

  7. a practical system

  8. practical system specimen user signup/create authenticate add/update-card get cart add/update-item

    remove-item get payment pay refund/cancel apply-offer receipt a generic e-commerce system
  9. practical system specimen a generic e-commerce system create (signup) authenticate

    (login) add-card add-item pay update-item
  10. benefits and problems

  11. problems inertia, learning curve thinking in properties slow, and ineffective

    high maintenance difficult to diagnose benefits better than humans at input generation more tests, more coverage better chance at finding bugs before users do
  12. problems inertia, learning curve thinking in properties slow, and ineffective

    high maintenance difficult to diagnose benefits better than humans at input generation more tests, more coverage better chance at finding bugs before users do
  13. problems inertia, learning curve thinking in properties slow, and ineffective

    high maintenance difficult to diagnose inertia John Hughes – Testing the Hard Stuff and Staying Sane
  14. learning curve Gary Fredericks – Building test check Generators problems

    inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
  15. thinking in properties purely functional data algorithms or data structures

    are relatively easy to test most practical systems involve making a series of stateful api calls, most of which are CRUD problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
  16. slow and ineffective CI duration grows exponentially bugs found in

    test failures aren’t necessarily relevant more tests ≠ more coverage problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
  17. high maintenance generally more complex to write and read arrange,

    and act portions of test are still dependent on source problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
  18. difficult to diagnose automatic shrinking really helps, but only until

    a point complex, stateful tests aren’t necessarily deterministic, and are difficult to reproduce finding root-cause of failure without the right tools can be very difficult problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
  19. terminology

  20. phase of testing level of testing focus of testing segments

  21. assume arrange act assert phase level focus

  22. generation execution assertion diagnosis phase level focus

  23. System Integration Unit + Value + Speed phase level focus

  24. System Integration Unit + Value + Speed phase level focus

  25. System phase level focus

  26. correctness (smoke, acceptance, verification) regression (functional, quality control) performance (load,

    stress, endurance, spike) phase level focus
  27. system under test fn input fn argument request state output

    return value response state state one fn many fns entire system
  28. action under test fn input fn argument request state output

    return value response state state one fn many fns entire system
  29. automation QA Engineer Test Developer

  30. patterns

  31. pattern /ˈpat(ə)n/ a regular and intelligible form or sequence discernible

    in the way in which something happens or is done
  32. generation execution assertion diagnosis patterns in generating input

  33. 1. derive parameter specification 2. declare action dependencies 3. probability

    matrices for flows 4. deterministic seed data 5. model external domain patterns in generating input
  34. derive param specification g1. (ns system.schema) (defschema CreateAccountParam {:email (s/maybe

    valid-email) :username non-empty-string :password non-empty-string}) (defschema UserAccount {:id String :email String :username String}) (ns system.account) (defaction user-create :name ::user/create :param-schema s/CreateAccountParam :value-schema s/UserAccount :fn ([{:as param}] ...)) param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
  35. derive param specification g1. (ns system.schema) (defschema CreateAccountParam {:email (s/maybe

    valid-email) :username non-empty-string :password non-empty-string}) (defschema UserAccount {:id String :email String :username String}) (ns system.account) (defaction user-create :name ::user/create :param-schema s/CreateAccountParam :value-schema s/UserAccount :fn ([{:as param}] ...)) param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
  36. g1. (ns system.test (:require [clojure.test.check.generators :as gen])) (def valid-email-gen (->

    string/join (gen/fmap (gen/vector gen/char-alphanumeric 1 20)) (gen/bind #(gen/return (str % "@example.com"))))) (gen/generate (params-gen ::user/create)) {:email "9be2PgzoX00S@example.com" :username "Jn*kLqvOXol:u^>XC" :password "y&aOrni}"} derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
  37. g1. (ns system.test (:require [clojure.test.check.generators :as gen])) (def valid-email-gen (->

    string/join (gen/fmap (gen/vector gen/char-alphanumeric 1 20)) (gen/bind #(gen/return (str % "@example.com"))))) (gen/generate (params-gen ::user/create)) {:email "9be2PgzoX00S@example.com" :username "Jn*kLqvOXol:u^>XC" :password "y&aOrni}"} derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
  38. g1. (defn error? [response] (or (not (<= 200 (:status response)

    299)) (exception? response) (timeout? response))) (defn assert-no-error [action] (let [params (gen/generate (params-gen action)) response (exec-action action params)] (is (not (error? response))))) (assert-no-error ::user/create) derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
  39. g1. (defn error? [response] (or (not (<= 200 (:status response)

    299)) (exception? response) (timeout? response))) (defn assert-no-error [action] (let [params (gen/generate (params-gen action)) response (exec-action action params)] (is (not (error? response))))) (assert-no-error ::user/create) derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
  40. g1. (defn error? [response] (or (not (<= 200 (:status response)

    299)) (exception? response) (timeout? response))) (defn assert-no-error [action] (let [params (gen/generate (params-gen action)) response (exec-action action params)] (is (not (error? response))))) (assert-no-error ::user/create) derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
  41. g2. (defaction user-create :name ::user/create :param-schema s/CreateAccountParam :value-schema s/UserAccount :fn

    ([{:as param}] ...)) declare action dependencies what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase
  42. g2. (def checkout-dependencies {::user/create [] ::user/authenticate [::user/create] ::user/add-card [::user/authenticate] ::user/update-card

    [::user/add-card] ::user/get [::user/create] ::cart/add-item [::user/authenticate] ::cart/update-item [::cart/add-item] ::cart/remove-item [::cart/add-item] ::cart/get [::user/authenticate] ::payment/apply-offer [::cart/add-item] ::payment/pay [::user/add-card ::cart/add-item] ::payment/refund [::user/pay] ::payment/cancel [::cart/add-item]}) declare action dependencies what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase
  43. g2. declare action dependencies (def checkout-dependencies {::user/create [] ::user/authenticate [::user/create]

    ::user/add-card [::user/authenticate] ::user/update-card [::user/add-card] ::user/get [::user/create] ::cart/add-item [::user/authenticate] ::cart/update-item [::cart/add-item] ::cart/remove-item [::cart/add-item] ::cart/get [::user/authenticate] ::payment/apply-offer [::cart/add-item] ::payment/pay [::user/add-card ::cart/add-item] ::payment/refund [::user/pay] ::payment/cancel [::cart/add-item]}) what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase
  44. g2. declare action dependencies (def all-actions #{::user/create ::user/authenticate ::user/add-card ::user/update-card

    ::user/get ::cart/add-item ::cart/update-item ::cart/remove-item ::cart/get ::payment/apply-offer ::payment/pay ::payment/refund ::payment/cancel}) (for [action actions] (assert-no-error action)) what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase
  45. g3. what action might occur after? probabilistic modelling of usage

    is effective 0 and 100 are special cases automate the “act” phase probability matrices for flows (def most-users-flow {::user/create {::user/authenticate 90} ::user/authenticate {::user/add-card 20 ::cart/add-item 50} ::user/add-card {::user/update-card 10} ::cart/add-item {::cart/add-item 30 ::cart/update-item 10 ::cart/remove-item 20 ::payment/pay 20} ::payment/pay {::payment/refund 5}}) (def infrequent-users-flow …) (def rich-users-flow ...)
  46. g3. probability matrices for flows (def most-users-flow {::user/create {::user/authenticate 90}

    ::user/authenticate {::user/add-card 20 ::cart/add-item 50} ::user/add-card {::user/update-card 10} ::cart/add-item {::cart/add-item 30 ::cart/update-item 10 ::cart/remove-item 20 ::payment/pay 20} ::payment/pay {::payment/refund 5}}) (def infrequent-users-flow …) (def rich-users-flow ...) what action might occur after? probabilistic modelling of usage is effective 0 and 100 are special cases automate the “act” phase
  47. g3. what action might occur after? probabilistic modelling of usage

    is effective 0 and 100 are special cases automate the “act” phase probability matrices for flows (def most-users-flow {::user/create {::user/authenticate 90} ::user/authenticate {::user/add-card 20 ::cart/add-item 50} ::user/add-card {::user/update-card 10} ::cart/add-item {::cart/add-item 30 ::cart/update-item 10 ::cart/remove-item 20 ::payment/pay 20} ::payment/pay {::payment/refund 5}}) (def infrequent-users-flow …) (def rich-users-flow ...)
  48. g3. what action might occur after? probabilistic modelling of usage

    is effective 0 and 100 are special cases automate the “act” phase probability matrices for flows (->> (action-seq-gen most-users-flow) gen/sample (take 2)) ([::user/create ::user/authenticate ::cart/add-item ::cart/update-item ::cart/update-item ::cart/get ::cart/update-item ::cart/get ::cart/remove-item ::user/add-card ::payment/apply-offer ::payment/apply-offer ::payment/cancel] [::user/create ::user/authenticate ::cart/add-item ::payment/apply-offer ::cart/add-item ::cart/add-item ::cart/add-item ::cart/add-item ::cart/add-item ::cart/get ::cart/update-item])
  49. g3. what action might occur after? probabilistic modelling of usage

    is effective 0 and 100 are special cases automate the “act” phase probability matrices for flows [{:action ::user/create :params {:email "d3*(@example.com" :username "sdf@37" ...}} {:action ::user/authenticate :params {:username “73@fds” :password "23nv#2&" ...}} {:action ::cart/add-item :params {:sku 143 :qty 1 ...}} ...]
  50. g4. model close-to real scenarios, not completely random ones simplifies

    debugging gives deterministic deterministic seed data [{:action ::user/create :params {:email "bezos@amazon.com" :username "bill" ...}} {:action ::user/authenticate :params {:username "bill" :password "bill_password" ...}} {:action ::user/add-card :params {:card-numer "1234123412341234" :cvc "606" ...}} ...]
  51. g5. software only models subset of domain tests need to

    model domain around usage model external domain money in user’s card payment gateway latency inventory of supplier
  52. generation execution assertion diagnosis patterns in executing generative tests

  53. 1. store everything immutably 2. params as latest values in

    state 3. catalog known errors 4. abstract request engine patterns in executing generative tests
  54. fn input output action lifecycle e1.

  55. fn input output param-fn generated params state params continue? action

    lifecycle e1.
  56. fn input output param-fn generated params state params DB continue?

    store everything e1.
  57. store everything immutably e1. ability to query for anything during

    execution, assertion or diagnosis tests need to be completely independent of source timeline matters (:require [datascript.core :as d]) (defn store-action [flow-id action-id request response entities] (let [metadata {:flow-id flow-id :action-id action-id}] (d/transact! db/conn [(-> request (assoc :data-type :request) (merge metadata))]) (d/transact! db/conn [(-> response (assoc :data-type :response) (merge metadata))]) (d/transact! db/conn [(-> entities compute-state (assoc :data-type :state) (merge metadata))])))
  58. store everything immutably e1. ability to query for anything during

    execution, assertion or diagnosis tests need to be completely independent of source timeline matters (:require [datascript.core :as d]) (defn store-action [flow-id action-id request response entities] (let [metadata {:flow-id flow-id :action-id action-id}] (d/transact! db/conn [(-> request (assoc :data-type :request) (merge metadata))]) (d/transact! db/conn [(-> response (assoc :data-type :response) (merge metadata))]) (d/transact! db/conn [(-> entities compute-state (assoc :data-type :state) (merge metadata))])))
  59. store everything immutably e1. ability to query for anything during

    execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn items [flow-id] (->> (d/q '[:find ?sku :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :cart/sku ?sku]] @db/conn flow-id) (map first)))
  60. store everything immutably e1. ability to query for anything during

    execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn items [flow-id] (->> (d/q '[:find ?sku :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :cart/sku ?sku]] @db/conn flow-id) (map first)))
  61. store everything immutably e1. ability to query for anything during

    execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn params-for [flow-id action] (->> (d/q '[:find ?request :in $ ?flow-id :where [?request :flow-id ?flow-id] [?request :data-type :request] [?request :action action]] @db/conn flow-id) (map first)))
  62. store everything immutably e1. ability to query for anything during

    execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn params-for [flow-id action] (->> (d/q '[:find ?request :in $ ?flow-id :where [?request :flow-id ?flow-id] [?request :data-type :request] [?request :action action]] @db/conn flow-id) (map first)))
  63. store everything immutably e1. ability to query for anything during

    execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn user-timeline [flow-id user-id] (->> (d/q '[:find ?e ?t :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :user-id ?user-id ?t]] @db/conn flow-id) (sort-by second >)))
  64. store everything immutably e1. ability to query for anything during

    execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn user-timeline [flow-id user-id] (->> (d/q '[:find ?e ?t :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :user-id ?user-id ?t]] @db/conn flow-id) (sort-by second >)))
  65. params as latest value most recent value is most relevant

    use hooks to manipulate engine behaviour e2. (defn latest-param [flow-id attr-name] (->> (d/q '[:find ?attr-val ?t :in $ ?attr-name ?flow-id :where [?as :data-type :action-spec] [?as :flow-id ?flow-id] [?as :action-id ?aid] [?e :action-id ?aid] [?e ?attr-name ?attr-val ?t]] @db/conn attr-name flow-id) (sort-by second >) ffirst))
  66. params as latest value most recent value is most relevant

    use hooks to manipulate engine behaviour e2. (defn latest-param [flow-id attr-name] (->> (d/q '[:find ?attr-val ?t :in $ ?attr-name ?flow-id :where [?as :data-type :action-spec] [?as :flow-id ?flow-id] [?as :action-id ?aid] [?e :action-id ?aid] [?e ?attr-name ?attr-val ?t]] @db/conn attr-name flow-id) (sort-by second >) ffirst))
  67. fn input output param-fn generated params state params DB continue?

    store everything e1.
  68. params as latest value most recent value is most relevant

    use hooks to manipulate engine behaviour e2. {::user/create {:entity-type :user :params-fn (fn [flow-id params] (assoc params :name (str "gen-user-" (u/uuid))))} ::payment/pay {:entity-type :payment :params-fn (fn [flow-id params] (->> (p/fetch-param flow-id :default-card) (assoc params :card-number)))}}
  69. catalog known errors improve relevance reduce redundancy absorb into system

    e3. {::user/create {:entity-type :user :known-exceptions #{:duplicate-user} :params-fn (fn ...)} ::payment/pay {:entity-type :payment :known-exceptions #{:inactive-session :payment-complete} :params-fn (fn ...)}}
  70. catalog known errors improve relevance reduce redundancy absorb into system

    e3. {::user/create {:entity-type :user :known-exceptions #{:duplicate-user} :params-fn (fn ...)} ::payment/pay {:entity-type :payment :known-exceptions #{:inactive-session :payment-complete} :params-fn (fn ...)}}
  71. catalog known errors improve relevance reduce redundancy absorb into system

    e3. (def known-errors {:duplicate-user (fn [flow-id params] (some? (exec-action ::user/get (:username params)))) :payment-complete (fn [flow-id params] (some? (db/find-action flow-id ::payment/pay)))})
  72. abstract request engine quickly switch to higher or lower level

    of specificity api vs controller makes debugging simpler e4. (defn make-request [medium action params] (case medium :fn ((:fn action) params) :bus (bus/request-value action params) :http (-> action http-request (client/request params))))
  73. generation execution assertion diagnosis patterns in assertions

  74. 1. algebraic properties 2. state machines as properties 3. domain

    based invariants patterns in assertions
  75. property unit (no order) integration / system (ordered) no-error* ✓

    ✓ egality ✗ ✓ idempotence ⍻ ✓ identity ✓ ✗ inverse ✗ ✓ commutativity ✗ ✓ associativity ✗ ✓ algebraic properties a1.
  76. algebraic properties no-error {:message "No :error" :result (not-any? #(= :error

    %) (map :status responses))} ;; status (if (or (not (<= 200 status 299)) (some? exception) (timeout?)) :error) no-exceptions no-timeouts always-available a1.
  77. algebraic properties egality (assert-equal [{:action ::user/signup}] [{:action ::user/get}]) (assert-equal [{:action

    ::cart/update-item :params {:deleted? true}}] [{:action ::cart/remove-item}]) f(a) ≡ g(b) effective equality different paths, same result a1.
  78. algebraic properties idempotence {:percent-flow 10 :adjacency {:immediate 10 :distant 90}

    :blacklist #{::payment/pay ::payment/refund ::payment/cancel ::user/signup}} f(a) · f(a) ≡ f(a) 2 kinds of adjacencies blacklist / whitelist sensitive / distributed systems a b c c d e a b c d c e immediate distant a1.
  79. algebraic properties inverse (assert-equal [{:action ::cart/add-item}] [{:action ::cart/remove-item}] :value-action ::cart/get)

    ;; normalised (assert-equal [{:action ::payment/pay} {:action ::payment/cancel} {:action ::payment/pay}] [{:action ::payment/pay}] :value-action ::payment/receipt) f(a) · f -1(a) ≡ a f -1(f(f -1(a)))=f -1(a) normalised inverse a1.
  80. algebraic properties commutativity (assert-equal [{:action ::cart/add-item :params {:sku 1}} {:action

    ::cart/add-item :params {:sku 2}}] [{:action ::cart/add-item :params {:sku 2}} {:action ::cart/add-item :params {:sku 1}}] :value-action ::cart/get) f(a) · g(a) ≡ g(a) · f(a) non-compliance is a property too a1.
  81. algebraic properties commutativity (assert-commutative [{:action ::cart/add-item :params {:sku 1}} {:action

    ::cart/add-item :params {:sku 2}}] :value-action ::cart/get) (assert-not-commutative [{:action ::cart/update-item :params {:sku 1 :qty 1}} {:action ::cart/update-item :params {:sku 1 :qty 6}}] :value-action ::cart/get) f(a) · g(a) ≡ g(a) · f(a) non-compliance is a property too a1.
  82. state machines as properties new has-items success cancelled refunded authenticate

    add-item update-item remove-item apply-offer pay refund cancel remove-item a2.
  83. (defn compute-expected-state [state action status] (match [state action status] [_

    ::user/authenticate :ok] "new" ["new" ::cart/add-item :ok] "has-items" ["has-items" ::cart/update-item :ok] "has-items" ["has-items" ::cart/remove-item :ok] "has-items" ["has-items" ::cart/remove-item :ok] "new" ["has-items" ::payment/apply-offer :ok] "has-items" ["has-items" ::payment/pay :ok] "success" ["success" ::payment/refund :ok] "refunded" ["has-items" ::payment/cancel :ok] "cancelled" [_ _ :error] "error" [_ _ :known-exception] "known-exception")) a2.
  84. domain based invariants for completeness create-count-is-count-of-list lookups-return-the-same-thing num-items-sold-matches-inventory receipt-totals-equal-profit a3.

  85. generation execution assertion diagnosis patterns in diagnosis

  86. 1. tests as portable data 2. domain based checkpoints 3.

    flow timeline, and walk throughs 4. automated bug reports patterns in diagnosis
  87. tests as portable data flow / action-specs are stored tests

    determinism 㱺 reproducibility store passing and failing tests version tests d1. [{:action ::user/create :params {:email "d3*(@example.com" :username "sdf@37" ...}} {:action ::user/authenticate :params {:username “73@fds” :password "23nv#2&" ...}} {:action ::cart/add-item :params {:sku 143 :qty 1 ...}} ...]
  88. tests as portable data flow / action-specs are stored tests

    determinism 㱺 reproducibility store passing and failing tests version tests d1. (-> "rc42-login-bug.edn" test/import test/run) (-> "v42-regression-test.edn" test/import test/run)
  89. domain based checkpoints think of code as a trie useful

    when there are 100s of flows as on CI d2. {:checkpoints [{:action ::user/authenticate :description "Logged in"} {:action ::cart/add-item :description "Added item to cart"} {:action ::payment/pay :description "Payment attempted"}]}
  90. flow timeline, walkthroughs d3. | action | status | action-id

    | time | |-----------------------+--------+-------------------+------------| | ::user/create | :ok | 1d8040ee-2486-... | 1524287925 | | ::user/authenticate | :ok | 92c6fb41-ad78-... | 1524287925 | | ::cart/add-item | :ok | 20c55130-85d2-... | 1524287927 | | ::payment/apply-offer | :ok | 239ebf75-a6c4-... | 1524287928 | | ::cart/add-item | :ok | d991dc62-5322-... | 1524287928 | | ::cart/add-item | :ok | f4b02a65-ac8f-... | 1524287928 | | ::cart/add-item | :ok | e0b85377-b6b7-... | 1524287931 | | ::cart/add-item | :ok | cb676220-8d6a-... | 1524287942 | | ::cart/add-item | :ok | 1d9546fb-7356-... | 1524287942 | | ::cart/get | :ok | b282a885-6e32-... | 1524287942 | | ::cart/update-item | :ex | ed062dca-f457-... | 1524287943 | timeline of actions relating to a user step-by-step diagnosis of events everything has an id, and a timestamp
  91. flow timeline, walkthroughs d3. (dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6") {:metadata {:action ::cart/update-item :flow-id

    "35cb3104f-e0b0-..." :action-id "1d9546fb-7356-..."}} [{:data-type :action-spec :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :request :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :response :response {:status :error, :error-type java.lang.NullPointerException, :report {:cause #error { :cause nil :via [{:type java.lang.NullPointerException :message nil :at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}] :trace [[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77] [demo-proj.modules.cart$extract_filename invoke "cart.clj" 74] [demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139] [demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134] [clojure.lang.Var invoke "Var.java" 379] [demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke "logging.clj" 11] [intbus.core$invoke_handler_fn invokeStatic "core.clj" 134] [intbus.core$invoke_handler_fn invoke "core.clj" 129] [intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}] timeline of actions relating to a user step-by-step diagnosis of events everything has an id, and a timestamp
  92. flow timeline, walkthroughs d3. (dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6") {:metadata {:action ::cart/update-item :flow-id

    "35cb3104f-e0b0-..." :action-id "1d9546fb-7356-..."}} [{:data-type :action-spec :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :request :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :response :response {:status :error, :error-type java.lang.NullPointerException, :report {:cause #error { :cause nil :via [{:type java.lang.NullPointerException :message nil :at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}] :trace [[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77] [demo-proj.modules.cart$extract_filename invoke "cart.clj" 74] [demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139] [demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134] [clojure.lang.Var invoke "Var.java" 379] [demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke "logging.clj" 11] [intbus.core$invoke_handler_fn invokeStatic "core.clj" 134] [intbus.core$invoke_handler_fn invoke "core.clj" 129] [intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}] timeline of actions relating to a user step-by-step diagnosis of events everything has an id, and a timestamp
  93. flow timeline, walkthroughs d3. (dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6") {:metadata {:action ::cart/update-item :flow-id

    "35cb3104f-e0b0-..." :action-id "1d9546fb-7356-..."}} [{:data-type :action-spec :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :request :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :response :response {:status :error, :error-type java.lang.NullPointerException, :report {:cause #error { :cause nil :via [{:type java.lang.NullPointerException :message nil :at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}] :trace [[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77] [demo-proj.modules.cart$extract_filename invoke "cart.clj" 74] [demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139] [demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134] [clojure.lang.Var invoke "Var.java" 379] [demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke "logging.clj" 11] [intbus.core$invoke_handler_fn invokeStatic "core.clj" 134] [intbus.core$invoke_handler_fn invoke "core.clj" 129] [intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}] timeline of actions relating to a user step-by-step diagnosis of events everything has an id, and a timestamp
  94. automated bug reports d4. write code to summarise events create

    issues from CI to github
  95. automated bug reports d4. write code to summarise events create

    issues from CI to github
  96. 1. derive parameter specification 2. declare action dependencies 3. probability

    matrices for flows 4. deterministic seed data 5. model external domain 1. store everything immutably 2. params as latest values in state 3. catalog known errors 4. abstract request engine 1. algebraic properties 2. state machines as properties 3. domain based invariants 1. tests as portable data 2. domain based checkpoints 3. flow timeline, and walk throughs 4. automated bug reports generation assertion execution diagnosis
  97. a thought paradigm automating software testing patterns

  98. references microsoft PEX patterns datomic/simulant

  99. practical generative testing patterns srihari sriraman | ø nilenso