Practical Generative Testing Patterns

Talk slides as presented at Dutch Clojure Days 2018

Transcript

1. practical generative testing patterns srihari sriraman | ø nilenso
2. about this talk why, what a practical system benefits and problems terminology patterns
3. why
4. tests are good why don’t we write them often enough? generative tests are better
5. what
6. a thought paradigm automating software testing patterns
7. a practical system
8. practical system specimen user signup/create authenticate add/update-card get cart add/update-item remove-item get payment pay refund/cancel apply-offer receipt a generic e-commerce system
9. practical system specimen a generic e-commerce system create (signup) authenticate (login) add-card add-item pay update-item
10. benefits and problems
11. problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose benefits better than humans at input generation more tests, more coverage better chance at finding bugs before users do
12. problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose benefits better than humans at input generation more tests, more coverage better chance at finding bugs before users do
13. problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose inertia John Hughes – Testing the Hard Stuff and Staying Sane
14. learning curve Gary Fredericks – Building test check Generators problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
15. thinking in properties purely functional data algorithms or data structures are relatively easy to test most practical systems involve making a series of stateful api calls, most of which are CRUD problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
16. slow and ineffective CI duration grows exponentially bugs found in test failures aren’t necessarily relevant more tests ≠ more coverage problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
17. high maintenance generally more complex to write and read arrange, and act portions of test are still dependent on source problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
18. difficult to diagnose automatic shrinking really helps, but only until a point complex, stateful tests aren’t necessarily deterministic, and are difficult to reproduce finding root-cause of failure without the right tools can be very difficult problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
19. terminology
20. phase of testing level of testing focus of testing segments
21. assume arrange act assert phase level focus
22. generation execution assertion diagnosis phase level focus
23. System Integration Unit + Value + Speed phase level focus
24. System Integration Unit + Value + Speed phase level focus
25. System phase level focus
26. correctness (smoke, acceptance, verification) regression (functional, quality control) performance (load, stress, endurance, spike) phase level focus
27. system under test fn input fn argument request state output return value response state state one fn many fns entire system
28. action under test fn input fn argument request state output return value response state state one fn many fns entire system
29. automation QA Engineer Test Developer
30. patterns
31. pattern /ˈpat(ə)n/ a regular and intelligible form or sequence discernible in the way in which something happens or is done
32. generation execution assertion diagnosis patterns in generating input
33. 1. derive parameter specification 2. declare action dependencies 3. probability matrices for flows 4. deterministic seed data 5. model external domain patterns in generating input
34. derive param specification g1. (ns system.schema) (defschema CreateAccountParam {:email (s/maybe valid-email) :username non-empty-string :password non-empty-string}) (defschema UserAccount {:id String :email String :username String}) (ns system.account) (defaction user-create :name ::user/create :param-schema s/CreateAccountParam :value-schema s/UserAccount :fn ([{:as param}] ...)) param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
35. derive param specification g1. (ns system.schema) (defschema CreateAccountParam {:email (s/maybe valid-email) :username non-empty-string :password non-empty-string}) (defschema UserAccount {:id String :email String :username String}) (ns system.account) (defaction user-create :name ::user/create :param-schema s/CreateAccountParam :value-schema s/UserAccount :fn ([{:as param}] ...)) param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
36. g1. (ns system.test (:require [clojure.test.check.generators :as gen])) (def valid-email-gen (-> string/join (gen/fmap (gen/vector gen/char-alphanumeric 1 20)) (gen/bind #(gen/return (str % "@example.com"))))) (gen/generate (params-gen ::user/create)) {:email "9be2PgzoX00S@example.com" :username "Jn*kLqvOXol:u^>XC" :password "y&aOrni}"} derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
37. g1. (ns system.test (:require [clojure.test.check.generators :as gen])) (def valid-email-gen (-> string/join (gen/fmap (gen/vector gen/char-alphanumeric 1 20)) (gen/bind #(gen/return (str % "@example.com"))))) (gen/generate (params-gen ::user/create)) {:email "9be2PgzoX00S@example.com" :username "Jn*kLqvOXol:u^>XC" :password "y&aOrni}"} derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
38. g1. (defn error? [response] (or (not (<= 200 (:status response) 299)) (exception? response) (timeout? response))) (defn assert-no-error [action] (let [params (gen/generate (params-gen action)) response (exec-action action params)] (is (not (error? response))))) (assert-no-error ::user/create) derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
39. g1. (defn error? [response] (or (not (<= 200 (:status response) 299)) (exception? response) (timeout? response))) (defn assert-no-error [action] (let [params (gen/generate (params-gen action)) response (exec-action action params)] (is (not (error? response))))) (assert-no-error ::user/create) derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
40. g1. (defn error? [response] (or (not (<= 200 (:status response) 299)) (exception? response) (timeout? response))) (defn assert-no-error [action] (let [params (gen/generate (params-gen action)) response (exec-action action params)] (is (not (error? response))))) (assert-no-error ::user/create) derive param specification param spec in action metadata generate input from spec no maintenance of test params spec can become api doc
41. g2. (defaction user-create :name ::user/create :param-schema s/CreateAccountParam :value-schema s/UserAccount :fn ([{:as param}] ...)) declare action dependencies what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase
42. g2. (def checkout-dependencies {::user/create [] ::user/authenticate [::user/create] ::user/add-card [::user/authenticate] ::user/update-card [::user/add-card] ::user/get [::user/create] ::cart/add-item [::user/authenticate] ::cart/update-item [::cart/add-item] ::cart/remove-item [::cart/add-item] ::cart/get [::user/authenticate] ::payment/apply-offer [::cart/add-item] ::payment/pay [::user/add-card ::cart/add-item] ::payment/refund [::user/pay] ::payment/cancel [::cart/add-item]}) declare action dependencies what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase
43. g2. declare action dependencies (def checkout-dependencies {::user/create [] ::user/authenticate [::user/create] ::user/add-card [::user/authenticate] ::user/update-card [::user/add-card] ::user/get [::user/create] ::cart/add-item [::user/authenticate] ::cart/update-item [::cart/add-item] ::cart/remove-item [::cart/add-item] ::cart/get [::user/authenticate] ::payment/apply-offer [::cart/add-item] ::payment/pay [::user/add-card ::cart/add-item] ::payment/refund [::user/pay] ::payment/cancel [::cart/add-item]}) what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase
44. g2. declare action dependencies (def all-actions #{::user/create ::user/authenticate ::user/add-card ::user/update-card ::user/get ::cart/add-item ::cart/update-item ::cart/remove-item ::cart/get ::payment/apply-offer ::payment/pay ::payment/refund ::payment/cancel}) (for [action actions] (assert-no-error action)) what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase
45. g3. what action might occur after? probabilistic modelling of usage is effective 0 and 100 are special cases automate the “act” phase probability matrices for flows (def most-users-flow {::user/create {::user/authenticate 90} ::user/authenticate {::user/add-card 20 ::cart/add-item 50} ::user/add-card {::user/update-card 10} ::cart/add-item {::cart/add-item 30 ::cart/update-item 10 ::cart/remove-item 20 ::payment/pay 20} ::payment/pay {::payment/refund 5}}) (def infrequent-users-flow …) (def rich-users-flow ...)
46. g3. probability matrices for flows (def most-users-flow {::user/create {::user/authenticate 90} ::user/authenticate {::user/add-card 20 ::cart/add-item 50} ::user/add-card {::user/update-card 10} ::cart/add-item {::cart/add-item 30 ::cart/update-item 10 ::cart/remove-item 20 ::payment/pay 20} ::payment/pay {::payment/refund 5}}) (def infrequent-users-flow …) (def rich-users-flow ...) what action might occur after? probabilistic modelling of usage is effective 0 and 100 are special cases automate the “act” phase
47. g3. what action might occur after? probabilistic modelling of usage is effective 0 and 100 are special cases automate the “act” phase probability matrices for flows (def most-users-flow {::user/create {::user/authenticate 90} ::user/authenticate {::user/add-card 20 ::cart/add-item 50} ::user/add-card {::user/update-card 10} ::cart/add-item {::cart/add-item 30 ::cart/update-item 10 ::cart/remove-item 20 ::payment/pay 20} ::payment/pay {::payment/refund 5}}) (def infrequent-users-flow …) (def rich-users-flow ...)
48. g3. what action might occur after? probabilistic modelling of usage is effective 0 and 100 are special cases automate the “act” phase probability matrices for flows (->> (action-seq-gen most-users-flow) gen/sample (take 2)) ([::user/create ::user/authenticate ::cart/add-item ::cart/update-item ::cart/update-item ::cart/get ::cart/update-item ::cart/get ::cart/remove-item ::user/add-card ::payment/apply-offer ::payment/apply-offer ::payment/cancel] [::user/create ::user/authenticate ::cart/add-item ::payment/apply-offer ::cart/add-item ::cart/add-item ::cart/add-item ::cart/add-item ::cart/add-item ::cart/get ::cart/update-item])
49. g3. what action might occur after? probabilistic modelling of usage is effective 0 and 100 are special cases automate the “act” phase probability matrices for flows [{:action ::user/create :params {:email "d3*(@example.com" :username "sdf@37" ...}} {:action ::user/authenticate :params {:username “73@fds” :password "23nv#2&" ...}} {:action ::cart/add-item :params {:sku 143 :qty 1 ...}} ...]
50. g4. model close-to real scenarios, not completely random ones simplifies debugging gives deterministic deterministic seed data [{:action ::user/create :params {:email "bezos@amazon.com" :username "bill" ...}} {:action ::user/authenticate :params {:username "bill" :password "bill_password" ...}} {:action ::user/add-card :params {:card-numer "1234123412341234" :cvc "606" ...}} ...]
51. g5. software only models subset of domain tests need to model domain around usage model external domain money in user’s card payment gateway latency inventory of supplier
52. generation execution assertion diagnosis patterns in executing generative tests
53. 1. store everything immutably 2. params as latest values in state 3. catalog known errors 4. abstract request engine patterns in executing generative tests
54. fn input output action lifecycle e1.
55. fn input output param-fn generated params state params continue? action lifecycle e1.
56. fn input output param-fn generated params state params DB continue? store everything e1.
57. store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (:require [datascript.core :as d]) (defn store-action [flow-id action-id request response entities] (let [metadata {:flow-id flow-id :action-id action-id}] (d/transact! db/conn [(-> request (assoc :data-type :request) (merge metadata))]) (d/transact! db/conn [(-> response (assoc :data-type :response) (merge metadata))]) (d/transact! db/conn [(-> entities compute-state (assoc :data-type :state) (merge metadata))])))
58. store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (:require [datascript.core :as d]) (defn store-action [flow-id action-id request response entities] (let [metadata {:flow-id flow-id :action-id action-id}] (d/transact! db/conn [(-> request (assoc :data-type :request) (merge metadata))]) (d/transact! db/conn [(-> response (assoc :data-type :response) (merge metadata))]) (d/transact! db/conn [(-> entities compute-state (assoc :data-type :state) (merge metadata))])))
59. store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn items [flow-id] (->> (d/q '[:find ?sku :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :cart/sku ?sku]] @db/conn flow-id) (map first)))
60. store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn items [flow-id] (->> (d/q '[:find ?sku :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :cart/sku ?sku]] @db/conn flow-id) (map first)))
61. store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn params-for [flow-id action] (->> (d/q '[:find ?request :in $ ?flow-id :where [?request :flow-id ?flow-id] [?request :data-type :request] [?request :action action]] @db/conn flow-id) (map first)))
62. store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn params-for [flow-id action] (->> (d/q '[:find ?request :in $ ?flow-id :where [?request :flow-id ?flow-id] [?request :data-type :request] [?request :action action]] @db/conn flow-id) (map first)))
63. store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn user-timeline [flow-id user-id] (->> (d/q '[:find ?e ?t :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :user-id ?user-id ?t]] @db/conn flow-id) (sort-by second >)))
64. store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn user-timeline [flow-id user-id] (->> (d/q '[:find ?e ?t :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :user-id ?user-id ?t]] @db/conn flow-id) (sort-by second >)))
65. params as latest value most recent value is most relevant use hooks to manipulate engine behaviour e2. (defn latest-param [flow-id attr-name] (->> (d/q '[:find ?attr-val ?t :in $ ?attr-name ?flow-id :where [?as :data-type :action-spec] [?as :flow-id ?flow-id] [?as :action-id ?aid] [?e :action-id ?aid] [?e ?attr-name ?attr-val ?t]] @db/conn attr-name flow-id) (sort-by second >) ffirst))
66. params as latest value most recent value is most relevant use hooks to manipulate engine behaviour e2. (defn latest-param [flow-id attr-name] (->> (d/q '[:find ?attr-val ?t :in $ ?attr-name ?flow-id :where [?as :data-type :action-spec] [?as :flow-id ?flow-id] [?as :action-id ?aid] [?e :action-id ?aid] [?e ?attr-name ?attr-val ?t]] @db/conn attr-name flow-id) (sort-by second >) ffirst))
67. fn input output param-fn generated params state params DB continue? store everything e1.
68. params as latest value most recent value is most relevant use hooks to manipulate engine behaviour e2. {::user/create {:entity-type :user :params-fn (fn [flow-id params] (assoc params :name (str "gen-user-" (u/uuid))))} ::payment/pay {:entity-type :payment :params-fn (fn [flow-id params] (->> (p/fetch-param flow-id :default-card) (assoc params :card-number)))}}
69. catalog known errors improve relevance reduce redundancy absorb into system e3. {::user/create {:entity-type :user :known-exceptions #{:duplicate-user} :params-fn (fn ...)} ::payment/pay {:entity-type :payment :known-exceptions #{:inactive-session :payment-complete} :params-fn (fn ...)}}
70. catalog known errors improve relevance reduce redundancy absorb into system e3. {::user/create {:entity-type :user :known-exceptions #{:duplicate-user} :params-fn (fn ...)} ::payment/pay {:entity-type :payment :known-exceptions #{:inactive-session :payment-complete} :params-fn (fn ...)}}
71. catalog known errors improve relevance reduce redundancy absorb into system e3. (def known-errors {:duplicate-user (fn [flow-id params] (some? (exec-action ::user/get (:username params)))) :payment-complete (fn [flow-id params] (some? (db/find-action flow-id ::payment/pay)))})
72. abstract request engine quickly switch to higher or lower level of specificity api vs controller makes debugging simpler e4. (defn make-request [medium action params] (case medium :fn ((:fn action) params) :bus (bus/request-value action params) :http (-> action http-request (client/request params))))
73. generation execution assertion diagnosis patterns in assertions
74. 1. algebraic properties 2. state machines as properties 3. domain based invariants patterns in assertions
75. property unit (no order) integration / system (ordered) no-error* ✓ ✓ egality ✗ ✓ idempotence ⍻ ✓ identity ✓ ✗ inverse ✗ ✓ commutativity ✗ ✓ associativity ✗ ✓ algebraic properties a1.
76. algebraic properties no-error {:message "No :error" :result (not-any? #(= :error %) (map :status responses))} ;; status (if (or (not (<= 200 status 299)) (some? exception) (timeout?)) :error) no-exceptions no-timeouts always-available a1.
77. algebraic properties egality (assert-equal [{:action ::user/signup}] [{:action ::user/get}]) (assert-equal [{:action ::cart/update-item :params {:deleted? true}}] [{:action ::cart/remove-item}]) f(a) ≡ g(b) effective equality different paths, same result a1.
78. algebraic properties idempotence {:percent-flow 10 :adjacency {:immediate 10 :distant 90} :blacklist #{::payment/pay ::payment/refund ::payment/cancel ::user/signup}} f(a) · f(a) ≡ f(a) 2 kinds of adjacencies blacklist / whitelist sensitive / distributed systems a b c c d e a b c d c e immediate distant a1.
79. algebraic properties inverse (assert-equal [{:action ::cart/add-item}] [{:action ::cart/remove-item}] :value-action ::cart/get) ;; normalised (assert-equal [{:action ::payment/pay} {:action ::payment/cancel} {:action ::payment/pay}] [{:action ::payment/pay}] :value-action ::payment/receipt) f(a) · f -1(a) ≡ a f -1(f(f -1(a)))=f -1(a) normalised inverse a1.
80. algebraic properties commutativity (assert-equal [{:action ::cart/add-item :params {:sku 1}} {:action ::cart/add-item :params {:sku 2}}] [{:action ::cart/add-item :params {:sku 2}} {:action ::cart/add-item :params {:sku 1}}] :value-action ::cart/get) f(a) · g(a) ≡ g(a) · f(a) non-compliance is a property too a1.
81. algebraic properties commutativity (assert-commutative [{:action ::cart/add-item :params {:sku 1}} {:action ::cart/add-item :params {:sku 2}}] :value-action ::cart/get) (assert-not-commutative [{:action ::cart/update-item :params {:sku 1 :qty 1}} {:action ::cart/update-item :params {:sku 1 :qty 6}}] :value-action ::cart/get) f(a) · g(a) ≡ g(a) · f(a) non-compliance is a property too a1.
82. state machines as properties new has-items success cancelled refunded authenticate add-item update-item remove-item apply-offer pay refund cancel remove-item a2.
83. (defn compute-expected-state [state action status] (match [state action status] [_ ::user/authenticate :ok] "new" ["new" ::cart/add-item :ok] "has-items" ["has-items" ::cart/update-item :ok] "has-items" ["has-items" ::cart/remove-item :ok] "has-items" ["has-items" ::cart/remove-item :ok] "new" ["has-items" ::payment/apply-offer :ok] "has-items" ["has-items" ::payment/pay :ok] "success" ["success" ::payment/refund :ok] "refunded" ["has-items" ::payment/cancel :ok] "cancelled" [_ _ :error] "error" [_ _ :known-exception] "known-exception")) a2.
84. domain based invariants for completeness create-count-is-count-of-list lookups-return-the-same-thing num-items-sold-matches-inventory receipt-totals-equal-profit a3.
85. generation execution assertion diagnosis patterns in diagnosis
86. 1. tests as portable data 2. domain based checkpoints 3. flow timeline, and walk throughs 4. automated bug reports patterns in diagnosis
87. tests as portable data flow / action-specs are stored tests determinism 㱺 reproducibility store passing and failing tests version tests d1. [{:action ::user/create :params {:email "d3*(@example.com" :username "sdf@37" ...}} {:action ::user/authenticate :params {:username “73@fds” :password "23nv#2&" ...}} {:action ::cart/add-item :params {:sku 143 :qty 1 ...}} ...]
88. tests as portable data flow / action-specs are stored tests determinism 㱺 reproducibility store passing and failing tests version tests d1. (-> "rc42-login-bug.edn" test/import test/run) (-> "v42-regression-test.edn" test/import test/run)
89. domain based checkpoints think of code as a trie useful when there are 100s of flows as on CI d2. {:checkpoints [{:action ::user/authenticate :description "Logged in"} {:action ::cart/add-item :description "Added item to cart"} {:action ::payment/pay :description "Payment attempted"}]}
90. flow timeline, walkthroughs d3. | action | status | action-id | time | |-----------------------+--------+-------------------+------------| | ::user/create | :ok | 1d8040ee-2486-... | 1524287925 | | ::user/authenticate | :ok | 92c6fb41-ad78-... | 1524287925 | | ::cart/add-item | :ok | 20c55130-85d2-... | 1524287927 | | ::payment/apply-offer | :ok | 239ebf75-a6c4-... | 1524287928 | | ::cart/add-item | :ok | d991dc62-5322-... | 1524287928 | | ::cart/add-item | :ok | f4b02a65-ac8f-... | 1524287928 | | ::cart/add-item | :ok | e0b85377-b6b7-... | 1524287931 | | ::cart/add-item | :ok | cb676220-8d6a-... | 1524287942 | | ::cart/add-item | :ok | 1d9546fb-7356-... | 1524287942 | | ::cart/get | :ok | b282a885-6e32-... | 1524287942 | | ::cart/update-item | :ex | ed062dca-f457-... | 1524287943 | timeline of actions relating to a user step-by-step diagnosis of events everything has an id, and a timestamp
91. flow timeline, walkthroughs d3. (dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6") {:metadata {:action ::cart/update-item :flow-id "35cb3104f-e0b0-..." :action-id "1d9546fb-7356-..."}} [{:data-type :action-spec :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :request :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :response :response {:status :error, :error-type java.lang.NullPointerException, :report {:cause #error { :cause nil :via [{:type java.lang.NullPointerException :message nil :at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}] :trace [[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77] [demo-proj.modules.cart$extract_filename invoke "cart.clj" 74] [demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139] [demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134] [clojure.lang.Var invoke "Var.java" 379] [demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke "logging.clj" 11] [intbus.core$invoke_handler_fn invokeStatic "core.clj" 134] [intbus.core$invoke_handler_fn invoke "core.clj" 129] [intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}] timeline of actions relating to a user step-by-step diagnosis of events everything has an id, and a timestamp
92. flow timeline, walkthroughs d3. (dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6") {:metadata {:action ::cart/update-item :flow-id "35cb3104f-e0b0-..." :action-id "1d9546fb-7356-..."}} [{:data-type :action-spec :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :request :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :response :response {:status :error, :error-type java.lang.NullPointerException, :report {:cause #error { :cause nil :via [{:type java.lang.NullPointerException :message nil :at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}] :trace [[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77] [demo-proj.modules.cart$extract_filename invoke "cart.clj" 74] [demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139] [demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134] [clojure.lang.Var invoke "Var.java" 379] [demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke "logging.clj" 11] [intbus.core$invoke_handler_fn invokeStatic "core.clj" 134] [intbus.core$invoke_handler_fn invoke "core.clj" 129] [intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}] timeline of actions relating to a user step-by-step diagnosis of events everything has an id, and a timestamp
93. flow timeline, walkthroughs d3. (dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6") {:metadata {:action ::cart/update-item :flow-id "35cb3104f-e0b0-..." :action-id "1d9546fb-7356-..."}} [{:data-type :action-spec :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :request :params {:cart-id 3412.. :sku 142.. :qty 947591}} {:data-type :response :response {:status :error, :error-type java.lang.NullPointerException, :report {:cause #error { :cause nil :via [{:type java.lang.NullPointerException :message nil :at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}] :trace [[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77] [demo-proj.modules.cart$extract_filename invoke "cart.clj" 74] [demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139] [demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134] [clojure.lang.Var invoke "Var.java" 379] [demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke "logging.clj" 11] [intbus.core$invoke_handler_fn invokeStatic "core.clj" 134] [intbus.core$invoke_handler_fn invoke "core.clj" 129] [intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}] timeline of actions relating to a user step-by-step diagnosis of events everything has an id, and a timestamp
94. automated bug reports d4. write code to summarise events create issues from CI to github
95. automated bug reports d4. write code to summarise events create issues from CI to github
96. 1. derive parameter specification 2. declare action dependencies 3. probability matrices for flows 4. deterministic seed data 5. model external domain 1. store everything immutably 2. params as latest values in state 3. catalog known errors 4. abstract request engine 1. algebraic properties 2. state machines as properties 3. domain based invariants 1. tests as portable data 2. domain based checkpoints 3. flow timeline, and walk throughs 4. automated bug reports generation assertion execution diagnosis
97. a thought paradigm automating software testing patterns
98. references microsoft PEX patterns datomic/simulant
99. practical generative testing patterns srihari sriraman | ø nilenso