practical system specimen user signup/create authenticate add/update-card get cart add/update-item remove-item get payment pay refund/cancel apply-offer receipt a generic e-commerce system
problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose benefits better than humans at input generation more tests, more coverage better chance at finding bugs before users do
problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose benefits better than humans at input generation more tests, more coverage better chance at finding bugs before users do
problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose inertia John Hughes – Testing the Hard Stuff and Staying Sane
learning curve Gary Fredericks – Building test check Generators problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
thinking in properties purely functional data algorithms or data structures are relatively easy to test most practical systems involve making a series of stateful api calls, most of which are CRUD problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
slow and ineffective CI duration grows exponentially bugs found in test failures aren’t necessarily relevant more tests ≠ more coverage problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
high maintenance generally more complex to write and read arrange, and act portions of test are still dependent on source problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
difficult to diagnose automatic shrinking really helps, but only until a point complex, stateful tests aren’t necessarily deterministic, and are difficult to reproduce finding root-cause of failure without the right tools can be very difficult problems inertia, learning curve thinking in properties slow, and ineffective high maintenance difficult to diagnose
1. derive parameter specification 2. declare action dependencies 3. probability matrices for flows 4. deterministic seed data 5. model external domain patterns in generating input
g2. (defaction user-create :name ::user/create :param-schema s/CreateAccountParam :value-schema s/UserAccount :fn ([{:as param}] ...)) declare action dependencies what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase
g2. (def checkout-dependencies {::user/create [] ::user/authenticate [::user/create] ::user/add-card [::user/authenticate] ::user/update-card [::user/add-card] ::user/get [::user/create] ::cart/add-item [::user/authenticate] ::cart/update-item [::cart/add-item] ::cart/remove-item [::cart/add-item] ::cart/get [::user/authenticate] ::payment/apply-offer [::cart/add-item] ::payment/pay [::user/add-card ::cart/add-item] ::payment/refund [::user/pay] ::payment/cancel [::cart/add-item]}) declare action dependencies what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase
g2. declare action dependencies (def checkout-dependencies {::user/create [] ::user/authenticate [::user/create] ::user/add-card [::user/authenticate] ::user/update-card [::user/add-card] ::user/get [::user/create] ::cart/add-item [::user/authenticate] ::cart/update-item [::cart/add-item] ::cart/remove-item [::cart/add-item] ::cart/get [::user/authenticate] ::payment/apply-offer [::cart/add-item] ::payment/pay [::user/add-card ::cart/add-item] ::payment/refund [::user/pay] ::payment/cancel [::cart/add-item]}) what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase
g2. declare action dependencies (def all-actions #{::user/create ::user/authenticate ::user/add-card ::user/update-card ::user/get ::cart/add-item ::cart/update-item ::cart/remove-item ::cart/get ::payment/apply-offer ::payment/pay ::payment/refund ::payment/cancel}) (for [action actions] (assert-no-error action)) what action must occur before? use a DAG dependencies are transitive, and can have order automate the “arrange” phase
g5. software only models subset of domain tests need to model domain around usage model external domain money in user’s card payment gateway latency inventory of supplier
1. store everything immutably 2. params as latest values in state 3. catalog known errors 4. abstract request engine patterns in executing generative tests
store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn items [flow-id] (->> (d/q '[:find ?sku :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :cart/sku ?sku]] @db/conn flow-id) (map first)))
store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn items [flow-id] (->> (d/q '[:find ?sku :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :cart/sku ?sku]] @db/conn flow-id) (map first)))
store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn user-timeline [flow-id user-id] (->> (d/q '[:find ?e ?t :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :user-id ?user-id ?t]] @db/conn flow-id) (sort-by second >)))
store everything immutably e1. ability to query for anything during execution, assertion or diagnosis tests need to be completely independent of source timeline matters (defn user-timeline [flow-id user-id] (->> (d/q '[:find ?e ?t :in $ ?flow-id :where [?e :flow-id ?flow-id] [?e :user-id ?user-id ?t]] @db/conn flow-id) (sort-by second >)))
algebraic properties idempotence {:percent-flow 10 :adjacency {:immediate 10 :distant 90} :blacklist #{::payment/pay ::payment/refund ::payment/cancel ::user/signup}} f(a) · f(a) ≡ f(a) 2 kinds of adjacencies blacklist / whitelist sensitive / distributed systems a b c c d e a b c d c e immediate distant a1.
domain based invariants for completeness create-count-is-count-of-list lookups-return-the-same-thing num-items-sold-matches-inventory receipt-totals-equal-profit a3.
tests as portable data flow / action-specs are stored tests determinism 㱺 reproducibility store passing and failing tests version tests d1. (-> "rc42-login-bug.edn" test/import test/run) (-> "v42-regression-test.edn" test/import test/run)
domain based checkpoints think of code as a trie useful when there are 100s of flows as on CI d2. {:checkpoints [{:action ::user/authenticate :description "Logged in"} {:action ::cart/add-item :description "Added item to cart"} {:action ::payment/pay :description "Payment attempted"}]}
1. derive parameter specification 2. declare action dependencies 3. probability matrices for flows 4. deterministic seed data 5. model external domain 1. store everything immutably 2. params as latest values in state 3. catalog known errors 4. abstract request engine 1. algebraic properties 2. state machines as properties 3. domain based invariants 1. tests as portable data 2. domain based checkpoints 3. flow timeline, and walk throughs 4. automated bug reports generation assertion execution diagnosis
srihari
keio_smilab
line_developers
cyberarchitect
codmoninc
masakamayama
devopstaiwan
bengo4com
oracle4engineer
ikezoemakoto
saboyutaka
mottox2
adavis
dotmariusz
brianwarren
bryan
chriscoyier
smashingmag
quramy
ddemaree
keathley
lynnandtonic
michaelherold
shpigford