Slide 1
Slide 1 text
practical generative
testing patterns
srihari sriraman | ø nilenso
Slide 2
Slide 2 text
about this talk
why, what
a practical system
benefits and problems
terminology
patterns
Slide 3
Slide 3 text
why
Slide 4
Slide 4 text
tests are good
why don’t we write them often enough?
generative tests are better
Slide 5
Slide 5 text
what
Slide 6
Slide 6 text
a thought paradigm
automating software testing
patterns
Slide 7
Slide 7 text
a practical system
Slide 8
Slide 8 text
practical system specimen
user
signup/create
authenticate
add/update-card
get
cart
add/update-item
remove-item
get
payment
pay
refund/cancel
apply-offer
receipt
a generic e-commerce system
Slide 9
Slide 9 text
practical system specimen
a generic e-commerce system
create (signup) authenticate (login)
add-card add-item
pay
update-item
Slide 10
Slide 10 text
benefits and problems
Slide 11
Slide 11 text
problems
inertia, learning curve
thinking in properties
slow, and ineffective
high maintenance
difficult to diagnose
benefits
better than humans at input
generation
more tests, more coverage
better chance at finding bugs
before users do
Slide 12
Slide 12 text
problems
inertia, learning curve
thinking in properties
slow, and ineffective
high maintenance
difficult to diagnose
benefits
better than humans at
input generation
more tests, more coverage
better chance at finding
bugs before users do
Slide 13
Slide 13 text
problems
inertia, learning curve
thinking in properties
slow, and ineffective
high maintenance
difficult to diagnose
inertia
John Hughes – Testing the Hard Stuff and Staying Sane
Slide 14
Slide 14 text
learning curve
Gary Fredericks – Building test check Generators
problems
inertia, learning curve
thinking in properties
slow, and ineffective
high maintenance
difficult to diagnose
Slide 15
Slide 15 text
thinking in properties
purely functional data algorithms or
data structures are relatively easy to
test
most practical systems involve making
a series of stateful api calls, most of
which are CRUD
problems
inertia, learning curve
thinking in properties
slow, and ineffective
high maintenance
difficult to diagnose
Slide 16
Slide 16 text
slow and ineffective
CI duration grows exponentially
bugs found in test failures aren’t
necessarily relevant
more tests ≠ more coverage
problems
inertia, learning curve
thinking in properties
slow, and ineffective
high maintenance
difficult to diagnose
Slide 17
Slide 17 text
high maintenance
generally more complex to write and
read
arrange, and act portions of test are still
dependent on source
problems
inertia, learning curve
thinking in properties
slow, and ineffective
high maintenance
difficult to diagnose
Slide 18
Slide 18 text
difficult to diagnose
automatic shrinking really helps, but
only until a point
complex, stateful tests aren’t
necessarily deterministic, and are
difficult to reproduce
finding root-cause of failure without
the right tools can be very difficult
problems
inertia, learning curve
thinking in properties
slow, and ineffective
high maintenance
difficult to diagnose
Slide 19
Slide 19 text
terminology
Slide 20
Slide 20 text
phase of testing
level of testing
focus of testing
segments
Slide 21
Slide 21 text
assume
arrange
act
assert
phase
level
focus
Slide 22
Slide 22 text
generation
execution
assertion
diagnosis
phase
level
focus
Slide 23
Slide 23 text
System
Integration
Unit
+ Value
+ Speed
phase
level
focus
Slide 24
Slide 24 text
System
Integration
Unit
+ Value
+ Speed
phase
level
focus
Slide 25
Slide 25 text
System
phase
level
focus
Slide 26
Slide 26 text
correctness
(smoke, acceptance, verification)
regression
(functional, quality control)
performance
(load, stress, endurance, spike)
phase
level
focus
Slide 27
Slide 27 text
system under test
fn
input
fn argument
request
state
output
return value
response
state
state
one fn
many fns
entire system
Slide 28
Slide 28 text
action under test
fn
input
fn argument
request
state
output
return value
response
state
state
one fn
many fns
entire system
Slide 29
Slide 29 text
automation
QA Engineer
Test Developer
Slide 30
Slide 30 text
patterns
Slide 31
Slide 31 text
pattern
/ˈpat(ə)n/
a regular and intelligible form or sequence discernible
in the way in which something happens or is done
Slide 32
Slide 32 text
generation
execution
assertion
diagnosis
patterns in
generating
input
Slide 33
Slide 33 text
1. derive parameter specification
2. declare action dependencies
3. probability matrices for flows
4. deterministic seed data
5. model external domain
patterns in
generating
input
Slide 34
Slide 34 text
derive param
specification
g1.
(ns system.schema)
(defschema CreateAccountParam
{:email (s/maybe valid-email)
:username non-empty-string
:password non-empty-string})
(defschema UserAccount
{:id String
:email String
:username String})
(ns system.account)
(defaction user-create
:name ::user/create
:param-schema s/CreateAccountParam
:value-schema s/UserAccount
:fn ([{:as param}]
...))
param spec in action metadata
generate input from spec
no maintenance of test params
spec can become api doc
Slide 35
Slide 35 text
derive param
specification
g1.
(ns system.schema)
(defschema CreateAccountParam
{:email (s/maybe valid-email)
:username non-empty-string
:password non-empty-string})
(defschema UserAccount
{:id String
:email String
:username String})
(ns system.account)
(defaction user-create
:name ::user/create
:param-schema s/CreateAccountParam
:value-schema s/UserAccount
:fn ([{:as param}]
...))
param spec in action metadata
generate input from spec
no maintenance of test params
spec can become api doc
Slide 36
Slide 36 text
g1.
(ns system.test
(:require
[clojure.test.check.generators :as gen]))
(def valid-email-gen
(-> string/join
(gen/fmap
(gen/vector gen/char-alphanumeric 1 20))
(gen/bind
#(gen/return (str % "@example.com")))))
(gen/generate (params-gen ::user/create))
{:email "[email protected]"
:username "Jn*kLqvOXol:u^>XC"
:password "y&aOrni}"}
derive param
specification
param spec in action metadata
generate input from spec
no maintenance of test params
spec can become api doc
Slide 37
Slide 37 text
g1.
(ns system.test
(:require
[clojure.test.check.generators :as gen]))
(def valid-email-gen
(-> string/join
(gen/fmap
(gen/vector gen/char-alphanumeric 1 20))
(gen/bind
#(gen/return (str % "@example.com")))))
(gen/generate (params-gen ::user/create))
{:email "[email protected]"
:username "Jn*kLqvOXol:u^>XC"
:password "y&aOrni}"}
derive param
specification
param spec in action metadata
generate input from spec
no maintenance of test params
spec can become api doc
Slide 38
Slide 38 text
g1.
(defn error? [response]
(or (not (<= 200 (:status response) 299))
(exception? response)
(timeout? response)))
(defn assert-no-error [action]
(let [params (gen/generate (params-gen action))
response (exec-action action params)]
(is (not (error? response)))))
(assert-no-error ::user/create)
derive param
specification
param spec in action metadata
generate input from spec
no maintenance of test params
spec can become api doc
Slide 39
Slide 39 text
g1.
(defn error? [response]
(or (not (<= 200 (:status response) 299))
(exception? response)
(timeout? response)))
(defn assert-no-error [action]
(let [params (gen/generate (params-gen action))
response (exec-action action params)]
(is (not (error? response)))))
(assert-no-error ::user/create)
derive param
specification
param spec in action metadata
generate input from spec
no maintenance of test params
spec can become api doc
Slide 40
Slide 40 text
g1.
(defn error? [response]
(or (not (<= 200 (:status response) 299))
(exception? response)
(timeout? response)))
(defn assert-no-error [action]
(let [params (gen/generate (params-gen action))
response (exec-action action params)]
(is (not (error? response)))))
(assert-no-error ::user/create)
derive param
specification
param spec in action metadata
generate input from spec
no maintenance of test params
spec can become api doc
Slide 41
Slide 41 text
g2.
(defaction user-create
:name ::user/create
:param-schema s/CreateAccountParam
:value-schema s/UserAccount
:fn ([{:as param}]
...))
declare action
dependencies
what action must occur before?
use a DAG
dependencies are transitive, and
can have order
automate the “arrange” phase
Slide 42
Slide 42 text
g2.
(def checkout-dependencies
{::user/create []
::user/authenticate [::user/create]
::user/add-card [::user/authenticate]
::user/update-card [::user/add-card]
::user/get [::user/create]
::cart/add-item [::user/authenticate]
::cart/update-item [::cart/add-item]
::cart/remove-item [::cart/add-item]
::cart/get [::user/authenticate]
::payment/apply-offer [::cart/add-item]
::payment/pay [::user/add-card
::cart/add-item]
::payment/refund [::user/pay]
::payment/cancel [::cart/add-item]})
declare action
dependencies
what action must occur before?
use a DAG
dependencies are transitive, and
can have order
automate the “arrange” phase
Slide 43
Slide 43 text
g2.
declare action
dependencies (def checkout-dependencies
{::user/create []
::user/authenticate [::user/create]
::user/add-card [::user/authenticate]
::user/update-card [::user/add-card]
::user/get [::user/create]
::cart/add-item [::user/authenticate]
::cart/update-item [::cart/add-item]
::cart/remove-item [::cart/add-item]
::cart/get [::user/authenticate]
::payment/apply-offer [::cart/add-item]
::payment/pay [::user/add-card
::cart/add-item]
::payment/refund [::user/pay]
::payment/cancel [::cart/add-item]})
what action must occur before?
use a DAG
dependencies are transitive, and
can have order
automate the “arrange” phase
Slide 44
Slide 44 text
g2.
declare action
dependencies
(def all-actions
#{::user/create
::user/authenticate
::user/add-card
::user/update-card
::user/get
::cart/add-item
::cart/update-item
::cart/remove-item
::cart/get
::payment/apply-offer
::payment/pay
::payment/refund
::payment/cancel})
(for [action actions]
(assert-no-error action))
what action must occur before?
use a DAG
dependencies are transitive, and
can have order
automate the “arrange” phase
Slide 45
Slide 45 text
g3.
what action might occur after?
probabilistic modelling of usage
is effective
0 and 100 are special cases
automate the “act” phase
probability
matrices for
flows
(def most-users-flow
{::user/create {::user/authenticate 90}
::user/authenticate {::user/add-card 20
::cart/add-item 50}
::user/add-card {::user/update-card 10}
::cart/add-item {::cart/add-item 30
::cart/update-item 10
::cart/remove-item 20
::payment/pay 20}
::payment/pay {::payment/refund 5}})
(def infrequent-users-flow …)
(def rich-users-flow ...)
Slide 46
Slide 46 text
g3.
probability
matrices for
flows
(def most-users-flow
{::user/create {::user/authenticate 90}
::user/authenticate {::user/add-card 20
::cart/add-item 50}
::user/add-card {::user/update-card 10}
::cart/add-item {::cart/add-item 30
::cart/update-item 10
::cart/remove-item 20
::payment/pay 20}
::payment/pay {::payment/refund 5}})
(def infrequent-users-flow …)
(def rich-users-flow ...)
what action might occur after?
probabilistic modelling of usage
is effective
0 and 100 are special cases
automate the “act” phase
Slide 47
Slide 47 text
g3.
what action might occur after?
probabilistic modelling of usage
is effective
0 and 100 are special cases
automate the “act” phase
probability
matrices for
flows
(def most-users-flow
{::user/create {::user/authenticate 90}
::user/authenticate {::user/add-card 20
::cart/add-item 50}
::user/add-card {::user/update-card 10}
::cart/add-item {::cart/add-item 30
::cart/update-item 10
::cart/remove-item 20
::payment/pay 20}
::payment/pay {::payment/refund 5}})
(def infrequent-users-flow …)
(def rich-users-flow ...)
Slide 48
Slide 48 text
g3.
what action might occur after?
probabilistic modelling of usage
is effective
0 and 100 are special cases
automate the “act” phase
probability
matrices for
flows
(->> (action-seq-gen most-users-flow)
gen/sample
(take 2))
([::user/create
::user/authenticate
::cart/add-item
::cart/update-item
::cart/update-item
::cart/get
::cart/update-item
::cart/get
::cart/remove-item
::user/add-card
::payment/apply-offer
::payment/apply-offer
::payment/cancel]
[::user/create
::user/authenticate
::cart/add-item
::payment/apply-offer
::cart/add-item
::cart/add-item
::cart/add-item
::cart/add-item
::cart/add-item
::cart/get
::cart/update-item])
Slide 49
Slide 49 text
g3.
what action might occur after?
probabilistic modelling of usage
is effective
0 and 100 are special cases
automate the “act” phase
probability
matrices for
flows
[{:action ::user/create
:params {:email "d3*(@example.com"
:username "sdf@37"
...}}
{:action ::user/authenticate
:params {:username “73@fds”
:password "23nv#2&"
...}}
{:action ::cart/add-item
:params {:sku 143
:qty 1
...}}
...]
Slide 50
Slide 50 text
g4.
model close-to real scenarios,
not completely random ones
simplifies debugging
gives deterministic
deterministic
seed data
[{:action ::user/create
:params {:email "[email protected]"
:username "bill"
...}}
{:action ::user/authenticate
:params {:username "bill"
:password "bill_password"
...}}
{:action ::user/add-card
:params {:card-numer "1234123412341234"
:cvc "606"
...}}
...]
Slide 51
Slide 51 text
g5.
software only models subset
of domain
tests need to model domain
around usage
model external
domain
money in user’s card
payment gateway latency
inventory of supplier
Slide 52
Slide 52 text
generation
execution
assertion
diagnosis
patterns in
executing
generative
tests
Slide 53
Slide 53 text
1. store everything immutably
2. params as latest values in state
3. catalog known errors
4. abstract request engine
patterns in
executing
generative
tests
Slide 54
Slide 54 text
fn
input output
action lifecycle
e1.
Slide 55
Slide 55 text
fn
input output
param-fn
generated
params
state
params
continue?
action lifecycle
e1.
Slide 56
Slide 56 text
fn
input output
param-fn
generated
params
state
params
DB continue?
store everything
e1.
Slide 57
Slide 57 text
store
everything
immutably
e1.
ability to query for anything during
execution, assertion or diagnosis
tests need to be completely
independent of source
timeline matters
(:require [datascript.core :as d])
(defn store-action
[flow-id action-id request response entities]
(let [metadata {:flow-id flow-id
:action-id action-id}]
(d/transact!
db/conn
[(-> request
(assoc :data-type :request)
(merge metadata))])
(d/transact!
db/conn
[(-> response
(assoc :data-type :response)
(merge metadata))])
(d/transact!
db/conn
[(-> entities
compute-state
(assoc :data-type :state)
(merge metadata))])))
Slide 58
Slide 58 text
store
everything
immutably
e1.
ability to query for anything during
execution, assertion or diagnosis
tests need to be completely
independent of source
timeline matters
(:require [datascript.core :as d])
(defn store-action
[flow-id action-id request response entities]
(let [metadata {:flow-id flow-id
:action-id action-id}]
(d/transact!
db/conn
[(-> request
(assoc :data-type :request)
(merge metadata))])
(d/transact!
db/conn
[(-> response
(assoc :data-type :response)
(merge metadata))])
(d/transact!
db/conn
[(-> entities
compute-state
(assoc :data-type :state)
(merge metadata))])))
Slide 59
Slide 59 text
store
everything
immutably
e1.
ability to query for anything during
execution, assertion or diagnosis
tests need to be completely
independent of source
timeline matters
(defn items [flow-id]
(->> (d/q '[:find ?sku
:in $ ?flow-id
:where
[?e :flow-id ?flow-id]
[?e :cart/sku ?sku]]
@db/conn
flow-id)
(map first)))
Slide 60
Slide 60 text
store
everything
immutably
e1.
ability to query for anything during
execution, assertion or diagnosis
tests need to be completely
independent of source
timeline matters
(defn items [flow-id]
(->> (d/q '[:find ?sku
:in $ ?flow-id
:where
[?e :flow-id ?flow-id]
[?e :cart/sku ?sku]]
@db/conn
flow-id)
(map first)))
Slide 61
Slide 61 text
store
everything
immutably
e1.
ability to query for anything during
execution, assertion or diagnosis
tests need to be completely
independent of source
timeline matters
(defn params-for [flow-id action]
(->> (d/q '[:find ?request
:in $ ?flow-id
:where
[?request :flow-id ?flow-id]
[?request :data-type :request]
[?request :action action]]
@db/conn
flow-id)
(map first)))
Slide 62
Slide 62 text
store
everything
immutably
e1.
ability to query for anything during
execution, assertion or diagnosis
tests need to be completely
independent of source
timeline matters
(defn params-for [flow-id action]
(->> (d/q '[:find ?request
:in $ ?flow-id
:where
[?request :flow-id ?flow-id]
[?request :data-type :request]
[?request :action action]]
@db/conn
flow-id)
(map first)))
Slide 63
Slide 63 text
store
everything
immutably
e1.
ability to query for anything during
execution, assertion or diagnosis
tests need to be completely
independent of source
timeline matters
(defn user-timeline [flow-id user-id]
(->> (d/q '[:find ?e ?t
:in $ ?flow-id
:where
[?e :flow-id ?flow-id]
[?e :user-id ?user-id ?t]]
@db/conn
flow-id)
(sort-by second >)))
Slide 64
Slide 64 text
store
everything
immutably
e1.
ability to query for anything during
execution, assertion or diagnosis
tests need to be completely
independent of source
timeline matters
(defn user-timeline [flow-id user-id]
(->> (d/q '[:find ?e ?t
:in $ ?flow-id
:where
[?e :flow-id ?flow-id]
[?e :user-id ?user-id ?t]]
@db/conn
flow-id)
(sort-by second >)))
Slide 65
Slide 65 text
params as
latest value
most recent value is most relevant
use hooks to manipulate engine
behaviour
e2.
(defn latest-param [flow-id attr-name]
(->> (d/q '[:find ?attr-val ?t
:in $ ?attr-name ?flow-id
:where
[?as :data-type :action-spec]
[?as :flow-id ?flow-id]
[?as :action-id ?aid]
[?e :action-id ?aid]
[?e ?attr-name ?attr-val ?t]]
@db/conn
attr-name
flow-id)
(sort-by second >)
ffirst))
Slide 66
Slide 66 text
params as
latest value
most recent value is most relevant
use hooks to manipulate engine
behaviour
e2.
(defn latest-param [flow-id attr-name]
(->> (d/q '[:find ?attr-val ?t
:in $ ?attr-name ?flow-id
:where
[?as :data-type :action-spec]
[?as :flow-id ?flow-id]
[?as :action-id ?aid]
[?e :action-id ?aid]
[?e ?attr-name ?attr-val ?t]]
@db/conn
attr-name
flow-id)
(sort-by second >)
ffirst))
Slide 67
Slide 67 text
fn
input output
param-fn
generated
params
state
params
DB continue?
store everything
e1.
Slide 68
Slide 68 text
params as
latest value
most recent value is most relevant
use hooks to manipulate engine
behaviour
e2.
{::user/create
{:entity-type :user
:params-fn
(fn [flow-id params]
(assoc params :name
(str "gen-user-" (u/uuid))))}
::payment/pay
{:entity-type :payment
:params-fn
(fn [flow-id params]
(->> (p/fetch-param flow-id :default-card)
(assoc params :card-number)))}}
Slide 69
Slide 69 text
catalog
known errors
improve relevance
reduce redundancy
absorb into system
e3.
{::user/create
{:entity-type :user
:known-exceptions #{:duplicate-user}
:params-fn (fn ...)}
::payment/pay
{:entity-type :payment
:known-exceptions #{:inactive-session
:payment-complete}
:params-fn (fn ...)}}
Slide 70
Slide 70 text
catalog
known errors
improve relevance
reduce redundancy
absorb into system
e3.
{::user/create
{:entity-type :user
:known-exceptions #{:duplicate-user}
:params-fn (fn ...)}
::payment/pay
{:entity-type :payment
:known-exceptions #{:inactive-session
:payment-complete}
:params-fn (fn ...)}}
Slide 71
Slide 71 text
catalog
known errors
improve relevance
reduce redundancy
absorb into system
e3.
(def known-errors
{:duplicate-user
(fn [flow-id params]
(some?
(exec-action ::user/get
(:username params))))
:payment-complete
(fn [flow-id params]
(some? (db/find-action flow-id
::payment/pay)))})
Slide 72
Slide 72 text
abstract
request engine
quickly switch to higher or
lower level of specificity
api vs controller
makes debugging simpler
e4.
(defn make-request [medium action params]
(case medium
:fn ((:fn action) params)
:bus (bus/request-value action params)
:http (-> action
http-request
(client/request params))))
Slide 73
Slide 73 text
generation
execution
assertion
diagnosis
patterns in
assertions
Slide 74
Slide 74 text
1. algebraic properties
2. state machines as properties
3. domain based invariants
patterns in
assertions
Slide 75
Slide 75 text
property
unit
(no order)
integration / system
(ordered)
no-error* ✓ ✓
egality ✗ ✓
idempotence ⍻ ✓
identity ✓ ✗
inverse ✗ ✓
commutativity ✗ ✓
associativity ✗ ✓
algebraic properties
a1.
Slide 76
Slide 76 text
algebraic properties
no-error {:message "No :error"
:result
(not-any? #(= :error %)
(map :status responses))}
;; status
(if (or (not (<= 200 status 299))
(some? exception)
(timeout?))
:error)
no-exceptions
no-timeouts
always-available
a1.
Slide 77
Slide 77 text
algebraic properties
egality (assert-equal
[{:action ::user/signup}]
[{:action ::user/get}])
(assert-equal
[{:action ::cart/update-item
:params {:deleted? true}}]
[{:action ::cart/remove-item}])
f(a) ≡ g(b)
effective equality
different paths, same result
a1.
Slide 78
Slide 78 text
algebraic properties
idempotence
{:percent-flow 10
:adjacency {:immediate 10
:distant 90}
:blacklist #{::payment/pay
::payment/refund
::payment/cancel
::user/signup}}
f(a) · f(a) ≡ f(a)
2 kinds of adjacencies
blacklist / whitelist
sensitive / distributed systems
a b c c d e
a b c d c e
immediate
distant
a1.
Slide 79
Slide 79 text
algebraic properties
inverse
(assert-equal
[{:action ::cart/add-item}]
[{:action ::cart/remove-item}]
:value-action ::cart/get)
;; normalised
(assert-equal
[{:action ::payment/pay}
{:action ::payment/cancel}
{:action ::payment/pay}]
[{:action ::payment/pay}]
:value-action ::payment/receipt)
f(a) · f -1(a) ≡ a
f -1(f(f -1(a)))=f -1(a)
normalised inverse
a1.
Slide 80
Slide 80 text
algebraic properties
commutativity
(assert-equal
[{:action ::cart/add-item
:params {:sku 1}}
{:action ::cart/add-item
:params {:sku 2}}]
[{:action ::cart/add-item
:params {:sku 2}}
{:action ::cart/add-item
:params {:sku 1}}]
:value-action ::cart/get)
f(a) · g(a) ≡ g(a) · f(a)
non-compliance is a property too
a1.
Slide 81
Slide 81 text
algebraic properties
commutativity
(assert-commutative
[{:action ::cart/add-item
:params {:sku 1}}
{:action ::cart/add-item
:params {:sku 2}}]
:value-action ::cart/get)
(assert-not-commutative
[{:action ::cart/update-item
:params {:sku 1 :qty 1}}
{:action ::cart/update-item
:params {:sku 1 :qty 6}}]
:value-action ::cart/get)
f(a) · g(a) ≡ g(a) · f(a)
non-compliance is a property too
a1.
Slide 82
Slide 82 text
state machines as properties
new has-items success
cancelled
refunded
authenticate
add-item
update-item
remove-item
apply-offer
pay
refund
cancel
remove-item
a2.
Slide 83
Slide 83 text
(defn compute-expected-state [state action status]
(match
[state action status]
[_ ::user/authenticate :ok] "new"
["new" ::cart/add-item :ok] "has-items"
["has-items" ::cart/update-item :ok] "has-items"
["has-items" ::cart/remove-item :ok] "has-items"
["has-items" ::cart/remove-item :ok] "new"
["has-items" ::payment/apply-offer :ok] "has-items"
["has-items" ::payment/pay :ok] "success"
["success" ::payment/refund :ok] "refunded"
["has-items" ::payment/cancel :ok] "cancelled"
[_ _ :error] "error"
[_ _ :known-exception] "known-exception"))
a2.
Slide 84
Slide 84 text
domain based invariants
for completeness
create-count-is-count-of-list
lookups-return-the-same-thing
num-items-sold-matches-inventory
receipt-totals-equal-profit
a3.
Slide 85
Slide 85 text
generation
execution
assertion
diagnosis
patterns in
diagnosis
Slide 86
Slide 86 text
1. tests as portable data
2. domain based checkpoints
3. flow timeline, and walk throughs
4. automated bug reports
patterns in
diagnosis
Slide 87
Slide 87 text
tests as
portable data
flow / action-specs are stored tests
determinism 㱺 reproducibility
store passing and failing tests
version tests
d1.
[{:action ::user/create
:params {:email "d3*(@example.com"
:username "sdf@37"
...}}
{:action ::user/authenticate
:params {:username “73@fds”
:password "23nv#2&"
...}}
{:action ::cart/add-item
:params {:sku 143
:qty 1
...}}
...]
Slide 88
Slide 88 text
tests as
portable data
flow / action-specs are stored tests
determinism 㱺 reproducibility
store passing and failing tests
version tests
d1.
(-> "rc42-login-bug.edn"
test/import
test/run)
(-> "v42-regression-test.edn"
test/import
test/run)
Slide 89
Slide 89 text
domain based
checkpoints
think of code as a trie
useful when there are 100s of flows
as on CI
d2.
{:checkpoints
[{:action ::user/authenticate
:description "Logged in"}
{:action ::cart/add-item
:description "Added item to cart"}
{:action ::payment/pay
:description "Payment attempted"}]}
Slide 90
Slide 90 text
flow timeline,
walkthroughs
d3.
| action | status | action-id | time |
|-----------------------+--------+-------------------+------------|
| ::user/create | :ok | 1d8040ee-2486-... | 1524287925 |
| ::user/authenticate | :ok | 92c6fb41-ad78-... | 1524287925 |
| ::cart/add-item | :ok | 20c55130-85d2-... | 1524287927 |
| ::payment/apply-offer | :ok | 239ebf75-a6c4-... | 1524287928 |
| ::cart/add-item | :ok | d991dc62-5322-... | 1524287928 |
| ::cart/add-item | :ok | f4b02a65-ac8f-... | 1524287928 |
| ::cart/add-item | :ok | e0b85377-b6b7-... | 1524287931 |
| ::cart/add-item | :ok | cb676220-8d6a-... | 1524287942 |
| ::cart/add-item | :ok | 1d9546fb-7356-... | 1524287942 |
| ::cart/get | :ok | b282a885-6e32-... | 1524287942 |
| ::cart/update-item | :ex | ed062dca-f457-... | 1524287943 |
timeline of actions relating to a user
step-by-step diagnosis of events
everything has an id, and a timestamp
Slide 91
Slide 91 text
flow timeline,
walkthroughs
d3.
(dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6")
{:metadata {:action ::cart/update-item
:flow-id "35cb3104f-e0b0-..."
:action-id "1d9546fb-7356-..."}}
[{:data-type :action-spec
:params {:cart-id 3412.. :sku 142.. :qty 947591}}
{:data-type :request
:params {:cart-id 3412.. :sku 142.. :qty 947591}}
{:data-type :response
:response
{:status :error,
:error-type java.lang.NullPointerException,
:report
{:cause #error
{
:cause nil
:via
[{:type java.lang.NullPointerException
:message nil
:at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}]
:trace
[[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]
[demo-proj.modules.cart$extract_filename invoke "cart.clj" 74]
[demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139]
[demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134]
[clojure.lang.Var invoke "Var.java" 379]
[demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke
"logging.clj" 11]
[intbus.core$invoke_handler_fn invokeStatic "core.clj" 134]
[intbus.core$invoke_handler_fn invoke "core.clj" 129]
[intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}]
timeline of actions relating to a user
step-by-step diagnosis of events
everything has an id, and a timestamp
Slide 92
Slide 92 text
flow timeline,
walkthroughs
d3.
(dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6")
{:metadata {:action ::cart/update-item
:flow-id "35cb3104f-e0b0-..."
:action-id "1d9546fb-7356-..."}}
[{:data-type :action-spec
:params {:cart-id 3412.. :sku 142.. :qty 947591}}
{:data-type :request
:params {:cart-id 3412.. :sku 142.. :qty 947591}}
{:data-type :response
:response
{:status :error,
:error-type java.lang.NullPointerException,
:report
{:cause #error
{
:cause nil
:via
[{:type java.lang.NullPointerException
:message nil
:at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}]
:trace
[[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]
[demo-proj.modules.cart$extract_filename invoke "cart.clj" 74]
[demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139]
[demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134]
[clojure.lang.Var invoke "Var.java" 379]
[demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke
"logging.clj" 11]
[intbus.core$invoke_handler_fn invokeStatic "core.clj" 134]
[intbus.core$invoke_handler_fn invoke "core.clj" 129]
[intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}]
timeline of actions relating to a user
step-by-step diagnosis of events
everything has an id, and a timestamp
Slide 93
Slide 93 text
flow timeline,
walkthroughs
d3.
(dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6")
{:metadata {:action ::cart/update-item
:flow-id "35cb3104f-e0b0-..."
:action-id "1d9546fb-7356-..."}}
[{:data-type :action-spec
:params {:cart-id 3412.. :sku 142.. :qty 947591}}
{:data-type :request
:params {:cart-id 3412.. :sku 142.. :qty 947591}}
{:data-type :response
:response
{:status :error,
:error-type java.lang.NullPointerException,
:report
{:cause #error
{
:cause nil
:via
[{:type java.lang.NullPointerException
:message nil
:at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}]
:trace
[[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]
[demo-proj.modules.cart$extract_filename invoke "cart.clj" 74]
[demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139]
[demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134]
[clojure.lang.Var invoke "Var.java" 379]
[demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke
"logging.clj" 11]
[intbus.core$invoke_handler_fn invokeStatic "core.clj" 134]
[intbus.core$invoke_handler_fn invoke "core.clj" 129]
[intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}]
timeline of actions relating to a user
step-by-step diagnosis of events
everything has an id, and a timestamp
Slide 94
Slide 94 text
automated
bug reports
d4.
write code to summarise events
create issues from CI to github
Slide 95
Slide 95 text
automated
bug reports
d4.
write code to summarise events
create issues from CI to github
Slide 96
Slide 96 text
1. derive parameter specification
2. declare action dependencies
3. probability matrices for flows
4. deterministic seed data
5. model external domain
1. store everything immutably
2. params as latest values in state
3. catalog known errors
4. abstract request engine
1. algebraic properties
2. state machines as properties
3. domain based invariants
1. tests as portable data
2. domain based checkpoints
3. flow timeline, and walk throughs
4. automated bug reports
generation
assertion
execution
diagnosis
Slide 97
Slide 97 text
a thought paradigm
automating software testing
patterns
Slide 98
Slide 98 text
references
microsoft PEX patterns
datomic/simulant
Slide 99
Slide 99 text
practical generative
testing patterns
srihari sriraman | ø nilenso