Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Practical Generative Testing Patterns

Practical Generative Testing Patterns

Talk slides as presented at Dutch Clojure Days 2018

Srihari Sriraman

April 21, 2018
Tweet

More Decks by Srihari Sriraman

Other Decks in Technology

Transcript

  1. practical generative
    testing patterns
    srihari sriraman | ø nilenso

    View Slide

  2. about this talk
    why, what
    a practical system
    benefits and problems
    terminology
    patterns

    View Slide

  3. why

    View Slide

  4. tests are good
    why don’t we write them often enough?
    generative tests are better

    View Slide

  5. what

    View Slide

  6. a thought paradigm
    automating software testing
    patterns

    View Slide

  7. a practical system

    View Slide

  8. practical system specimen
    user
    signup/create
    authenticate
    add/update-card
    get
    cart
    add/update-item
    remove-item
    get
    payment
    pay
    refund/cancel
    apply-offer
    receipt
    a generic e-commerce system

    View Slide

  9. practical system specimen
    a generic e-commerce system
    create (signup) authenticate (login)
    add-card add-item
    pay
    update-item

    View Slide

  10. benefits and problems

    View Slide

  11. problems
    inertia, learning curve
    thinking in properties
    slow, and ineffective
    high maintenance
    difficult to diagnose
    benefits
    better than humans at input
    generation
    more tests, more coverage
    better chance at finding bugs
    before users do

    View Slide

  12. problems
    inertia, learning curve
    thinking in properties
    slow, and ineffective
    high maintenance
    difficult to diagnose
    benefits
    better than humans at
    input generation
    more tests, more coverage
    better chance at finding
    bugs before users do

    View Slide

  13. problems
    inertia, learning curve
    thinking in properties
    slow, and ineffective
    high maintenance
    difficult to diagnose
    inertia
    John Hughes – Testing the Hard Stuff and Staying Sane

    View Slide

  14. learning curve
    Gary Fredericks – Building test check Generators
    problems
    inertia, learning curve
    thinking in properties
    slow, and ineffective
    high maintenance
    difficult to diagnose

    View Slide

  15. thinking in properties
    purely functional data algorithms or
    data structures are relatively easy to
    test
    most practical systems involve making
    a series of stateful api calls, most of
    which are CRUD
    problems
    inertia, learning curve
    thinking in properties
    slow, and ineffective
    high maintenance
    difficult to diagnose

    View Slide

  16. slow and ineffective
    CI duration grows exponentially
    bugs found in test failures aren’t
    necessarily relevant
    more tests ≠ more coverage
    problems
    inertia, learning curve
    thinking in properties
    slow, and ineffective
    high maintenance
    difficult to diagnose

    View Slide

  17. high maintenance
    generally more complex to write and
    read
    arrange, and act portions of test are still
    dependent on source
    problems
    inertia, learning curve
    thinking in properties
    slow, and ineffective
    high maintenance
    difficult to diagnose

    View Slide

  18. difficult to diagnose
    automatic shrinking really helps, but
    only until a point
    complex, stateful tests aren’t
    necessarily deterministic, and are
    difficult to reproduce
    finding root-cause of failure without
    the right tools can be very difficult
    problems
    inertia, learning curve
    thinking in properties
    slow, and ineffective
    high maintenance
    difficult to diagnose

    View Slide

  19. terminology

    View Slide

  20. phase of testing
    level of testing
    focus of testing
    segments

    View Slide

  21. assume
    arrange
    act
    assert
    phase
    level
    focus

    View Slide

  22. generation
    execution
    assertion
    diagnosis
    phase
    level
    focus

    View Slide

  23. System
    Integration
    Unit
    + Value
    + Speed
    phase
    level
    focus

    View Slide

  24. System
    Integration
    Unit
    + Value
    + Speed
    phase
    level
    focus

    View Slide

  25. System
    phase
    level
    focus

    View Slide

  26. correctness
    (smoke, acceptance, verification)
    regression
    (functional, quality control)
    performance
    (load, stress, endurance, spike)
    phase
    level
    focus

    View Slide

  27. system under test
    fn
    input
    fn argument
    request
    state
    output
    return value
    response
    state
    state
    one fn
    many fns
    entire system

    View Slide

  28. action under test
    fn
    input
    fn argument
    request
    state
    output
    return value
    response
    state
    state
    one fn
    many fns
    entire system

    View Slide

  29. automation
    QA Engineer
    Test Developer

    View Slide

  30. patterns

    View Slide

  31. pattern
    /ˈpat(ə)n/
    a regular and intelligible form or sequence discernible
    in the way in which something happens or is done

    View Slide

  32. generation
    execution
    assertion
    diagnosis
    patterns in
    generating
    input

    View Slide

  33. 1. derive parameter specification
    2. declare action dependencies
    3. probability matrices for flows
    4. deterministic seed data
    5. model external domain
    patterns in
    generating
    input

    View Slide

  34. derive param
    specification
    g1.
    (ns system.schema)
    (defschema CreateAccountParam
    {:email (s/maybe valid-email)
    :username non-empty-string
    :password non-empty-string})
    (defschema UserAccount
    {:id String
    :email String
    :username String})
    (ns system.account)
    (defaction user-create
    :name ::user/create
    :param-schema s/CreateAccountParam
    :value-schema s/UserAccount
    :fn ([{:as param}]
    ...))
    param spec in action metadata
    generate input from spec
    no maintenance of test params
    spec can become api doc

    View Slide

  35. derive param
    specification
    g1.
    (ns system.schema)
    (defschema CreateAccountParam
    {:email (s/maybe valid-email)
    :username non-empty-string
    :password non-empty-string})
    (defschema UserAccount
    {:id String
    :email String
    :username String})
    (ns system.account)
    (defaction user-create
    :name ::user/create
    :param-schema s/CreateAccountParam
    :value-schema s/UserAccount
    :fn ([{:as param}]
    ...))
    param spec in action metadata
    generate input from spec
    no maintenance of test params
    spec can become api doc

    View Slide

  36. g1.
    (ns system.test
    (:require
    [clojure.test.check.generators :as gen]))
    (def valid-email-gen
    (-> string/join
    (gen/fmap
    (gen/vector gen/char-alphanumeric 1 20))
    (gen/bind
    #(gen/return (str % "@example.com")))))
    (gen/generate (params-gen ::user/create))
    {:email "[email protected]"
    :username "Jn*kLqvOXol:u^>XC"
    :password "y&aOrni}"}
    derive param
    specification
    param spec in action metadata
    generate input from spec
    no maintenance of test params
    spec can become api doc

    View Slide

  37. g1.
    (ns system.test
    (:require
    [clojure.test.check.generators :as gen]))
    (def valid-email-gen
    (-> string/join
    (gen/fmap
    (gen/vector gen/char-alphanumeric 1 20))
    (gen/bind
    #(gen/return (str % "@example.com")))))
    (gen/generate (params-gen ::user/create))
    {:email "[email protected]"
    :username "Jn*kLqvOXol:u^>XC"
    :password "y&aOrni}"}
    derive param
    specification
    param spec in action metadata
    generate input from spec
    no maintenance of test params
    spec can become api doc

    View Slide

  38. g1.
    (defn error? [response]
    (or (not (<= 200 (:status response) 299))
    (exception? response)
    (timeout? response)))
    (defn assert-no-error [action]
    (let [params (gen/generate (params-gen action))
    response (exec-action action params)]
    (is (not (error? response)))))
    (assert-no-error ::user/create)
    derive param
    specification
    param spec in action metadata
    generate input from spec
    no maintenance of test params
    spec can become api doc

    View Slide

  39. g1.
    (defn error? [response]
    (or (not (<= 200 (:status response) 299))
    (exception? response)
    (timeout? response)))
    (defn assert-no-error [action]
    (let [params (gen/generate (params-gen action))
    response (exec-action action params)]
    (is (not (error? response)))))
    (assert-no-error ::user/create)
    derive param
    specification
    param spec in action metadata
    generate input from spec
    no maintenance of test params
    spec can become api doc

    View Slide

  40. g1.
    (defn error? [response]
    (or (not (<= 200 (:status response) 299))
    (exception? response)
    (timeout? response)))
    (defn assert-no-error [action]
    (let [params (gen/generate (params-gen action))
    response (exec-action action params)]
    (is (not (error? response)))))
    (assert-no-error ::user/create)
    derive param
    specification
    param spec in action metadata
    generate input from spec
    no maintenance of test params
    spec can become api doc

    View Slide

  41. g2.
    (defaction user-create
    :name ::user/create
    :param-schema s/CreateAccountParam
    :value-schema s/UserAccount
    :fn ([{:as param}]
    ...))
    declare action
    dependencies
    what action must occur before?
    use a DAG
    dependencies are transitive, and
    can have order
    automate the “arrange” phase

    View Slide

  42. g2.
    (def checkout-dependencies
    {::user/create []
    ::user/authenticate [::user/create]
    ::user/add-card [::user/authenticate]
    ::user/update-card [::user/add-card]
    ::user/get [::user/create]
    ::cart/add-item [::user/authenticate]
    ::cart/update-item [::cart/add-item]
    ::cart/remove-item [::cart/add-item]
    ::cart/get [::user/authenticate]
    ::payment/apply-offer [::cart/add-item]
    ::payment/pay [::user/add-card
    ::cart/add-item]
    ::payment/refund [::user/pay]
    ::payment/cancel [::cart/add-item]})
    declare action
    dependencies
    what action must occur before?
    use a DAG
    dependencies are transitive, and
    can have order
    automate the “arrange” phase

    View Slide

  43. g2.
    declare action
    dependencies (def checkout-dependencies
    {::user/create []
    ::user/authenticate [::user/create]
    ::user/add-card [::user/authenticate]
    ::user/update-card [::user/add-card]
    ::user/get [::user/create]
    ::cart/add-item [::user/authenticate]
    ::cart/update-item [::cart/add-item]
    ::cart/remove-item [::cart/add-item]
    ::cart/get [::user/authenticate]
    ::payment/apply-offer [::cart/add-item]
    ::payment/pay [::user/add-card
    ::cart/add-item]
    ::payment/refund [::user/pay]
    ::payment/cancel [::cart/add-item]})
    what action must occur before?
    use a DAG
    dependencies are transitive, and
    can have order
    automate the “arrange” phase

    View Slide

  44. g2.
    declare action
    dependencies
    (def all-actions
    #{::user/create
    ::user/authenticate
    ::user/add-card
    ::user/update-card
    ::user/get
    ::cart/add-item
    ::cart/update-item
    ::cart/remove-item
    ::cart/get
    ::payment/apply-offer
    ::payment/pay
    ::payment/refund
    ::payment/cancel})
    (for [action actions]
    (assert-no-error action))
    what action must occur before?
    use a DAG
    dependencies are transitive, and
    can have order
    automate the “arrange” phase

    View Slide

  45. g3.
    what action might occur after?
    probabilistic modelling of usage
    is effective
    0 and 100 are special cases
    automate the “act” phase
    probability
    matrices for
    flows
    (def most-users-flow
    {::user/create {::user/authenticate 90}
    ::user/authenticate {::user/add-card 20
    ::cart/add-item 50}
    ::user/add-card {::user/update-card 10}
    ::cart/add-item {::cart/add-item 30
    ::cart/update-item 10
    ::cart/remove-item 20
    ::payment/pay 20}
    ::payment/pay {::payment/refund 5}})
    (def infrequent-users-flow …)
    (def rich-users-flow ...)

    View Slide

  46. g3.
    probability
    matrices for
    flows
    (def most-users-flow
    {::user/create {::user/authenticate 90}
    ::user/authenticate {::user/add-card 20
    ::cart/add-item 50}
    ::user/add-card {::user/update-card 10}
    ::cart/add-item {::cart/add-item 30
    ::cart/update-item 10
    ::cart/remove-item 20
    ::payment/pay 20}
    ::payment/pay {::payment/refund 5}})
    (def infrequent-users-flow …)
    (def rich-users-flow ...)
    what action might occur after?
    probabilistic modelling of usage
    is effective
    0 and 100 are special cases
    automate the “act” phase

    View Slide

  47. g3.
    what action might occur after?
    probabilistic modelling of usage
    is effective
    0 and 100 are special cases
    automate the “act” phase
    probability
    matrices for
    flows
    (def most-users-flow
    {::user/create {::user/authenticate 90}
    ::user/authenticate {::user/add-card 20
    ::cart/add-item 50}
    ::user/add-card {::user/update-card 10}
    ::cart/add-item {::cart/add-item 30
    ::cart/update-item 10
    ::cart/remove-item 20
    ::payment/pay 20}
    ::payment/pay {::payment/refund 5}})
    (def infrequent-users-flow …)
    (def rich-users-flow ...)

    View Slide

  48. g3.
    what action might occur after?
    probabilistic modelling of usage
    is effective
    0 and 100 are special cases
    automate the “act” phase
    probability
    matrices for
    flows
    (->> (action-seq-gen most-users-flow)
    gen/sample
    (take 2))
    ([::user/create
    ::user/authenticate
    ::cart/add-item
    ::cart/update-item
    ::cart/update-item
    ::cart/get
    ::cart/update-item
    ::cart/get
    ::cart/remove-item
    ::user/add-card
    ::payment/apply-offer
    ::payment/apply-offer
    ::payment/cancel]
    [::user/create
    ::user/authenticate
    ::cart/add-item
    ::payment/apply-offer
    ::cart/add-item
    ::cart/add-item
    ::cart/add-item
    ::cart/add-item
    ::cart/add-item
    ::cart/get
    ::cart/update-item])

    View Slide

  49. g3.
    what action might occur after?
    probabilistic modelling of usage
    is effective
    0 and 100 are special cases
    automate the “act” phase
    probability
    matrices for
    flows
    [{:action ::user/create
    :params {:email "d3*(@example.com"
    :username "sdf@37"
    ...}}
    {:action ::user/authenticate
    :params {:username “73@fds”
    :password "23nv#2&"
    ...}}
    {:action ::cart/add-item
    :params {:sku 143
    :qty 1
    ...}}
    ...]

    View Slide

  50. g4.
    model close-to real scenarios,
    not completely random ones
    simplifies debugging
    gives deterministic
    deterministic
    seed data
    [{:action ::user/create
    :params {:email "[email protected]"
    :username "bill"
    ...}}
    {:action ::user/authenticate
    :params {:username "bill"
    :password "bill_password"
    ...}}
    {:action ::user/add-card
    :params {:card-numer "1234123412341234"
    :cvc "606"
    ...}}
    ...]

    View Slide

  51. g5.
    software only models subset
    of domain
    tests need to model domain
    around usage
    model external
    domain
    money in user’s card
    payment gateway latency
    inventory of supplier

    View Slide

  52. generation
    execution
    assertion
    diagnosis
    patterns in
    executing
    generative
    tests

    View Slide

  53. 1. store everything immutably
    2. params as latest values in state
    3. catalog known errors
    4. abstract request engine
    patterns in
    executing
    generative
    tests

    View Slide

  54. fn
    input output
    action lifecycle
    e1.

    View Slide

  55. fn
    input output
    param-fn
    generated
    params
    state
    params
    continue?
    action lifecycle
    e1.

    View Slide

  56. fn
    input output
    param-fn
    generated
    params
    state
    params
    DB continue?
    store everything
    e1.

    View Slide

  57. store
    everything
    immutably
    e1.
    ability to query for anything during
    execution, assertion or diagnosis
    tests need to be completely
    independent of source
    timeline matters
    (:require [datascript.core :as d])
    (defn store-action
    [flow-id action-id request response entities]
    (let [metadata {:flow-id flow-id
    :action-id action-id}]
    (d/transact!
    db/conn
    [(-> request
    (assoc :data-type :request)
    (merge metadata))])
    (d/transact!
    db/conn
    [(-> response
    (assoc :data-type :response)
    (merge metadata))])
    (d/transact!
    db/conn
    [(-> entities
    compute-state
    (assoc :data-type :state)
    (merge metadata))])))

    View Slide

  58. store
    everything
    immutably
    e1.
    ability to query for anything during
    execution, assertion or diagnosis
    tests need to be completely
    independent of source
    timeline matters
    (:require [datascript.core :as d])
    (defn store-action
    [flow-id action-id request response entities]
    (let [metadata {:flow-id flow-id
    :action-id action-id}]
    (d/transact!
    db/conn
    [(-> request
    (assoc :data-type :request)
    (merge metadata))])
    (d/transact!
    db/conn
    [(-> response
    (assoc :data-type :response)
    (merge metadata))])
    (d/transact!
    db/conn
    [(-> entities
    compute-state
    (assoc :data-type :state)
    (merge metadata))])))

    View Slide

  59. store
    everything
    immutably
    e1.
    ability to query for anything during
    execution, assertion or diagnosis
    tests need to be completely
    independent of source
    timeline matters
    (defn items [flow-id]
    (->> (d/q '[:find ?sku
    :in $ ?flow-id
    :where
    [?e :flow-id ?flow-id]
    [?e :cart/sku ?sku]]
    @db/conn
    flow-id)
    (map first)))

    View Slide

  60. store
    everything
    immutably
    e1.
    ability to query for anything during
    execution, assertion or diagnosis
    tests need to be completely
    independent of source
    timeline matters
    (defn items [flow-id]
    (->> (d/q '[:find ?sku
    :in $ ?flow-id
    :where
    [?e :flow-id ?flow-id]
    [?e :cart/sku ?sku]]
    @db/conn
    flow-id)
    (map first)))

    View Slide

  61. store
    everything
    immutably
    e1.
    ability to query for anything during
    execution, assertion or diagnosis
    tests need to be completely
    independent of source
    timeline matters
    (defn params-for [flow-id action]
    (->> (d/q '[:find ?request
    :in $ ?flow-id
    :where
    [?request :flow-id ?flow-id]
    [?request :data-type :request]
    [?request :action action]]
    @db/conn
    flow-id)
    (map first)))

    View Slide

  62. store
    everything
    immutably
    e1.
    ability to query for anything during
    execution, assertion or diagnosis
    tests need to be completely
    independent of source
    timeline matters
    (defn params-for [flow-id action]
    (->> (d/q '[:find ?request
    :in $ ?flow-id
    :where
    [?request :flow-id ?flow-id]
    [?request :data-type :request]
    [?request :action action]]
    @db/conn
    flow-id)
    (map first)))

    View Slide

  63. store
    everything
    immutably
    e1.
    ability to query for anything during
    execution, assertion or diagnosis
    tests need to be completely
    independent of source
    timeline matters
    (defn user-timeline [flow-id user-id]
    (->> (d/q '[:find ?e ?t
    :in $ ?flow-id
    :where
    [?e :flow-id ?flow-id]
    [?e :user-id ?user-id ?t]]
    @db/conn
    flow-id)
    (sort-by second >)))

    View Slide

  64. store
    everything
    immutably
    e1.
    ability to query for anything during
    execution, assertion or diagnosis
    tests need to be completely
    independent of source
    timeline matters
    (defn user-timeline [flow-id user-id]
    (->> (d/q '[:find ?e ?t
    :in $ ?flow-id
    :where
    [?e :flow-id ?flow-id]
    [?e :user-id ?user-id ?t]]
    @db/conn
    flow-id)
    (sort-by second >)))

    View Slide

  65. params as
    latest value
    most recent value is most relevant
    use hooks to manipulate engine
    behaviour
    e2.
    (defn latest-param [flow-id attr-name]
    (->> (d/q '[:find ?attr-val ?t
    :in $ ?attr-name ?flow-id
    :where
    [?as :data-type :action-spec]
    [?as :flow-id ?flow-id]
    [?as :action-id ?aid]
    [?e :action-id ?aid]
    [?e ?attr-name ?attr-val ?t]]
    @db/conn
    attr-name
    flow-id)
    (sort-by second >)
    ffirst))

    View Slide

  66. params as
    latest value
    most recent value is most relevant
    use hooks to manipulate engine
    behaviour
    e2.
    (defn latest-param [flow-id attr-name]
    (->> (d/q '[:find ?attr-val ?t
    :in $ ?attr-name ?flow-id
    :where
    [?as :data-type :action-spec]
    [?as :flow-id ?flow-id]
    [?as :action-id ?aid]
    [?e :action-id ?aid]
    [?e ?attr-name ?attr-val ?t]]
    @db/conn
    attr-name
    flow-id)
    (sort-by second >)
    ffirst))

    View Slide

  67. fn
    input output
    param-fn
    generated
    params
    state
    params
    DB continue?
    store everything
    e1.

    View Slide

  68. params as
    latest value
    most recent value is most relevant
    use hooks to manipulate engine
    behaviour
    e2.
    {::user/create
    {:entity-type :user
    :params-fn
    (fn [flow-id params]
    (assoc params :name
    (str "gen-user-" (u/uuid))))}
    ::payment/pay
    {:entity-type :payment
    :params-fn
    (fn [flow-id params]
    (->> (p/fetch-param flow-id :default-card)
    (assoc params :card-number)))}}

    View Slide

  69. catalog
    known errors
    improve relevance
    reduce redundancy
    absorb into system
    e3.
    {::user/create
    {:entity-type :user
    :known-exceptions #{:duplicate-user}
    :params-fn (fn ...)}
    ::payment/pay
    {:entity-type :payment
    :known-exceptions #{:inactive-session
    :payment-complete}
    :params-fn (fn ...)}}

    View Slide

  70. catalog
    known errors
    improve relevance
    reduce redundancy
    absorb into system
    e3.
    {::user/create
    {:entity-type :user
    :known-exceptions #{:duplicate-user}
    :params-fn (fn ...)}
    ::payment/pay
    {:entity-type :payment
    :known-exceptions #{:inactive-session
    :payment-complete}
    :params-fn (fn ...)}}

    View Slide

  71. catalog
    known errors
    improve relevance
    reduce redundancy
    absorb into system
    e3.
    (def known-errors
    {:duplicate-user
    (fn [flow-id params]
    (some?
    (exec-action ::user/get
    (:username params))))
    :payment-complete
    (fn [flow-id params]
    (some? (db/find-action flow-id
    ::payment/pay)))})

    View Slide

  72. abstract
    request engine
    quickly switch to higher or
    lower level of specificity
    api vs controller
    makes debugging simpler
    e4.
    (defn make-request [medium action params]
    (case medium
    :fn ((:fn action) params)
    :bus (bus/request-value action params)
    :http (-> action
    http-request
    (client/request params))))

    View Slide

  73. generation
    execution
    assertion
    diagnosis
    patterns in
    assertions

    View Slide

  74. 1. algebraic properties
    2. state machines as properties
    3. domain based invariants
    patterns in
    assertions

    View Slide

  75. property
    unit
    (no order)
    integration / system
    (ordered)
    no-error* ✓ ✓
    egality ✗ ✓
    idempotence ⍻ ✓
    identity ✓ ✗
    inverse ✗ ✓
    commutativity ✗ ✓
    associativity ✗ ✓
    algebraic properties
    a1.

    View Slide

  76. algebraic properties
    no-error {:message "No :error"
    :result
    (not-any? #(= :error %)
    (map :status responses))}
    ;; status
    (if (or (not (<= 200 status 299))
    (some? exception)
    (timeout?))
    :error)
    no-exceptions
    no-timeouts
    always-available
    a1.

    View Slide

  77. algebraic properties
    egality (assert-equal
    [{:action ::user/signup}]
    [{:action ::user/get}])
    (assert-equal
    [{:action ::cart/update-item
    :params {:deleted? true}}]
    [{:action ::cart/remove-item}])
    f(a) ≡ g(b)
    effective equality
    different paths, same result
    a1.

    View Slide

  78. algebraic properties
    idempotence
    {:percent-flow 10
    :adjacency {:immediate 10
    :distant 90}
    :blacklist #{::payment/pay
    ::payment/refund
    ::payment/cancel
    ::user/signup}}
    f(a) · f(a) ≡ f(a)
    2 kinds of adjacencies
    blacklist / whitelist
    sensitive / distributed systems
    a b c c d e
    a b c d c e
    immediate
    distant
    a1.

    View Slide

  79. algebraic properties
    inverse
    (assert-equal
    [{:action ::cart/add-item}]
    [{:action ::cart/remove-item}]
    :value-action ::cart/get)
    ;; normalised
    (assert-equal
    [{:action ::payment/pay}
    {:action ::payment/cancel}
    {:action ::payment/pay}]
    [{:action ::payment/pay}]
    :value-action ::payment/receipt)
    f(a) · f -1(a) ≡ a
    f -1(f(f -1(a)))=f -1(a)
    normalised inverse
    a1.

    View Slide

  80. algebraic properties
    commutativity
    (assert-equal
    [{:action ::cart/add-item
    :params {:sku 1}}
    {:action ::cart/add-item
    :params {:sku 2}}]
    [{:action ::cart/add-item
    :params {:sku 2}}
    {:action ::cart/add-item
    :params {:sku 1}}]
    :value-action ::cart/get)
    f(a) · g(a) ≡ g(a) · f(a)
    non-compliance is a property too
    a1.

    View Slide

  81. algebraic properties
    commutativity
    (assert-commutative
    [{:action ::cart/add-item
    :params {:sku 1}}
    {:action ::cart/add-item
    :params {:sku 2}}]
    :value-action ::cart/get)
    (assert-not-commutative
    [{:action ::cart/update-item
    :params {:sku 1 :qty 1}}
    {:action ::cart/update-item
    :params {:sku 1 :qty 6}}]
    :value-action ::cart/get)
    f(a) · g(a) ≡ g(a) · f(a)
    non-compliance is a property too
    a1.

    View Slide

  82. state machines as properties
    new has-items success
    cancelled
    refunded
    authenticate
    add-item
    update-item
    remove-item
    apply-offer
    pay
    refund
    cancel
    remove-item
    a2.

    View Slide

  83. (defn compute-expected-state [state action status]
    (match
    [state action status]
    [_ ::user/authenticate :ok] "new"
    ["new" ::cart/add-item :ok] "has-items"
    ["has-items" ::cart/update-item :ok] "has-items"
    ["has-items" ::cart/remove-item :ok] "has-items"
    ["has-items" ::cart/remove-item :ok] "new"
    ["has-items" ::payment/apply-offer :ok] "has-items"
    ["has-items" ::payment/pay :ok] "success"
    ["success" ::payment/refund :ok] "refunded"
    ["has-items" ::payment/cancel :ok] "cancelled"
    [_ _ :error] "error"
    [_ _ :known-exception] "known-exception"))
    a2.

    View Slide

  84. domain based invariants
    for completeness
    create-count-is-count-of-list
    lookups-return-the-same-thing
    num-items-sold-matches-inventory
    receipt-totals-equal-profit
    a3.

    View Slide

  85. generation
    execution
    assertion
    diagnosis
    patterns in
    diagnosis

    View Slide

  86. 1. tests as portable data
    2. domain based checkpoints
    3. flow timeline, and walk throughs
    4. automated bug reports
    patterns in
    diagnosis

    View Slide

  87. tests as
    portable data
    flow / action-specs are stored tests
    determinism 㱺 reproducibility
    store passing and failing tests
    version tests
    d1.
    [{:action ::user/create
    :params {:email "d3*(@example.com"
    :username "sdf@37"
    ...}}
    {:action ::user/authenticate
    :params {:username “73@fds”
    :password "23nv#2&"
    ...}}
    {:action ::cart/add-item
    :params {:sku 143
    :qty 1
    ...}}
    ...]

    View Slide

  88. tests as
    portable data
    flow / action-specs are stored tests
    determinism 㱺 reproducibility
    store passing and failing tests
    version tests
    d1.
    (-> "rc42-login-bug.edn"
    test/import
    test/run)
    (-> "v42-regression-test.edn"
    test/import
    test/run)

    View Slide

  89. domain based
    checkpoints
    think of code as a trie
    useful when there are 100s of flows
    as on CI
    d2.
    {:checkpoints
    [{:action ::user/authenticate
    :description "Logged in"}
    {:action ::cart/add-item
    :description "Added item to cart"}
    {:action ::payment/pay
    :description "Payment attempted"}]}

    View Slide

  90. flow timeline,
    walkthroughs
    d3.
    | action | status | action-id | time |
    |-----------------------+--------+-------------------+------------|
    | ::user/create | :ok | 1d8040ee-2486-... | 1524287925 |
    | ::user/authenticate | :ok | 92c6fb41-ad78-... | 1524287925 |
    | ::cart/add-item | :ok | 20c55130-85d2-... | 1524287927 |
    | ::payment/apply-offer | :ok | 239ebf75-a6c4-... | 1524287928 |
    | ::cart/add-item | :ok | d991dc62-5322-... | 1524287928 |
    | ::cart/add-item | :ok | f4b02a65-ac8f-... | 1524287928 |
    | ::cart/add-item | :ok | e0b85377-b6b7-... | 1524287931 |
    | ::cart/add-item | :ok | cb676220-8d6a-... | 1524287942 |
    | ::cart/add-item | :ok | 1d9546fb-7356-... | 1524287942 |
    | ::cart/get | :ok | b282a885-6e32-... | 1524287942 |
    | ::cart/update-item | :ex | ed062dca-f457-... | 1524287943 |
    timeline of actions relating to a user
    step-by-step diagnosis of events
    everything has an id, and a timestamp

    View Slide

  91. flow timeline,
    walkthroughs
    d3.
    (dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6")
    {:metadata {:action ::cart/update-item
    :flow-id "35cb3104f-e0b0-..."
    :action-id "1d9546fb-7356-..."}}
    [{:data-type :action-spec
    :params {:cart-id 3412.. :sku 142.. :qty 947591}}
    {:data-type :request
    :params {:cart-id 3412.. :sku 142.. :qty 947591}}
    {:data-type :response
    :response
    {:status :error,
    :error-type java.lang.NullPointerException,
    :report
    {:cause #error
    {
    :cause nil
    :via
    [{:type java.lang.NullPointerException
    :message nil
    :at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}]
    :trace
    [[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]
    [demo-proj.modules.cart$extract_filename invoke "cart.clj" 74]
    [demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139]
    [demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134]
    [clojure.lang.Var invoke "Var.java" 379]
    [demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke
    "logging.clj" 11]
    [intbus.core$invoke_handler_fn invokeStatic "core.clj" 134]
    [intbus.core$invoke_handler_fn invoke "core.clj" 129]
    [intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}]
    timeline of actions relating to a user
    step-by-step diagnosis of events
    everything has an id, and a timestamp

    View Slide

  92. flow timeline,
    walkthroughs
    d3.
    (dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6")
    {:metadata {:action ::cart/update-item
    :flow-id "35cb3104f-e0b0-..."
    :action-id "1d9546fb-7356-..."}}
    [{:data-type :action-spec
    :params {:cart-id 3412.. :sku 142.. :qty 947591}}
    {:data-type :request
    :params {:cart-id 3412.. :sku 142.. :qty 947591}}
    {:data-type :response
    :response
    {:status :error,
    :error-type java.lang.NullPointerException,
    :report
    {:cause #error
    {
    :cause nil
    :via
    [{:type java.lang.NullPointerException
    :message nil
    :at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}]
    :trace
    [[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]
    [demo-proj.modules.cart$extract_filename invoke "cart.clj" 74]
    [demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139]
    [demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134]
    [clojure.lang.Var invoke "Var.java" 379]
    [demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke
    "logging.clj" 11]
    [intbus.core$invoke_handler_fn invokeStatic "core.clj" 134]
    [intbus.core$invoke_handler_fn invoke "core.clj" 129]
    [intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}]
    timeline of actions relating to a user
    step-by-step diagnosis of events
    everything has an id, and a timestamp

    View Slide

  93. flow timeline,
    walkthroughs
    d3.
    (dig/action "ed062dca-f457-4c8a-9ffd-77a19200d9d6")
    {:metadata {:action ::cart/update-item
    :flow-id "35cb3104f-e0b0-..."
    :action-id "1d9546fb-7356-..."}}
    [{:data-type :action-spec
    :params {:cart-id 3412.. :sku 142.. :qty 947591}}
    {:data-type :request
    :params {:cart-id 3412.. :sku 142.. :qty 947591}}
    {:data-type :response
    :response
    {:status :error,
    :error-type java.lang.NullPointerException,
    :report
    {:cause #error
    {
    :cause nil
    :via
    [{:type java.lang.NullPointerException
    :message nil
    :at [demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]}]
    :trace
    [[demo-proj.modules.cart$extract_filename invokeStatic "cart.clj" 77]
    [demo-proj.modules.cart$extract_filename invoke "cart.clj" 74]
    [demo-proj.modules.cart$delete_by_id_BANG_ invokeStatic "cart.clj" 139]
    [demo-proj.modules.cart$delete_by_id_BANG_ invoke "cart.clj" 134]
    [clojure.lang.Var invoke "Var.java" 379]
    [demo-proj.intbus.middleware.logging$wrap_intbus_logging$fn__41859 invoke
    "logging.clj" 11]
    [intbus.core$invoke_handler_fn invokeStatic "core.clj" 134]
    [intbus.core$invoke_handler_fn invoke "core.clj" 129]
    [intbus.core$invoke_handler invokeStatic "core.clj" 157]]}}}}]
    timeline of actions relating to a user
    step-by-step diagnosis of events
    everything has an id, and a timestamp

    View Slide

  94. automated
    bug reports
    d4.
    write code to summarise events
    create issues from CI to github

    View Slide

  95. automated
    bug reports
    d4.
    write code to summarise events
    create issues from CI to github

    View Slide

  96. 1. derive parameter specification
    2. declare action dependencies
    3. probability matrices for flows
    4. deterministic seed data
    5. model external domain
    1. store everything immutably
    2. params as latest values in state
    3. catalog known errors
    4. abstract request engine
    1. algebraic properties
    2. state machines as properties
    3. domain based invariants
    1. tests as portable data
    2. domain based checkpoints
    3. flow timeline, and walk throughs
    4. automated bug reports
    generation
    assertion
    execution
    diagnosis

    View Slide

  97. a thought paradigm
    automating software testing
    patterns

    View Slide

  98. references
    microsoft PEX patterns
    datomic/simulant

    View Slide

  99. practical generative
    testing patterns
    srihari sriraman | ø nilenso

    View Slide