Generating Generators

925e5591fbb086894c2b09b54a18f1e2?s=47 miner
November 20, 2014

Generating Generators

Presented at Clojure Conj 2104

Property-based testing provides good test coverage and automatic shrinking of failure cases. However, coding universal properties with the test.check generator combinators is somewhat of an art. In many cases, it's easier to start from a declarative description of the test data. The Herbert library automatically generates test.check generators from EDN schemas. Learn how schemas can offer simplified testing, easier maintenance and better documentation for your Clojure code.

video: https://www.youtube.com/watch?v=4JGu33WF0Us

925e5591fbb086894c2b09b54a18f1e2?s=128

miner

November 20, 2014
Tweet

Transcript

  1. 2.

    Background • The Way to EDN at Conj 2013 •

    Testing the Hard Stuff and Staying Sane
 John Hughes at Clojure/West 2014 • Powerful Testing with test.check
 Reid Draper at Clojure/West 2014
  2. 5.
  3. 6.

    edn-format.org edn is a system for the conveyance of values.

    It is not a type system, and has no schemas.
  4. 10.

    Patterns • Literals - :kw, “foo”, 42, nil, true, false

    • Symbols - int, kw, str, sym, list, vec • Maps - {:a int :b sym} • Vectors - [int kw] • Operators - (or nil (and int (not 42))
  5. 12.

    {:a int :b? sym :c [str*]} matches: {:a 42 :b

    foo :c [“abc” “def”]} {:a 42 :c [“x” “y”]} {:a 42 :b foo :c []}
  6. 13.

    Herbert API (conforms? pattern value) (conform pattern) (def my-test? (conform

    ‘[(:= K kw) sym+])) (my-test? ‘[:a foo bar baz])
 ;=> {K :a}
  7. 15.

    Property-based testing • Test against randomly generated inputs • Universally

    quantified
 (for-all [n gen/int] (> (* n n) n)))) • Shrinking failures
 {:result false, :shrunk {:smallest [0]}} • Predicate holds across generated data • Round trip consistency • Oracle, known-good implementation
  8. 16.

    Properties • for-all
 (for-all [a gen/int b gen/int] (>= (+

    a b) a))
 (for-all* [gen/int gen/int] (fn [a b] (>= (+ a b) a))) • quick-check
 (quick-check 100 (for-all [a gen/s-pos-int] (> (* a a) a)))
 {:result false, :shrunk {:smallest [1]}}
  9. 17.
  10. 18.

    Generators & Combinators • choose, return, int, string, keyword, etc

    • hash-map, map, vector, list, tuple • one-of, element, frequency, such-that • fmap • bind • new: recursive-gen, shuffle
  11. 19.
  12. 20.

    Vector and an Element (gen/bind (gen/not-empty (gen/vector gen/int))
 #(gen/tuple (gen/return

    %) (gen/elements %)))) [[2 2] 2], [[4 4 -1 2] 4], [[-3 -1 -5 3 2] -5] (hg/generator ‘[(:= V [int+]) (in V)])
  13. 21.

    Optional Keys {:a? int :b? sym} {:b L} {:a 0}

    {:b ZB, :a -9} {:b 7B.i2x/P, :a 0} {:a -9} {} {:b XV2/o?---} {:b !h+?0, :a 1} {:a 12} {} 
 {:b Q?-JM9*9/M, :a 10}
  14. 23.

    Bindings [[(:= Low int 9) (:= Hi int 10 99)]

    (int+ Low Hi)]
 [[0 99] 86] [[9 71] 9 66 9 9 9 71 39 7] [[0 99] 99 0 0 63 42 0 0 17 26 0 0 99 0 0 99]
  15. 24.

    Property from Schemas (defn property 
 ([pred schema] (prop/for-all* [(generator

    schema)] pred))
 
 ([pred schema1 schema2] 
 (prop/for-all* [(generator schema1) (generator schema2)] pred))
 
 ([pred schema1 schema2 & more]
 (prop/for-all* (list* (generator schema1) (generator schema2) 
 (map generator more)) 
 pred)))
  16. 25.

    Herbert check (defn check
 ([pred schema] (check 100 pred schema))


    ([trials pred schema] 
 (tc/quick-check trials (property pred schema)))) (check #(pos? (count %)) ‘[int*]) {:result false, :failing-size 0, :fail [[]], :shrunk {:smallest [[]]}}
  17. 26.

    Self-testing (defn gen-test
 ([schema] (gen-test schema 100))
 ([schema num]
 (let

    [confn (conform schema)
 result (hg/check num confn schema)]
 (is (:result result) 
 (str "Schema: " schema " failed "
 (first (get-in result [:shrunk :smallest])))))))
  18. 27.

    Herbert Summary • schema = EDN in EDN • test.check

    property = predicate + generator • (conform schema) — predicate • (generator schema) — generator
  19. 28.
  20. 29.

    Generator Limitations • Negation • (not 42) — works with

    literals and simple types • (not (and int pos)) — throws • (or (not int) (not pos))) — works • Conjunction • (and int (not 42)) — works • (and (str #“foo.*”) (str #“.*bar”)) — throws • (or (and even odd) 42) — throws
  21. 30.

    To Do • Performance • Nested binding generators • Extensible

    generators • Cognitect Transit types • Debugging and visualization
  22. 31.

    Thanks • Rich Hickey - edn-format.org • Eric Normand -

    squarepeg parser • Reid Draper - test.check • Gary Fredericks - re-rand and test.chuck • Zach Tellman - collection-check • http://tos.trekcore.com for Star Trek photos
  23. 34.