Generating Generators

Presented at Clojure Conj 2104

Property-based testing provides good test coverage and automatic shrinking of failure cases. However, coding universal properties with the test.check generator combinators is somewhat of an art. In many cases, it's easier to start from a declarative description of the test data. The Herbert library automatically generates test.check generators from EDN schemas. Learn how schemas can offer simplified testing, easier maintenance and better documentation for your Clojure code.

video: https://www.youtube.com/watch?v=4JGu33WF0Us

Transcript

1. Steve Miner @miner Generating Generators Clojure Conj 2014
2. Background • The Way to EDN at Conj 2013 • Testing the Hard Stuff and Staying Sane
 John Hughes at Clojure/West 2014 • Powerful Testing with test.check
 Reid Draper at Clojure/West 2014
3. Overview • Herbert schemas • Property-based testing • Generating generators
4. Conj 2013 @miner
6. edn-format.org edn is a system for the conveyance of values. It is not a type system, and has no schemas.
7. Herbert a schema language for EDN
8. Schema the shape of data
9. Herbert Schemas • Documentation • Validation • Pattern matching • test.check generators
10. Patterns • Literals - :kw, “foo”, 42, nil, true, false • Symbols - int, kw, str, sym, list, vec • Maps - {:a int :b sym} • Vectors - [int kw] • Operators - (or nil (and int (not 42))
11. Quantifiers • (? kw) • (* int) • (+ sym) • kw? int* sym+
12. {:a int :b? sym :c [str*]} matches: {:a 42 :b foo :c [“abc” “def”]} {:a 42 :c [“x” “y”]} {:a 42 :b foo :c []}
13. Herbert API (conforms? pattern value) (conform pattern) (def my-test? (conform ‘[(:= K kw) sym+])) (my-test? ‘[:a foo bar baz])
 ;=> {K :a}
14. Property-based testing test.check
15. Property-based testing • Test against randomly generated inputs • Universally quantified
 (for-all [n gen/int] (> (* n n) n)))) • Shrinking failures
 {:result false, :shrunk {:smallest [0]}} • Predicate holds across generated data • Round trip consistency • Oracle, known-good implementation
16. Properties • for-all
 (for-all [a gen/int b gen/int] (>= (+ a b) a))
 (for-all* [gen/int gen/int] (fn [a b] (>= (+ a b) a))) • quick-check
 (quick-check 100 (for-all [a gen/s-pos-int] (> (* a a) a)))
 {:result false, :shrunk {:smallest [1]}}
17. Monad
18. Generators & Combinators • choose, return, int, string, keyword, etc • hash-map, map, vector, list, tuple • one-of, element, frequency, such-that • fmap • bind • new: recursive-gen, shuffle
20. Vector and an Element (gen/bind (gen/not-empty (gen/vector gen/int))
 #(gen/tuple (gen/return %) (gen/elements %)))) [[2 2] 2], [[4 4 -1 2] 4], [[-3 -1 -5 3 2] -5] (hg/generator ‘[(:= V [int+]) (in V)])
21. Optional Keys {:a? int :b? sym} {:b L} {:a 0} {:b ZB, :a -9} {:b 7B.i2x/P, :a 0} {:a -9} {} {:b XV2/o?---} {:b !h+?0, :a 1} {:a 12} {} 
 {:b Q?-JM9*9/M, :a 10}
22. Regexs (str #"fo*bar+") "fbarrr" "foobarrrrrrrrrrrrrrrrr" "fooooooooooooobarrrr" “fbarr" (kw #":[a-z]") :m :l :u :v :j :l :k :n :p :b :x :u :x :j :h :d :m :d :l :q
23. Bindings [[(:= Low int 9) (:= Hi int 10 99)] (int+ Low Hi)]
 [[0 99] 86] [[9 71] 9 66 9 9 9 71 39 7] [[0 99] 99 0 0 63 42 0 0 17 26 0 0 99 0 0 99]
24. Property from Schemas (defn property 
 ([pred schema] (prop/for-all* [(generator schema)] pred))
 
 ([pred schema1 schema2] 
 (prop/for-all* [(generator schema1) (generator schema2)] pred))
 
 ([pred schema1 schema2 & more]
 (prop/for-all* (list* (generator schema1) (generator schema2) 
 (map generator more)) 
 pred)))
25. Herbert check (defn check
 ([pred schema] (check 100 pred schema))
 ([trials pred schema] 
 (tc/quick-check trials (property pred schema)))) (check #(pos? (count %)) ‘[int*]) {:result false, :failing-size 0, :fail [[]], :shrunk {:smallest [[]]}}
26. Self-testing (defn gen-test
 ([schema] (gen-test schema 100))
 ([schema num]
 (let [confn (conform schema)
 result (hg/check num confn schema)]
 (is (:result result) 
 (str "Schema: " schema " failed "
 (first (get-in result [:shrunk :smallest])))))))
27. Herbert Summary • schema = EDN in EDN • test.check property = predicate + generator • (conform schema) — predicate • (generator schema) — generator
29. Generator Limitations • Negation • (not 42) — works with literals and simple types • (not (and int pos)) — throws • (or (not int) (not pos))) — works • Conjunction • (and int (not 42)) — works • (and (str #“foo.*”) (str #“.*bar”)) — throws • (or (and even odd) 42) — throws
30. To Do • Performance • Nested binding generators • Extensible generators • Cognitect Transit types • Debugging and visualization
31. Thanks • Rich Hickey - edn-format.org • Eric Normand - squarepeg parser • Reid Draper - test.check • Gary Fredericks - re-rand and test.chuck • Zach Tellman - collection-check • http://tos.trekcore.com for Star Trek photos
32. github.com/miner/herbert
33. Star Trek II: 
 The Wrath of Khan 1982 Genesis Device Clojure Conj 2014
34. THE END