Generating Generators

925e5591fbb086894c2b09b54a18f1e2?s=47 miner
November 20, 2014

Generating Generators

Presented at Clojure Conj 2104

Property-based testing provides good test coverage and automatic shrinking of failure cases. However, coding universal properties with the test.check generator combinators is somewhat of an art. In many cases, it's easier to start from a declarative description of the test data. The Herbert library automatically generates test.check generators from EDN schemas. Learn how schemas can offer simplified testing, easier maintenance and better documentation for your Clojure code.

video: https://www.youtube.com/watch?v=4JGu33WF0Us

925e5591fbb086894c2b09b54a18f1e2?s=128

miner

November 20, 2014
Tweet

Transcript

  1. Steve Miner @miner Generating Generators Clojure Conj 2014

  2. Background • The Way to EDN at Conj 2013 •

    Testing the Hard Stuff and Staying Sane
 John Hughes at Clojure/West 2014 • Powerful Testing with test.check
 Reid Draper at Clojure/West 2014
  3. Overview • Herbert schemas • Property-based testing • Generating generators

  4. Conj 2013 @miner

  5. None
  6. edn-format.org edn is a system for the conveyance of values.

    It is not a type system, and has no schemas.
  7. Herbert a schema language for EDN

  8. Schema the shape of data

  9. Herbert Schemas • Documentation • Validation • Pattern matching •

    test.check generators
  10. Patterns • Literals - :kw, “foo”, 42, nil, true, false

    • Symbols - int, kw, str, sym, list, vec • Maps - {:a int :b sym} • Vectors - [int kw] • Operators - (or nil (and int (not 42))
  11. Quantifiers • (? kw) • (* int) • (+ sym)

    • kw? int* sym+
  12. {:a int :b? sym :c [str*]} matches: {:a 42 :b

    foo :c [“abc” “def”]} {:a 42 :c [“x” “y”]} {:a 42 :b foo :c []}
  13. Herbert API (conforms? pattern value) (conform pattern) (def my-test? (conform

    ‘[(:= K kw) sym+])) (my-test? ‘[:a foo bar baz])
 ;=> {K :a}
  14. Property-based testing test.check

  15. Property-based testing • Test against randomly generated inputs • Universally

    quantified
 (for-all [n gen/int] (> (* n n) n)))) • Shrinking failures
 {:result false, :shrunk {:smallest [0]}} • Predicate holds across generated data • Round trip consistency • Oracle, known-good implementation
  16. Properties • for-all
 (for-all [a gen/int b gen/int] (>= (+

    a b) a))
 (for-all* [gen/int gen/int] (fn [a b] (>= (+ a b) a))) • quick-check
 (quick-check 100 (for-all [a gen/s-pos-int] (> (* a a) a)))
 {:result false, :shrunk {:smallest [1]}}
  17. Monad

  18. Generators & Combinators • choose, return, int, string, keyword, etc

    • hash-map, map, vector, list, tuple • one-of, element, frequency, such-that • fmap • bind • new: recursive-gen, shuffle
  19. None
  20. Vector and an Element (gen/bind (gen/not-empty (gen/vector gen/int))
 #(gen/tuple (gen/return

    %) (gen/elements %)))) [[2 2] 2], [[4 4 -1 2] 4], [[-3 -1 -5 3 2] -5] (hg/generator ‘[(:= V [int+]) (in V)])
  21. Optional Keys {:a? int :b? sym} {:b L} {:a 0}

    {:b ZB, :a -9} {:b 7B.i2x/P, :a 0} {:a -9} {} {:b XV2/o?---} {:b !h+?0, :a 1} {:a 12} {} 
 {:b Q?-JM9*9/M, :a 10}
  22. Regexs (str #"fo*bar+") "fbarrr" "foobarrrrrrrrrrrrrrrrr" "fooooooooooooobarrrr" “fbarr" (kw #":[a-z]") :m

    :l :u :v :j :l :k :n :p :b :x :u :x :j :h :d :m :d :l :q
  23. Bindings [[(:= Low int 9) (:= Hi int 10 99)]

    (int+ Low Hi)]
 [[0 99] 86] [[9 71] 9 66 9 9 9 71 39 7] [[0 99] 99 0 0 63 42 0 0 17 26 0 0 99 0 0 99]
  24. Property from Schemas (defn property 
 ([pred schema] (prop/for-all* [(generator

    schema)] pred))
 
 ([pred schema1 schema2] 
 (prop/for-all* [(generator schema1) (generator schema2)] pred))
 
 ([pred schema1 schema2 & more]
 (prop/for-all* (list* (generator schema1) (generator schema2) 
 (map generator more)) 
 pred)))
  25. Herbert check (defn check
 ([pred schema] (check 100 pred schema))


    ([trials pred schema] 
 (tc/quick-check trials (property pred schema)))) (check #(pos? (count %)) ‘[int*]) {:result false, :failing-size 0, :fail [[]], :shrunk {:smallest [[]]}}
  26. Self-testing (defn gen-test
 ([schema] (gen-test schema 100))
 ([schema num]
 (let

    [confn (conform schema)
 result (hg/check num confn schema)]
 (is (:result result) 
 (str "Schema: " schema " failed "
 (first (get-in result [:shrunk :smallest])))))))
  27. Herbert Summary • schema = EDN in EDN • test.check

    property = predicate + generator • (conform schema) — predicate • (generator schema) — generator
  28. None
  29. Generator Limitations • Negation • (not 42) — works with

    literals and simple types • (not (and int pos)) — throws • (or (not int) (not pos))) — works • Conjunction • (and int (not 42)) — works • (and (str #“foo.*”) (str #“.*bar”)) — throws • (or (and even odd) 42) — throws
  30. To Do • Performance • Nested binding generators • Extensible

    generators • Cognitect Transit types • Debugging and visualization
  31. Thanks • Rich Hickey - edn-format.org • Eric Normand -

    squarepeg parser • Reid Draper - test.check • Gary Fredericks - re-rand and test.chuck • Zach Tellman - collection-check • http://tos.trekcore.com for Star Trek photos
  32. github.com/miner/herbert

  33. Star Trek II: 
 The Wrath of Khan 1982 Genesis

    Device Clojure Conj 2014
  34. THE END