Generating Generators

Presented at Clojure Conj 2104

Property-based testing provides good test coverage and automatic shrinking of failure cases. However, coding universal properties with the test.check generator combinators is somewhat of an art. In many cases, it's easier to start from a declarative description of the test data. The Herbert library automatically generates test.check generators from EDN schemas. Learn how schemas can offer simplified testing, easier maintenance and better documentation for your Clojure code.

video: https://www.youtube.com/watch?v=4JGu33WF0Us

Transcript

1. Steve Miner
   @miner
   Generating Generators
   Clojure Conj 2014
   

   View Slide

2. Background
   • The Way to EDN at Conj 2013
   • Testing the Hard Stuff and Staying Sane

   John Hughes at Clojure/West 2014
   • Powerful Testing with test.check

   Reid Draper at Clojure/West 2014
   

   View Slide

3. Overview
   • Herbert schemas
   • Property-based testing
   • Generating generators
   

   View Slide

4. Conj 2013
   @miner
   

   View Slide

5. View Slide

6. edn-format.org
   edn is a system for the conveyance of values.
   It is not a type system, and has no schemas.
   

   View Slide

7. Herbert
   a schema language for EDN
   

   View Slide

8. Schema
   the shape of data
   

   View Slide

9. Herbert Schemas
   • Documentation
   • Validation
   • Pattern matching
   • test.check generators
   

   View Slide

10. Patterns
    • Literals - :kw, “foo”, 42, nil, true, false
    • Symbols - int, kw, str, sym, list, vec
    • Maps - {:a int :b sym}
    • Vectors - [int kw]
    • Operators - (or nil (and int (not 42))
    

    View Slide

11. Quantifiers
    • (? kw)
    • (* int)
    • (+ sym)
    • kw? int* sym+
    

    View Slide

12. {:a int :b? sym :c [str*]}
    matches:
    {:a 42 :b foo :c [“abc” “def”]}
    {:a 42 :c [“x” “y”]}
    {:a 42 :b foo :c []}
    

    View Slide

13. Herbert API
    (conforms? pattern value)
    (conform pattern)
    (def my-test? (conform ‘[(:= K kw) sym+]))
    (my-test? ‘[:a foo bar baz])

    ;=> {K :a}
    

    View Slide

14. Property-based testing
    test.check
    

    View Slide

15. Property-based testing
    • Test against randomly generated inputs
    • Universally quantified

    (for-all [n gen/int] (> (* n n) n))))
    • Shrinking failures

    {:result false, :shrunk {:smallest [0]}}
    • Predicate holds across generated data
    • Round trip consistency
    • Oracle, known-good implementation
    

    View Slide

16. Properties
    • for-all

    (for-all [a gen/int b gen/int] (>= (+ a b) a))

    (for-all* [gen/int gen/int] (fn [a b] (>= (+ a b) a)))
    • quick-check

    (quick-check 100 (for-all [a gen/s-pos-int] (> (* a a) a)))

    {:result false, :shrunk {:smallest [1]}}
    

    View Slide

17. Monad
    

    View Slide

18. Generators & Combinators
    • choose, return, int, string, keyword, etc
    • hash-map, map, vector, list, tuple
    • one-of, element, frequency, such-that
    • fmap
    • bind
    • new: recursive-gen, shuffle
    

    View Slide

19. View Slide

20. Vector and an Element
    (gen/bind (gen/not-empty (gen/vector gen/int))

    #(gen/tuple (gen/return %) (gen/elements %))))
    [[2 2] 2], [[4 4 -1 2] 4], [[-3 -1 -5 3 2] -5]
    (hg/generator ‘[(:= V [int+]) (in V)])
    

    View Slide

21. Optional Keys
    {:a? int :b? sym}
    {:b L} {:a 0} {:b ZB, :a -9} {:b 7B.i2x/P, :a 0} {:a -9} {}
    {:b XV2/o?---} {:b !h+?0, :a 1} {:a 12} {} 

    {:b Q?-JM9*9/M, :a 10}
    

    View Slide

22. Regexs
    (str #"fo*bar+")
    "fbarrr" "foobarrrrrrrrrrrrrrrrr" "fooooooooooooobarrrr" “fbarr"
    (kw #":[a-z]")
    :m :l :u :v :j :l :k :n :p :b :x :u :x :j :h :d :m :d :l :q
    

    View Slide

23. Bindings
    [[(:= Low int 9) (:= Hi int 10 99)] (int+ Low Hi)]

    [[0 99] 86]
    [[9 71] 9 66 9 9 9 71 39 7]
    [[0 99] 99 0 0 63 42 0 0 17 26 0 0 99 0 0 99]
    

    View Slide

24. Property from Schemas
    (defn property 

    ([pred schema] (prop/for-all* [(generator schema)] pred))

    

    ([pred schema1 schema2] 

    (prop/for-all* [(generator schema1) (generator schema2)] pred))

    

    ([pred schema1 schema2 & more]

    (prop/for-all* (list* (generator schema1) (generator schema2) 

    (map generator more)) 

    pred)))
    

    View Slide

25. Herbert check
    (defn check

    ([pred schema] (check 100 pred schema))

    ([trials pred schema] 

    (tc/quick-check trials (property pred schema))))
    (check #(pos? (count %)) ‘[int*])
    {:result false, :failing-size 0, :fail [[]], :shrunk {:smallest
    [[]]}}
    

    View Slide

26. Self-testing
    (defn gen-test

    ([schema] (gen-test schema 100))

    ([schema num]

    (let [confn (conform schema)

    result (hg/check num confn schema)]

    (is (:result result) 

    (str "Schema: " schema " failed "

    (first (get-in result [:shrunk :smallest])))))))
    

    View Slide

27. Herbert Summary
    • schema = EDN in EDN
    • test.check property = predicate + generator
    • (conform schema) — predicate
    • (generator schema) — generator
    

    View Slide

28. View Slide

29. Generator Limitations
    • Negation
    • (not 42) — works with literals and simple types
    • (not (and int pos)) — throws
    • (or (not int) (not pos))) — works
    • Conjunction
    • (and int (not 42)) — works
    • (and (str #“foo.*”) (str #“.*bar”)) — throws
    • (or (and even odd) 42) — throws
    

    View Slide

30. To Do
    • Performance
    • Nested binding generators
    • Extensible generators
    • Cognitect Transit types
    • Debugging and visualization
    

    View Slide

31. Thanks
    • Rich Hickey - edn-format.org
    • Eric Normand - squarepeg parser
    • Reid Draper - test.check
    • Gary Fredericks - re-rand and test.chuck
    • Zach Tellman - collection-check
    • http://tos.trekcore.com for Star Trek photos
    

    View Slide

32. github.com/miner/herbert
    

    View Slide

33. Star Trek II: 

    The Wrath of Khan
    1982
    Genesis Device
    Clojure Conj 2014
    

    View Slide

34. THE END
    

    View Slide