Property-based testing with Clojure test.check

Traditional unit or example-based tests are a useful tool for developing confidence in correctness. Trouble is, they require linear effort to increase this confidence -- it takes twice as long to write eight tests as it does four. Furthermore, example-based tests are limited by their authors imagination.

Property-based testing takes a different approach: tests are written as properties that should hold true for an entire domain of randomly-generated input. The number of test-cases generated are limited only by your compute-resources and patience. In this talk, we'll take a look at the principles of property-based testing, as well as some pragmatic examples.

Transcript

1. Property-based testing with Clojure test.check Reid Draper @reiddraper
2. (= a b )
3. (= [5 2 1] (reverse [1 2 5])) (= [1 2 3 4 5] (reverse [5 4 3 2 1])) (= [5] (reverse [5])) (= [] (reverse []))
4. (= [5 2 1] (reverse [1 2 5])) (= [1 2 3 4 5] (reverse [5 4 3 2 1])) (= [5] (reverse [5])) (= [] (reverse [])) (= [7 8 2 5] (reverse [5 2 8 7])) (= [55 54 53] (reverse [53 54 55])) (= [-15 15] (reverse [15 -15])) (= [true false] (reverse [false true]))
5. (def reverse-property (prop/for-all [a (gen/vector gen/int)] (reversed? a (reverse a))))
9. Property-based testing is a powerful testing technique, embodied by Clojure test.check.
10. example-based testing property-based testing test.check
11. example-based testing property-based testing test.check
14. 0 ‘() [] {} Integer/MAX_VALUE false -1
15. Our tests are only as mischievous as our imagination
16. {:3aBA "~e", ({"" false} \") ("r" "" {{} 0, :y {0 \+}}), [2 {} [[] {0 :v7}]] -1/4, (1 2 \') {}, :4 ["" ({0 true})], {false {}, {(0) (\^)} 0, ("m" [false]) ((true) {}), [[\t] ()] {}} -3}
17. (= a b )
18. A change in perspective is worth 80 IQ points. - Alan Kay
19. example-based testing property-based testing test.check
20. generative testing random testing
21. never forgets edge-cases
22. thousands of tests
23. encodes a function’s invariants directly
24. (concat a b) ;; Returns a lazy seq representing the concatenation of the elements in the supplied colls.
25. (prop/for-all [a (gen/vector gen/any) b (gen/vector gen/any)] (= (count (concat a b)) (+ (count a) (count b))))
26. (prop/for-all [a (gen/vector gen/any) b (gen/vector gen/any)] (= (count (concat a b)) (+ (count a) (count b)))) universal-quantification
27. (prop/for-all [a (gen/vector gen/any) b (gen/vector gen/any)] (= (count (concat a b)) (+ (count a) (count b)))) binding
28. (prop/for-all [a (gen/vector gen/any) b (gen/vector gen/any)] (= (count (concat a b)) (+ (count a) (count b)))) property
29. (tc/quick-check 100 my-property) ;; => {:result true, ;; :num-tests 100, ;; :seed 1395110930836}
30. thinking in properties
31. roundtrip trusted-implementation input/output relation
32. roundtrip
33. (prop/for-all [a gen/any-printable] (= a (-> a prn-str edn/read-string))) serialize de-serialize
34. trusted-implementation
35. (def transient-property (prop/for-all [a (gen/vector gen-action)] (= (apply-actions #{} a) (apply-actions #{} (filter-transients a)))))
36. input/output relation
37. (prop/for-all [a (gen/vector gen/any) b (gen/vector gen/any)] (= (count (concat a b)) (+ (count a) (count b))))
38. example-based testing property-based testing test.check
39. github.com/clojure/ test.check
40. clojure clojurescript pedestal core.matrix collection-check byte-streams
41. quickcheck Haskell (QuickCheck) 2000 Erlang (EQC) 2006 Scala (ScalaCheck) ~2008
42. let's see an example
43. (-> #{} (conj 109) (conj -110) transient (disj! -110) persistent! (conj -110))
44. (-> #{} (conj 109) (conj -110) transient (disj! -110) persistent! (conj -110))
45. (def transient-property (prop/for-all [a (gen/vector gen-action)] (= (apply-actions #{} a) (apply-actions #{} (filter-transients a)))))
46. (tc/quick-check 1e5 ;; 100,000 transient-property)
47. {:result false, :failing-size 92, :num-tests 2893, :fail "...", :shrunk {:total-nodes-visited 440 :depth 83 :result false :smallest "..."}}
48. [[:conj 109] [:conj -110] [:transient ] [:disj -110] [:persistent!] [:conj -110]]
49. [[:disj -108] [:disj -24] [:disj 89] [:transient] [:conj 12] [:disj -136] [:transient] [:conj -145] [:conj -8] [:persistent!] [:persistent!] [:conj 25] [:conj 142] [:conj 146] [:persistent!] [:transient] [:conj 95] [:persistent!] [:disj -86] [:transient] [:persistent!] [:transient] [:persistent!] [:conj -109] [:transient] [:transient] [:persistent!] [:disj -6] [:transient] [:persistent!] [:conj -50] [:persistent!] [:conj -166] [:transient] [:conj 136] [:persistent!] [:persistent!] [:transient] [:disj 155] [:persistent!] [:transient] [:transient] [:disj 143] [:persistent!] [:persistent!] [:transient] [:conj -10] [:transient] [:conj -59] [:conj -34] [:transient] [:transient] [:disj 111] [:transient] [:disj -33] [:transient] [:conj -96] [:transient] [:disj 93] [:disj -130] [:conj 88] [:disj 14] [:transient] [:disj 9] [:persistent!] [:disj 86] [:conj 109] [:conj -45] [:persistent!] [:persistent!] [:disj -141] [:conj -14] [:persistent!] [:persistent!] [:conj -104] [:conj -83] [:persistent!] [:persistent!] [:conj -73] [:conj 13] [:persistent!] [:persistent!] [:persistent!] [:disj -14] [:conj -129] [:disj -53] [:transient] [:conj -110] [:conj 60] [:conj 11] [:disj -128] [:disj -42] [:disj 18] [:disj 73] [:disj 33] [:transient] [:persistent!] [:persistent!] [:transient] [:persistent!] [:transient] [:disj -123] [:disj -141] [:conj -26] [:conj -92] [:conj -116] [:conj 101] [:disj -133] [:conj 58] [:transient] [:disj -110] [:persistent!] [:disj 105] [:conj 26] [:conj -110] [:persistent!] [:transient] [:persistent!] [:persistent!] [:disj -5] [:conj 117] [:transient] [:persistent!] [:disj 96] [:persistent!] [:disj 52] [:persistent!] [:disj -132] [:transient] [:transient] [:conj -108] [:disj 121] [:persistent!]]
50. [[:conj 109] [:conj -110] [:transient ] [:disj -110] [:persistent!] [:conj -110]]
51. CLJ-1285
52. fixed in clojure 1.6
53. github.com/reiddraper/ clojure-transient-test
54. generators
55. (prop/for-all [a (gen/vector gen/any) b (gen/vector gen/any)] (= (count (concat a b)) (+ (count a) (count b)))) generators
56. any any-printable boolean byte bytes char char- alpha-numeric char-ascii hash-map int keyword list map nat neg-int pos-int ratio s-neg-int s-pos- int string string-alpha-numeric string-ascii tuple vector
57. (gen/sample gen/boolean) (false false false false false true false false true true) boolean
58. (gen/sample gen/int) (0 0 -2 -3 1 3 -2 0 0 -9) int
59. (gen/sample (gen/vector gen/int)) ([] [1] [2 0] [1 2 -1] [] [0] [-4] [] [-8] [4 0 2 -8 -9]) vector
60. (last (gen/sample gen/any-printable)) {:3aBA "~e", ({"" false} \") ("r" "" {{} 0, :y {0 \+}}), [2 {} [[] {0 :v7}]] -1/4, (1 2 \') {}, :4 ["" ({0 true})], {false {}, {(0) (\^)} 0, ("m" [false]) ((true) {}), [[\t] ()] {}} -3} any
61. bind choose elements frequency no-shrink not- empty resize return sample sample-seq shrink-2 sized such-that
62. (gen/sample (gen/return :foo)) (:foo :foo :foo :foo :foo :foo :foo :foo :foo :foo) return
63. (gen/sample (gen/choose 5 10)) (7 8 9 6 6 5 8 9 10 10) choose
64. (gen/sample (gen/elements [:clojure :haskell :erlang :python :ruby])) (:clojure :haskell :haskell :clojure :clojure :python :cloj ure :clojure :ruby :ruby) elements
65. (gen/sample (gen/one-of [gen/boolean gen/byte])) (119 true -24 120 -93 true false 27 false false) one-of
66. (gen/sample (gen/frequency [[5 (gen/return :weekday)] [2 (gen/return :weekend)]])) (:weekday :weekday :weekend :weekday :weekday :weekend :wee kend :weekend :weekday :weekday) frequency
67. (gen/sample (gen/such-that #(not= 0 %) gen/int)) (-1 -2 -1 2 3 -5 -6 6 -5 -1) such-that
68. (gen/sample (gen/fmap odd? gen/int)) (false true false true true false false false false false) fmap
69. (gen/sample (gen/fmap odd? gen/int)) (false true false true true false false false false false) apply odd? to every generated value
70. (gen/sample (gen/bind (gen/not-empty (gen/vector gen/int)) #(gen/tuple (gen/return %) (gen/elements %)))) bind
71. ([[0] 0] [[1 -2] 1] [[2 2] 2] [[-3 -3] -3] [[4 -4] -4] [[2 1] 2] [[-6 -6 -5 -1] -5] [[8 -5 6 -6 -6 -5 7 -7] 8] [[-3 2 -5 -4] -5] [[7 5] 5]) tuples of vector and chosen element
72. example-based testing property-based testing test.check
73. [org.clojure/test.check "0.6.1"]
74. A change in perspective is worth 80 IQ points. - Alan Kay
75. github.com/clojure/ test.check github.com/reiddraper/ clojure-transient-test