Powerful Testing with test.check

Clojure/West 2014. Slides to accompany https://www.youtube.com/watch?v=JMhNINPo__g.

Unit-testing is a common tool for gaining confidence in code correctness. However, the benefit from these tests is often linear to the number of hand-written test-cases. test.check (formerly simple-check) is a property-based testing tool based on QuickCheck, which allows for thousands of tests with constant effort.

Tests are written as universal quantifications -- properties which should hold true for all input. test.check then randomly generates test cases, for which the property should pass. In this talk, we'll take a look at the foundations of test.check, implementation details regarding shrinking, and see sophisticated tests of concurrent and non-deterministic code.

Transcript

1. clojure.test.check Reid Draper @reiddraper
2. testing
3. performance
9. test.check
10. github.com/clojure/ test.check
11. previously called simple-check
12. let's see an example
13. (-> #{} (conj 109) (conj -110)! transient (disj! -110)! persistent! (conj -110))!
14. (-> #{} (conj 109) (conj -110)! transient (disj! -110)! persistent! (conj -110))!
15. (def transient-property! (prop/for-all! [a (gen/vector gen-action)]! (= (apply-actions #{} a)! (apply-actions #{} (filter-actions a)))))!
16. (tc/quick-check! 1e5 ;; 100,000! transient-property)!
17. {:result false,! :failing-size 92,! :num-tests 2893,! :fail "...",! :shrunk! {:total-nodes-visited 440! :depth 83! :result false! :smallest "..."}}!
18. [[:conj 109] [:conj -110] [:transient ]! [:disj -110] [:persistent!] [:conj -110]]!
19. CLJ-1285
20. fixed in clojure 1.6
21. github.com/reiddraper/ clojure-transient-test
22. clojure clojurescript pedestal core.matrix collection-check byte-streams
23. quickcheck Haskell (QuickCheck) 2000 Erlang (EQC) 2006 Scala (ScalaCheck) ~2008
24. property-based testing generative-testing random-testing
25. (prop/for-all! [a (gen/vector gen/any)! b (gen/vector gen/any)]! (= (count (concat a b))! (+ (count a) (count b))))!
26. (prop/for-all! [a (gen/vector gen/any)! b (gen/vector gen/any)]! (= (count (concat a b))! (+ (count a) (count b))))! universal-quantification
27. (prop/for-all! [a (gen/vector gen/any)! b (gen/vector gen/any)]! (= (count (concat a b))! (+ (count a) (count b))))! binding
28. (prop/for-all! [a (gen/vector gen/any)! b (gen/vector gen/any)]! (= (count (concat a b))! (+ (count a) (count b))))! property
29. ;; `p` is the aforementioned code! (tc/quick-check 100 p)! ;; => {:result true, :num-tests ! 100, :seed 1395110930836}!
30. thinking in properties
31. roundtrip trusted-implementation input/output relation
32. roundtrip
33. (prop/for-all! [a gen/any-printable]! (= a (-> a prn-str! edn/read-string)))! serialize de-serialize
34. trusted-implementation
35. (def transient-property! (prop/for-all! [a (gen/vector gen-action)]! (= (apply-actions #{} a)! (apply-actions #{} (filter-actions a)))))!
36. input/output relation
37. (prop/for-all! [a (gen/vector gen/any)! b (gen/vector gen/any)]! (= (count (concat a b))! (+ (count a) (count b))))!
38. generators
39. (prop/for-all! [a (gen/vector gen/any)! b (gen/vector gen/any)]! (= (count (concat a b))! (+ (count a) (count b))))! generators
40. any any-printable boolean byte bytes char char- alpha-numeric char-ascii hash-map int keyword list map nat neg-int pos-int ratio s-neg-int s-pos- int string string-alpha-numeric string-ascii tuple vector
41. (gen/sample gen/boolean)! (false false false false false true false false true true)! ! boolean
42. (gen/sample gen/int)! (0 0 -2 -3 1 3 -2 0 0 -9)! int
43. (gen/sample (gen/vector gen/int))! ([] [1] [2 0] [1 2 -1] [] [0] [-4] [] [-8] [4 0 2 -8 -9])! ! vector
44. (last (gen/sample gen/any-printable))! {:3aBA "~e",! ({"" false} \") ("r" "" {{} 0, :y {0 \+}}),! [2 {} [[] {0 :v7}]] -1/4,! (1 2 \') {},! :4 ["" ({0 true})],! {false {}, {(0) (\^)} 0, ("m" [false]) ((true) {}), [[\t] ()] {}} -3}! any
45. bind choose elements frequency no-shrink not- empty resize return sample sample-seq shrink-2 sized such-that
46. (gen/sample (gen/return :foo))! (:foo :foo :foo :foo :foo :foo :foo ! :foo :foo :foo)! return
47. (gen/sample (gen/choose 5 10))! (7 8 9 6 6 5 8 9 10 10)! choose
48. (gen/sample (gen/elements ! [:clojure :haskell :erlang :python :ruby]))! ! (:clojure :haskell :haskell :clojure :clojure :python :c lojure :clojure :ruby :ruby)! elements
49. (gen/sample (gen/one-of [gen/boolean gen/byte]))! ! (119 true -24 120 -93 true false 27 false false)! one-of
50. (gen/sample (gen/frequency ! [[5 (gen/return :weekday)]! [2 (gen/return :weekend)]]))! ! (:weekday :weekday :weekend :weekday :weekday :weekend : weekend :weekend :weekday :weekday)! frequency
51. (gen/sample (gen/such-that #(not= 0 %) gen/int))! (-1 -2 -1 2 3 -5 -6 6 -5 -1)! ! ! ! ! such-that
52. (gen/sample (gen/fmap odd? gen/int))! (false true false true true false false false false false)! ! fmap
53. (gen/sample (gen/fmap odd? gen/int))! (false true false true true false false false false false)! ! apply odd? to every generated value
54. (gen/sample (gen/bind! (gen/not-empty (gen/vector gen/int))! #(gen/tuple (gen/return %) (gen/elements %))))! ! ! ! bind
55. ([[0] 0]! [[1 -2] 1]! [[2 2] 2]! [[-3 -3] -3]! [[4 -4] -4]! [[2 1] 2]! [[-6 -6 -5 -1] -5]! [[8 -5 6 -6 -6 -5 7 -7] 8]! [[-3 2 -5 -4] -5]! [[7 5] 5])! ! ! tuples of vector and chosen element
56. shrinking
57. remember the counterexample before….
58. [[:conj 109] [:conj -110] [:transient ]! [:disj -110] [:persistent!] [:conj -110]]!
59. [[:disj -108] [:disj -24] [:disj 89] [:transient] [:conj 12] [:disj -136] [:transient] [:conj -145] [:conj -8] [:persistent!] [:persistent!] [:conj 25] [:conj 142] [:conj 146] [:persistent!] [:transient] [:conj 95] [:persistent!] [:disj -86] [:transient] [:persistent!] [:transient] [:persistent!] [:conj -109] [:transient] [:transient] [:persistent!] [:disj -6] [:transient] [:persistent!] [:conj -50] [:persistent!] [:conj -166] [:transient] [:conj 136] [:persistent!] [:persistent!] [:transient] [:disj 155] [:persistent!] [:transient] [:transient] [:disj 143] [:persistent!] [:persistent!] [:transient] [:conj -10] [:transient] [:conj -59] [:conj -34] [:transient] [:transient] [:disj 111] [:transient] [:disj -33] [:transient] [:conj -96] [:transient] [:disj 93] [:disj -130] [:conj 88] [:disj 14] [:transient] [:disj 9] [:persistent!] [:disj 86] [:conj 109] [:conj -45] [:persistent!] [:persistent!] [:disj -141] [:conj -14] [:persistent!] [:persistent!] [:conj -104] [:conj -83] [:persistent!] [:persistent!] [:conj -73] [:conj 13] [:persistent!] [:persistent!] [:persistent!] [:disj -14] [:conj -129] [:disj -53] [:transient] [:conj -110] [:conj 60] [:conj 11] [:disj -128] [:disj -42] [:disj 18] [:disj 73] [:disj 33] [:transient] [:persistent!] [:persistent!] [:transient] [:persistent!] [:transient] [:disj -123] [:disj -141] [:conj -26] [:conj -92] [:conj -116] [:conj 101] [:disj -133] [:conj 58] [:transient] [:disj -110] [:persistent!] [:disj 105] [:conj 26] [:conj -110] [:persistent!] [:transient] [:persistent!] [:persistent!] [:disj -5] [:conj 117] [:transient] [:persistent!] [:disj 96] [:persistent!] [:disj 52] [:persistent!] [:disj -132] [:transient] [:transient] [:conj -108] [:disj 121] [:persistent!]]!
60. [[:conj 109] [:conj -110] [:transient ]! [:disj -110] [:persistent!] [:conj -110]]!
62. (defspec! first-element-is-min! 100 ;; num iterations! (prop/for-all! [a gen/int]! …))
63. github.com/cemerick/ double-check cljs compatible fork
64. concurrency testing
65. concurrent programs are hard to test
66. why are concurrent programs hard to test?
67. their execution is non- deterministic
68. let's make it deterministic
69. based on PULSE
70. for all thread interleavings…
71. (sched/shedule! (let [p (promise)]! (future (deliver p true))! (future (Thread/sleep 1000) (deliver p false))! (= true @p)))!
74. github.com/clojure/ test.check github.com/reiddraper/ clojure-transient-test