How to test proper{l,t}y

Writing unit tests is pretty much established practice and in addition to that, property testing has caught up on popularity. Most functional languages have one, sometimes even many implementations. But "property testing" has a lot of aspects: randomized or exhaustive, minimization and generalization of counter examples, custom generators and filters, to name a few. Very often, property tests don't exploit all the features of the framework.

In this talk, I'll give an overview of the state of the art of property testing in Scala and Haskell and show some common use cases, techniques and pitfalls.

Transcript

1. How to test proper{t,l}y Lars Hupel May 2nd, 2016
2. Agenda 1 Basic property testing 2 Generating things 3 Polymorphic properties 4 Outlook 2
3. Property testing ▶ simple idea: specify a property with unknowns, let the framework insert values ▶ basic features: ▶ automatic and custom generators ▶ filtering of inputs ▶ shrinking of counterexamples ▶ classification of inputs ▶ input and property labelling 3
4. Framework ecosystem ▶ Haskell ▶ QuickCheck ▶ SmallCheck ▶ Scala ▶ ScalaCheck ▶ scalaprops ▶ Java ▶ Javaslang ▶ Rust ▶ Quickcheck ▶ Erlang ▶ QuickCheck ▶ triq ▶ Python ▶ pytest-quickcheck ▶ Hypothesis ▶ Clojure ▶ simple-check ▶ test.check ▶ your favourite language ... 4
6. Agenda 1 Basic property testing 2 Generating things 3 Polymorphic properties 4 Outlook 6
7. Generators How does ScalaCheck come up with values? 7
8. Generators How does ScalaCheck come up with values? Gen[R] RNG Params Prop seed size value: R 7
9. Generating functions def getItemByName(string: String): Future[Int] = ??? ▶ assume: this function makes an expensive database access bad for testing ▶ we want to generate a lookup function randomly ▶ ScalaCheck can generate functions 8
10. Theory Let’s count! ▶ assume: there are n different Ts ▶ assume: there are m different Rs ▶ pop quiz: how many T=>Rs are there? 9
11. Theory Let’s count! ▶ assume: there are n different Ts ▶ assume: there are m different Rs ▶ pop quiz: how many T=>Rs are there? answer: mn 9
12. Strategies How can we generate functions? precompute a table impractical make up values on the spot side-effecting constant functions not good enough 10
13. Generating functions: ScalaCheck ≤ 1.12 def arbFn[T, R](a: Arbitrary[R]): Arbitrary[T => R] = Arbitrary( for (r <- a.arbitrary) yield (t: T) => r ) Pick a single value, produce a constant function 11
14. Generating functions: ScalaCheck ≥ 1.13 ▶ function outputs must only depend on its inputs. ▶ hack: use inputs to perturb random generator 12
15. Generating functions: ScalaCheck ≥ 1.13 ▶ function outputs must only depend on its inputs. ▶ hack: use inputs to perturb random generator RNG Gen[R] Prop seed value: R 12
16. Generating functions: ScalaCheck ≥ 1.13 ▶ function outputs must only depend on its inputs. ▶ hack: use inputs to perturb random generator RNG Gen[R] Prop value: R Cogen[T] seed perturbed seed 12
17. Generating functions: ScalaCheck ≥ 1.13 ▶ function outputs must only depend on its inputs. ▶ hack: use inputs to perturb random generator trait Cogen[T] { def perturb(seed: Seed, t: T): Seed } 12
18. Generating functions: ScalaCheck ≥ 1.13 ▶ function outputs must only depend on its inputs. ▶ hack: use inputs to perturb random generator trait Cogen[T] { def perturb(seed: Seed, t: T): Seed def value[A](seed: Seed, t: T, g: Gen[A]) = g.value(perturb(seed, t)) } 12
20. Randomly generate all the things? Advantages ▶ likely to find input you didn’t think about ▶ can easily cover wide variety of input 14
21. Randomly generate all the things? Advantages ▶ likely to find input you didn’t think about ▶ can easily cover wide variety of input Disadvantages ▶ inherently non-reproducible (random failures) ▶ often high discard ratio ▶ existential properties are difficult 14
22. Enumerating things trait Enumerable[T] { def enumerate(depth: BigInt): Stream[T] } ... where depth means number of constructors 15
23. SmallCheck Haskell implementation ▶ supports quantifiers: ∃, ∃1 , ∀ ▶ supports implication 16
24. SmallCheck Haskell implementation ▶ supports quantifiers: ∃, ∃1 , ∀ ▶ supports implication ▶ supports quantifiers nested in implications 16
26. Agenda 1 Basic property testing 2 Generating things 3 Polymorphic properties 4 Outlook 18
27. What if my properties are polymorphic? def reverse[A](xs: List[A]): List[A] ▶ How should we instantiate the types? ▶ Should we come up with fresh types? 19
28. What if my properties are polymorphic? def reverse[A](xs: List[A]): List[A] ▶ How should we instantiate the types? ▶ Should we come up with fresh types? 19
29. What if types were first-class? Existential types to the rescue! sealed trait Type { type T val name: String def forAll(prop: T => Prop): Prop } 20
30. What if types were first-class? Existential types to the rescue! Prop.forAll { typ: SmallType => Prop.forAll { xs: List[typ.T] => reverse(xs) == xs } } 20
31. What if types were first-class? Existential types to the rescue! Prop.forAll { typ: SmallType => Prop.forAll { xs: List[typ.T] => reverse(xs) == xs } } ! Test.reverse: Falsified after 157 passed tests. > ARG_0: Type[small.Two] > ARG_1: List(”B”, ”A”) 20
33. Agenda 1 Basic property testing 2 Generating things 3 Polymorphic properties 4 Outlook 22
34. Minimization on steroids (xs zip ys).length == xs.length Easy to find a small counterexample: val xs = 1 :: Nil val ys = Nil ... but: tells us nothing about the structure 23
35. Minimization on steroids (xs zip ys).length == xs.length Mathematicians would write: val xs = 1 :: xs0 // for arbitrary xs0 val ys = Nil Tells us about classes of failures! 23
36. Signature exploration ▶ writing properties by hand is tedious ▶ Why not automate? ▶ Idea: give a signature, try to compose well-typed terms, run QuickCheck 24
37. The End @larsr h larsrh.github.io typelevel.org/blog