How to test proper{l,t}y

Writing unit tests is pretty much established practice and in addition to that, property testing has caught up on popularity. Most functional languages have one, sometimes even many implementations. But "property testing" has a lot of aspects: randomized or exhaustive, minimization and generalization of counter examples, custom generators and filters, to name a few. Very often, property tests don't exploit all the features of the framework. In this talk, he'll give an overview of the state of the art of property testing in Scala and show some common use cases, techniques and pitfalls.

Transcript

1. How to test proper{t,l}y Lars Hupel September 27th, 2016
2. Agenda 1 Basic property testing 2 Generating things 3 Polymorphic properties 2
3. Property testing ▶ simple idea: specify a property with unknowns, let the framework insert values ▶ basic features: ▶ automatic and custom generators ▶ filtering of inputs ▶ shrinking of counterexamples ▶ classification of inputs ▶ input and property labelling 3
4. Goal forAll { (x: Money, y: Money) => x + y == y + x } 4
5. Goal forAll { (x: Money, y: Money) => x + y == y + x } implicit val precision = Precision(.0001) forAll { (loan: Money, rate: Interest) => rate > 0 ==> (loan * rate) / rate ~= loan } 4
6. Framework ecosystem ▶ Haskell ▶ QuickCheck ▶ SmallCheck ▶ Scala ▶ ScalaCheck ▶ scalaprops ▶ Java ▶ Javaslang ▶ Rust ▶ Quickcheck ▶ Erlang ▶ QuickCheck ▶ triq ▶ Python ▶ pytest-quickcheck ▶ Hypothesis ▶ Clojure ▶ simple-check ▶ test.check ▶ your favourite language ... 5
7. Agenda 1 Basic property testing 2 Generating things 3 Polymorphic properties 6
8. Generators How does ScalaCheck come up with values? 7
9. Generators How does ScalaCheck come up with values? Gen[R] RNG Params Prop seed size value: R 7
11. Generating functions def getItemByName(string: String): Future[Int] = ??? ▶ assume: this function makes an expensive database access bad for testing ▶ we want to generate a lookup function randomly ▶ ScalaCheck can generate functions 9
12. Theory Let’s count! ▶ assume: there are n different Ts ▶ assume: there are m different Rs ▶ pop quiz: how many T=>Rs are there? 10
13. Theory Let’s count! ▶ assume: there are n different Ts ▶ assume: there are m different Rs ▶ pop quiz: how many T=>Rs are there? answer: mn 10
14. Strategies How can we generate functions? precompute a table impractical make up values on the spot side-effecting constant functions not good enough 11
15. Generating functions: ScalaCheck ≤ 1.12 def arbFn[T, R](a: Arbitrary[R]): Arbitrary[T => R] = Arbitrary( for (r <- a.arbitrary) yield (t: T) => r ) Pick a single value, produce a constant function 12
16. Generating functions: ScalaCheck ≥ 1.13 ▶ function outputs must only depend on its inputs. ▶ hack: use inputs to perturb random generator 13
17. Generating functions: ScalaCheck ≥ 1.13 ▶ function outputs must only depend on its inputs. ▶ hack: use inputs to perturb random generator RNG Gen[R] Prop seed value: R 13
18. Generating functions: ScalaCheck ≥ 1.13 ▶ function outputs must only depend on its inputs. ▶ hack: use inputs to perturb random generator RNG Gen[R] Prop value: R Cogen[T] seed perturbed seed 13
19. Generating functions: ScalaCheck ≥ 1.13 ▶ function outputs must only depend on its inputs. ▶ hack: use inputs to perturb random generator trait Cogen[T] { def perturb(seed: Seed, t: T): Seed } 13
20. Generating functions: ScalaCheck ≥ 1.13 ▶ function outputs must only depend on its inputs. ▶ hack: use inputs to perturb random generator trait Cogen[T] { def perturb(seed: Seed, t: T): Seed def value[A](seed: Seed, t: T, g: Gen[A]) = g.value(perturb(seed, t)) } 13
22. Randomly generate all the things? Advantages ▶ likely to find input you didn’t think about ▶ can easily cover wide variety of input 15
23. Randomly generate all the things? Advantages ▶ likely to find input you didn’t think about ▶ can easily cover wide variety of input Disadvantages ▶ inherently non-reproducible (random failures) ▶ often high discard ratio ▶ existential properties are difficult 15
24. Enumerating things trait Enumerable[T] { def enumerate(depth: BigInt): Stream[T] } ... where depth means number of constructors 16
26. Agenda 1 Basic property testing 2 Generating things 3 Polymorphic properties 18
27. What if my properties are polymorphic? def reverse[A](xs: List[A]): List[A] ▶ How should we instantiate the types? ▶ Should we come up with fresh types? 19
28. What if my properties are polymorphic? def reverse[A](xs: List[A]): List[A] ▶ How should we instantiate the types? ▶ Should we come up with fresh types? 19
29. What if types were first-class? Existential types to the rescue! sealed trait Type { type T val name: String def forAll(prop: T => Prop): Prop } 20
30. What if types were first-class? Existential types to the rescue! Prop.forAll { typ: SmallType => Prop.forAll { xs: List[typ.T] => reverse(xs) == xs } } 20
31. What if types were first-class? Existential types to the rescue! Prop.forAll { typ: SmallType => Prop.forAll { xs: List[typ.T] => reverse(xs) == xs } } ! Test.reverse: Falsified after 157 passed tests. > ARG_0: Type[small.Two] > ARG_1: List(”B”, ”A”) 20
33. Q & A @larsr h larsrh.github.io typelevel.org/blog