Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to test proper{l,t}y

How to test proper{l,t}y

Writing unit tests is pretty much established practice and in addition to that, property testing has caught up on popularity. Most functional languages have one, sometimes even many implementations. But "property testing" has a lot of aspects: randomized or exhaustive, minimization and generalization of counter examples, custom generators and filters, to name a few. Very often, property tests don't exploit all the features of the framework.

In this talk, I'll give an overview of the state of the art of property testing in Scala and Haskell and show some common use cases, techniques and pitfalls.

A1216674d5c9747bcdcc716872439137?s=128

Lars Hupel

May 02, 2016
Tweet

Transcript

  1. How to test proper{t,l}y Lars Hupel May 2nd, 2016

  2. Agenda 1 Basic property testing 2 Generating things 3 Polymorphic

    properties 4 Outlook 2
  3. Property testing ▶ simple idea: specify a property with unknowns,

    let the framework insert values ▶ basic features: ▶ automatic and custom generators ▶ filtering of inputs ▶ shrinking of counterexamples ▶ classification of inputs ▶ input and property labelling 3
  4. Framework ecosystem ▶ Haskell ▶ QuickCheck ▶ SmallCheck ▶ Scala

    ▶ ScalaCheck ▶ scalaprops ▶ Java ▶ Javaslang ▶ Rust ▶ Quickcheck ▶ Erlang ▶ QuickCheck ▶ triq ▶ Python ▶ pytest-quickcheck ▶ Hypothesis ▶ Clojure ▶ simple-check ▶ test.check ▶ your favourite language ... 4
  5. None
  6. Agenda 1 Basic property testing 2 Generating things 3 Polymorphic

    properties 4 Outlook 6
  7. Generators How does ScalaCheck come up with values? 7

  8. Generators How does ScalaCheck come up with values? Gen[R] RNG

    Params Prop seed size value: R 7
  9. Generating functions def getItemByName(string: String): Future[Int] = ??? ▶ assume:

    this function makes an expensive database access bad for testing ▶ we want to generate a lookup function randomly ▶ ScalaCheck can generate functions 8
  10. Theory Let’s count! ▶ assume: there are n different Ts

    ▶ assume: there are m different Rs ▶ pop quiz: how many T=>Rs are there? 9
  11. Theory Let’s count! ▶ assume: there are n different Ts

    ▶ assume: there are m different Rs ▶ pop quiz: how many T=>Rs are there? answer: mn 9
  12. Strategies How can we generate functions? precompute a table impractical

    make up values on the spot side-effecting constant functions not good enough 10
  13. Generating functions: ScalaCheck ≤ 1.12 def arbFn[T, R](a: Arbitrary[R]): Arbitrary[T

    => R] = Arbitrary( for (r <- a.arbitrary) yield (t: T) => r ) Pick a single value, produce a constant function 11
  14. Generating functions: ScalaCheck ≥ 1.13 ▶ function outputs must only

    depend on its inputs. ▶ hack: use inputs to perturb random generator 12
  15. Generating functions: ScalaCheck ≥ 1.13 ▶ function outputs must only

    depend on its inputs. ▶ hack: use inputs to perturb random generator RNG Gen[R] Prop seed value: R 12
  16. Generating functions: ScalaCheck ≥ 1.13 ▶ function outputs must only

    depend on its inputs. ▶ hack: use inputs to perturb random generator RNG Gen[R] Prop value: R Cogen[T] seed perturbed seed 12
  17. Generating functions: ScalaCheck ≥ 1.13 ▶ function outputs must only

    depend on its inputs. ▶ hack: use inputs to perturb random generator trait Cogen[T] { def perturb(seed: Seed, t: T): Seed } 12
  18. Generating functions: ScalaCheck ≥ 1.13 ▶ function outputs must only

    depend on its inputs. ▶ hack: use inputs to perturb random generator trait Cogen[T] { def perturb(seed: Seed, t: T): Seed def value[A](seed: Seed, t: T, g: Gen[A]) = g.value(perturb(seed, t)) } 12
  19. None
  20. Randomly generate all the things? Advantages ▶ likely to find

    input you didn’t think about ▶ can easily cover wide variety of input 14
  21. Randomly generate all the things? Advantages ▶ likely to find

    input you didn’t think about ▶ can easily cover wide variety of input Disadvantages ▶ inherently non-reproducible (random failures) ▶ often high discard ratio ▶ existential properties are difficult 14
  22. Enumerating things trait Enumerable[T] { def enumerate(depth: BigInt): Stream[T] }

    ... where depth means number of constructors 15
  23. SmallCheck Haskell implementation ▶ supports quantifiers: ∃, ∃1 , ∀

    ▶ supports implication 16
  24. SmallCheck Haskell implementation ▶ supports quantifiers: ∃, ∃1 , ∀

    ▶ supports implication ▶ supports quantifiers nested in implications 16
  25. None
  26. Agenda 1 Basic property testing 2 Generating things 3 Polymorphic

    properties 4 Outlook 18
  27. What if my properties are polymorphic? def reverse[A](xs: List[A]): List[A]

    ▶ How should we instantiate the types? ▶ Should we come up with fresh types? 19
  28. What if my properties are polymorphic? def reverse[A](xs: List[A]): List[A]

    ▶ How should we instantiate the types? ▶ Should we come up with fresh types? 19
  29. What if types were first-class? Existential types to the rescue!

    sealed trait Type { type T val name: String def forAll(prop: T => Prop): Prop } 20
  30. What if types were first-class? Existential types to the rescue!

    Prop.forAll { typ: SmallType => Prop.forAll { xs: List[typ.T] => reverse(xs) == xs } } 20
  31. What if types were first-class? Existential types to the rescue!

    Prop.forAll { typ: SmallType => Prop.forAll { xs: List[typ.T] => reverse(xs) == xs } } ! Test.reverse: Falsified after 157 passed tests. > ARG_0: Type[small.Two] > ARG_1: List(”B”, ”A”) 20
  32. None
  33. Agenda 1 Basic property testing 2 Generating things 3 Polymorphic

    properties 4 Outlook 22
  34. Minimization on steroids (xs zip ys).length == xs.length Easy to

    find a small counterexample: val xs = 1 :: Nil val ys = Nil ... but: tells us nothing about the structure 23
  35. Minimization on steroids (xs zip ys).length == xs.length Mathematicians would

    write: val xs = 1 :: xs0 // for arbitrary xs0 val ys = Nil Tells us about classes of failures! 23
  36. Signature exploration ▶ writing properties by hand is tedious ▶

    Why not automate? ▶ Idea: give a signature, try to compose well-typed terms, run QuickCheck 24
  37. The End @larsr h larsrh.github.io typelevel.org/blog