Property-based testing for the masses

Talk given at corkdev.io.

(See https://www.meetup.com/corkdev-io/events/237848006/)

Transcript

1. Property-based testing for the masses Julien Cretel @_jubobs_ Jubobs/corkdev-feb-2017
2. Good unit tests (GUTs) matter
3. Good unit tests (GUTs) matter
4. Most unit tests are example-based
5. The Seven Deadly Sins of Example-Based Testing
6. Lust • reuse of test data • coupling between test cases • hurts evolvability
7. Gluttony • too many test cases for the same behaviour • bloated test suite • tedious to write, read, maintain
8. Greed • Slave to code-coverage metrics
9. Sloth • Test cases are too few. • Tests lack imagination, miss edge cases.
10. Example: Partitioner
11. Sloth • Tests are hard to understand.
12. Wrath • Scenarios cannot lay claim to completeness. • Leaves reader to infer actual general behaviour from tests
13. Envy • The size of the test suite is fixed. • A bigger test suite requires writing more test cases.
14. Pride • Frivolous, noisy data • Fun, but distracting…
15. Pride
16. We want to make sure the tests pull their weight by making them expressive, so that we can tell what's important when we read them and when they fail, and by making sure they don’t become a maintenance drag themselves. Nat Pryce & Steve Freeman (GOOS book, p227)
17. Property-based testing to the rescue! Don’t write tests; generate them!
18. What is a property?
19. Generators
20. Shrinkers
21. Libraries QuickCheck QuickCheck junit-quickcheck ScalaCheck FsCheck Hypothesis elm-check test-check JSVerify
22. Example: Partitioner (revisited)
23. Lust • reuse of test data • coupling between test cases • hurts evolvability • generators provide test data
24. Gluttony • too many test cases for the same behaviour • bloated test suite • tedious to write, read, maintain • one behaviour = one property
25. Greed • Slave to code-coverage metrics • "For all input values allowed by the compiler, does the implementation fulfill the API’s contract?”
26. Sloth • Test cases are too few. • Tests lack imagination, miss edge cases. • Leave the tedium to the library. • More readable tests.
27. Wrath • Scenarios cannot lay claim to completeness. • Leaves reader to infer actual general behaviour from tests • Higher level of abstraction, no ambiguity
28. Envy • The size of the test suite is fixed. • A bigger test suite requires writing more test cases. • The size of the test suite can be tuned.
29. Pride • Frivolous, noisy data • Fun, but distracting… • High power-to-weight ratio • The truth, all the truth, nothing but the truth.
30. Example: Recently Used List
31. Common objections to property-based testing
32. “Property-test all the things?”
33. “I can’t think of properties about my code”
34. “Increased risk of bugs in testing code”
35. The whole point is that [production code and testing code] double-check on each other. You have a problem when something goes red because they are mismatched. The basic, fundamental thing is that it’s a double-check mechanism. Martin Fowler (https://www.youtube.com/watch? v=dGtasFJnUxI&t=23m)
36. “Too complicated!”
37. In conclusion, property-based tests… • raise the level of abstraction • increase your confidence in the system under test • reduce the maintenance burden of your tests • allow you to spend more time on other valuable activities Questions?