Property-based testing

Talk given at Bordeaux's Functional-Programming meetup group.

(See https://www.meetup.com/bdx-fp/events/233327084/)

Transcript

1. PROPERTY-BASED TESTING
2. A little bit about me…
5. Let’s talk about testing…
9. Most tests are example-based
12. The Seven Deadly Sins of Example-Based Testing
13. Lust
14. Lust • reuse of test data • coupling between test cases • hurts evolvability
15. Gluttony
16. Gluttony • too many test cases for the same behaviour • bloated test suite • tedious to write, read, maintain
17. Greed
18. Greed • slave to code-coverage metrics
19. Greed
20. Greed
21. Greed You don't need static type checking if you have 100% unit test coverage.
22. Greed • slave to code-coverage metrics • Low code coverage indicates inadequate testing, but high code coverage guarantees nothing.
23. Sloth
24. Sloth • The lights are on, but no one’s home. • Test cases are too few. • Test cases lack imagination, miss edge cases.
25. Wrath
26. Wrath • Tests that written with their role as executable specifications in mind • However, only the concrete scenario that follows is executable! • Scenarios cannot lay claim to completeness. • Leaves reader to infer actual general behaviour from tests • Sometimes leads to wrong assumptions about behaviour! • Lack empathy and expressiveness
27. Envy
28. Envy • The size of the test suite is fixed. • A bigger test suite requires writing more test cases.
29. Pride
30. Pride • Frivolous / noisy data • Fun at the beginning, but distracting…
31. Pride
32. Property-based testing to the rescue!
33. Origin Don’t write tests; generate them!
34. Origin
35. What is a property?
37. Example: sqrt
38. Generators
39. Generators
40. Shrinkers
41. Example: recently used list
42. Recently used list
43. What’s in it for me?
44. Lust
45. Lust • generators are the only source of test data
46. Gluttony
47. Gluttony • one behaviour = one property
48. Greed
49. Greed • Less focus on code-coverage metrics • Rich properties and generators will give you high code- coverage. • "For all input values allowed by the compiler, does the implementation fulfill the API’s contract?”
50. Sloth
51. Sloth • Leave the tedium to the library. • Less typing; more thinking. • Reason about the actual behaviour of your code… • … at a higher level of abstraction.
52. Wrath
53. Wrath • Reason about the actual behaviour of your code… • … at a higher level of abstraction. • Tell the reader of your code.
54. Envy
55. Envy • Tunable number of test cases generated!
56. Pride
57. Pride • High power-to-weight ratio • The truth, all the truth, nothing but the truth.
58. Property-test all the things?
59. Common objections to property-based testing
60. “Too complicated!”
61. “I can’t think of properties about my code”
62. “Generators are code!”
63. Would you like to know more?
64. Libraries • Haskell: QuickCheck • Erlang: QuickCheck • Java: junit-quickcheck • Scala: ScalaCheck • Python: HypothesisWorks/hypothesis-python • etc…
65. Talks • Practical Property-Based Testing - Charles O’Farrell • Testing the Hard Stuff and Staying Sane - John Hughes • Property-Based Testing, enfin pour tout le monde - Cyrille Martraire & Romeu Moura
66. Blogs • Nat Pryce: http://www.natpryce.com/ • Jessica Kerr: http://blog.jessitron.com/
67. Books