Hillel Wayne - Beyond Unit Tests: Taking Your Testing to the Next Level

You've used pytest and you've used mypy, but bugs are still slipping through your code. What's next? In this talk, we cover two simple but powerful tools for keeping your code problem-free. Property-based testing, provided by the [Hypothesis](https://hypothesis.readthedocs.io/en/latest/) library, lets you run hundreds of tests from a single template. Contracts, via [dpcontracts](https://github.com/deadpixi/contracts), make your program test itself. You'll learn how and why to use these tools and how to combine them with the rest of your testing suite.

https://us.pycon.org/2018/schedule/presentation/130/

Transcript

1. Beyond Unit Tests Hillel Wayne hillelwayne.com @hillelogram
2. Shoutouts to • PyCon Volunteers • Jay Parlar @hillelogram hillelwayne.com
3. Ingredients • Oats • Honey • Sugar • Butter • Chocolate • Coconut • Apricots • Pumpkin Seeds • Oranges • Raisins • Cranberries @hillelogram hillelwayne.com
4. Just tweet / email / hunt me down during the conf hillelwayne.com @hillelogram
5. This code is not Pythonic. hillelwayne.com @hillelogram
6. Unit Tests @hillelogram hillelwayne.com
7. def add(a: int, b: int) -> int: """Adds two numbers.""" ... @hillelogram hillelwayne.com
8. def test_works1(): assert add(1, 2) == 3 def test_works2(): assert add(0, 0) == 0 def test_works3(): assert add(1, 0) == 1 def test_works4(): assert add(5, -2) == 3 @hillelogram hillelwayne.com
9. WRITE UNIT TESTS hillelwayne.com @hillelogram
10. WRITE UNIT TESTS hillelwayne.com @hillelogram
11. @hillelogram This Photo by Unknown Author is licensed under CC BY-SA Example Tests "Automanual" Tests hillelwayne.com
12. hillelwayne.com
13. def clamp(input, min, max) @hillelogram hillelwayne.com
14. def slope(p1: point, p2: point) @hillelogram This Photo by Unknown Author is licensed under CC BY-SA hillelwayne.com
15. def leftpad(c: char, i: int, s: str) @hillelogram hillelwayne.com
16. Property-Based Testing @hillelogram "Invariant" Testing hillelwayne.com
17. Hypothesis https://hypothesis.works hillelwayne.com @hillelogram
18. @hillelogram hillelwayne.com
19. @given(integers()) def test_adds_zero(a): assert add(a, 0) == a @hillelogram hillelwayne.com
20. @hillelogram hillelwayne.com
21. @hillelogram hillelwayne.com
22. @given(integers(), integers()) def test_commutative(a, b): assert add(a, b) == add(b, a) @hillelogram hillelwayne.com
23. @hillelogram hillelwayne.com
24. @hillelogram a = 0, b = 1 @given(integers(), integers()) def test_commutative(a, b): > assert add(a, b) == add(b, a) E assert 0 == 1 E + where 0 = add(0, 1) E + and 1 = add(1, 0) Falsifying example: test_commutative(a=0, b=1) hillelwayne.com
25. def add(a: int, b: int) -> int: """Adds two numbers.""" if '2' in str(b): return 3 return a @hillelogram hillelwayne.com
26. What about regressions? @hillelogram hillelwayne.com
27. @given(integers(), integers()) @example(1, 0) def test_commutative(a, b): assert add(a, b) == add(b, a) @hillelogram hillelwayne.com
28. @hillelogram hillelwayne.com
29. "What's a good property?" @hillelogram hillelwayne.com
30. For a given list, return the maximum possible product of two elements in it. max_product([1, 5, 0, 2]) == 10 @hillelogram hillelwayne.com
31. def max_product(l: List[int]) -> int: if len(l) < 2: raise TabError b1, b2 = sorted(l, reverse=True)[0:2] return b1*b2 @hillelogram hillelwayne.com
32. def test1(): assert max_product([1, 2, 5]) == 10 def test2(): assert max_product([5, 1, 2, 3]) == 15 def test3(): assert max_product([5, 1, 2, 6, 3, 1]) == 30 def test4(): with raises(TabError): max_product([1]) @hillelogram hillelwayne.com
33. @given(lists(integers())) def test_fuzz(li): max_product(li) @hillelogram hillelwayne.com
34. @given(lists(integers(), min_size=2)) def test_fuzz(li): max_product(li) @hillelogram hillelwayne.com
35. [ 1 , 5 , 2 , 8 , 3 , 0 , 1 ] hillelwayne.com @hillelogram
36. @given(lists(integers(), min_size=2)) def test_always_better(li): assert max_product(li) >= li[0]*li[1] @hillelogram hillelwayne.com
37. @hillelogram li = [-1, -1, 0] @given(lists(integers(), min_size=2)) def test_always_better(li): > assert max_product(li) >= li[0]*li[1] E assert 0 >= (-1 * -1) E + where 0 = max_product([-1, -1, 0]) Falsifying example: test_always_better(li=[-1, -1, 0]) hillelwayne.com
38. Integration Tests @hillelogram hillelwayne.com
39. @hillelogram This Photo by Unknown Author is licensed under CC BY-SA hillelwayne.com
40. @hillelogram hillelwayne.com
41. @hillelogram hillelwayne.com
43. Mypy @hillelogram hillelwayne.com
44. def add(x: int, y: int) -> int: @hillelogram hillelwayne.com
45. def add(x: int, y: int) -> int: return str(x + y) "error: Incompatible return value type (got "str", expected "int")" @hillelogram hillelwayne.com
46. def add(x: int, y: int) -> int: return str(x + y) "error: Incompatible return value type (got "str", expected "int")" @hillelogram hillelwayne.com
47. def add(x: int, y: int) -> int: return x + y def g(): add(1, "2") @hillelogram hillelwayne.com
48. "Types will save us!" hillelwayne.com @hillelogram
49. def add(x: int, y: int) -> int: return 0 @hillelogram hillelwayne.com
50. def add(x, y): #requires assert isinstance(x, int) assert isinstance(y, int) z = x + y #ensures assert isinstance(z, int) return z @hillelogram hillelwayne.com
51. For a given list, return all-but-the first element in it. tail([9, 2, 1, "a"]) == [2, 1, "a"] @hillelogram hillelwayne.com
52. def tail(l: List[Any]) -> List[Any]: return [] @hillelogram hillelwayne.com
53. @hillelogram >>> tail([1,2,3]) [] hillelwayne.com
54. tail([9, 2, 1, "a"]) == [2, 1, "a"] [9] + [2, 1, "a"] == [9, 2, 1, "a"] @hillelogram hillelwayne.com
55. def tail(l: List[Any]) -> List[Any]: output = [] assert [l[0]] + output == l return output @hillelogram hillelwayne.com
56. @hillelogram >>> tail([1,2,3]) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in tail AssertionError: Postcondition >>> hillelwayne.com
57. def tail(l: List[Any]) -> List[Any]: output = l[1:] assert [l[0]] + output == l return output @hillelogram hillelwayne.com
58. @hillelogram >>> tail([1,2,3]) [2, 3] >>> tail([]) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in tail IndexError: list index out of range >>> hillelwayne.com
59. def tail(l: List[Any]) -> List[Any]: assert len(l) > 0 output = l[1:] assert [l[0]] + output == l return output @hillelogram hillelwayne.com
60. @hillelogram >>> tail([]) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 2, in tail AssertionError: Precondition >>> hillelwayne.com
61. Design By Contract hillelwayne.com @hillelogram
62. "Contracts" https://github.com/deadpixi/contracts hillelwayne.com @hillelogram
63. @require("l must not be empty", lambda args: len(args.l) > 0) def tail(l: List[Any]) -> List[Any]: assert len(l) > 0 output = l[1:] assert [l[0]] + output == l return output @hillelogram hillelwayne.com
64. @hillelogram @require("l must not be empty", lambda args: len(args.l) > 0) @ensure("result is tail of list", lambda args, result: [args.l[0]] + result == args.l) def tail(l: List[Any]) -> List[Any]: output = l[1:] assert [l[0]] + output == l return output hillelwayne.com
65. @hillelogram @require("l must not be empty", lambda a: len(a.l) > 0) @ensure("result is tail of list", lambda a, r: [a.l[0]] + r == a.l) def tail(l: List[Any]) -> List[Any]: return l[1:] hillelwayne.com
66. What’s our property test? @given(lists(integers(), 1)) def test_tail(l): tail(l) @hillelogram hillelwayne.com
67. "Full Specification" hillelwayne.com @hillelogram
68. def is_max_prod(a, r): pairs = combinations(a.l, 2) return all((r >= x * y for x, y in pairs)) @hillelogram hillelwayne.com
69. @require("2+ elements", lambda a: len(a.l) > 1) @ensure("is the max prod", is_max_prod) def max_product(l: List[int]) -> int: # efficient version @hillelogram hillelwayne.com
70. @given(lists(integers(), min_size=2)) def test_max_product(li): max_product(li) @hillelogram hillelwayne.com
71. Contracts + Property Tests = Integration Tests @hillelogram hillelwayne.com
72. @hillelogram hillelwayne.com
73. @hillelogram hillelwayne.com
74. @hillelogram hillelwayne.com
75. @hillelogram hillelwayne.com
76. @hillelogram hillelwayne.com
77. Can we go further? @hillelogram hillelwayne.com
78. Class Invariants @invariant("No overdrafts", lambda self: self.account >= 0) class BankAccount: def withdraw(self, val): ... def deposit(self, val): ... @hillelogram hillelwayne.com
79. Stateful Testing class NumberModifier(RuleBasedStateMachine): num = 0 @rule(x=integers()) def add_num(x): self.num += x @precondition(lambda self: self.num != 0) @rule() def reciprocal(self): self.num = 1 / self.num @hillelogram hillelwayne.com
80. Crazier Stuff • Test Generators (Clojure) • Contract Inheritance (Eiffel) • Global Contracts (Ada) • Static verification (Dafny) • ABC metatyping (Python???) @hillelogram hillelwayne.com
81. Conclusion @hillelogram Unit Tests => Property Tests Mypy => Contracts Integration Tests => Property Tests + Contracts hillelwayne.com
82. Hypothesis: http://hypothesis.works/ Contracts: https://github.com/deadpixi/contracts @hillelogram hillelwayne.com
83. WRITE UNIT TESTS hillelwayne.com @hillelogram
84. WRITE UNIT TESTS hillelwayne.com @hillelogram
85. WRITE UNIT TESTS hillelwayne.com @hillelogram
86. Formal Methods hillelwayne.com @hillelogram
87. Come to the formal methods open space! Saturday 2 PM Room 9 hillelwayne.com @hillelogram
88. Hillel Wayne hillelwayne.com @hillelogram