Tweet
Tweet

Slide 1

Slide 1 text

Beyond Unit Tests Hillel Wayne hillelwayne.com @hillelogram

Slide 2

Slide 2 text

Shoutouts to • PyCon Volunteers • Jay Parlar @hillelogram hillelwayne.com

Slide 3

Slide 3 text

Ingredients • Oats • Honey • Sugar • Butter • Chocolate • Coconut • Apricots • Pumpkin Seeds • Oranges • Raisins • Cranberries @hillelogram hillelwayne.com

Slide 4

Slide 4 text

Just tweet / email / hunt me down during the conf hillelwayne.com @hillelogram

Slide 5

Slide 5 text

This code is not Pythonic. hillelwayne.com @hillelogram

Slide 6

Slide 6 text

Unit Tests @hillelogram hillelwayne.com

Slide 7

Slide 7 text

def add(a: int, b: int) -> int: """Adds two numbers.""" ... @hillelogram hillelwayne.com

Slide 8

Slide 8 text

def test_works1(): assert add(1, 2) == 3 def test_works2(): assert add(0, 0) == 0 def test_works3(): assert add(1, 0) == 1 def test_works4(): assert add(5, -2) == 3 @hillelogram hillelwayne.com

Slide 9

Slide 9 text

WRITE UNIT TESTS hillelwayne.com @hillelogram

Slide 10

Slide 10 text

WRITE UNIT TESTS hillelwayne.com @hillelogram

Slide 11

Slide 11 text

@hillelogram This Photo by Unknown Author is licensed under CC BY-SA Example Tests "Automanual" Tests hillelwayne.com

Slide 12

Slide 12 text

hillelwayne.com

Slide 13

Slide 13 text

def clamp(input, min, max) @hillelogram hillelwayne.com

Slide 14

Slide 14 text

def slope(p1: point, p2: point) @hillelogram This Photo by Unknown Author is licensed under CC BY-SA hillelwayne.com

Slide 15

Slide 15 text

def leftpad(c: char, i: int, s: str) @hillelogram hillelwayne.com

Slide 16

Slide 16 text

Property-Based Testing @hillelogram "Invariant" Testing hillelwayne.com

Slide 17

Slide 17 text

Hypothesis https://hypothesis.works hillelwayne.com @hillelogram

Slide 18

Slide 18 text

@hillelogram hillelwayne.com

Slide 19

Slide 19 text

@given(integers()) def test_adds_zero(a): assert add(a, 0) == a @hillelogram hillelwayne.com

Slide 20

Slide 20 text

@hillelogram hillelwayne.com

Slide 21

Slide 21 text

@hillelogram hillelwayne.com

Slide 22

Slide 22 text

@given(integers(), integers()) def test_commutative(a, b): assert add(a, b) == add(b, a) @hillelogram hillelwayne.com

Slide 23

Slide 23 text

@hillelogram hillelwayne.com

Slide 24

Slide 24 text

@hillelogram a = 0, b = 1 @given(integers(), integers()) def test_commutative(a, b): > assert add(a, b) == add(b, a) E assert 0 == 1 E + where 0 = add(0, 1) E + and 1 = add(1, 0) Falsifying example: test_commutative(a=0, b=1) hillelwayne.com

Slide 25

Slide 25 text

def add(a: int, b: int) -> int: """Adds two numbers.""" if '2' in str(b): return 3 return a @hillelogram hillelwayne.com

Slide 26

Slide 26 text

What about regressions? @hillelogram hillelwayne.com

Slide 27

Slide 27 text

@given(integers(), integers()) @example(1, 0) def test_commutative(a, b): assert add(a, b) == add(b, a) @hillelogram hillelwayne.com

Slide 28

Slide 28 text

@hillelogram hillelwayne.com

Slide 29

Slide 29 text

"What's a good property?" @hillelogram hillelwayne.com

Slide 30

Slide 30 text

For a given list, return the maximum possible product of two elements in it. max_product([1, 5, 0, 2]) == 10 @hillelogram hillelwayne.com

Slide 31

Slide 31 text

def max_product(l: List[int]) -> int: if len(l) < 2: raise TabError b1, b2 = sorted(l, reverse=True)[0:2] return b1*b2 @hillelogram hillelwayne.com

Slide 32

Slide 32 text

def test1(): assert max_product([1, 2, 5]) == 10 def test2(): assert max_product([5, 1, 2, 3]) == 15 def test3(): assert max_product([5, 1, 2, 6, 3, 1]) == 30 def test4(): with raises(TabError): max_product([1]) @hillelogram hillelwayne.com

Slide 33

Slide 33 text

@given(lists(integers())) def test_fuzz(li): max_product(li) @hillelogram hillelwayne.com

Slide 34

Slide 34 text

@given(lists(integers(), min_size=2)) def test_fuzz(li): max_product(li) @hillelogram hillelwayne.com

Slide 35

Slide 35 text

[ 1 , 5 , 2 , 8 , 3 , 0 , 1 ] hillelwayne.com @hillelogram

Slide 36

Slide 36 text

@given(lists(integers(), min_size=2)) def test_always_better(li): assert max_product(li) >= li[0]*li[1] @hillelogram hillelwayne.com

Slide 37

Slide 37 text

@hillelogram li = [-1, -1, 0] @given(lists(integers(), min_size=2)) def test_always_better(li): > assert max_product(li) >= li[0]*li[1] E assert 0 >= (-1 * -1) E + where 0 = max_product([-1, -1, 0]) Falsifying example: test_always_better(li=[-1, -1, 0]) hillelwayne.com

Slide 38

Slide 38 text

Integration Tests @hillelogram hillelwayne.com

Slide 39

Slide 39 text

@hillelogram This Photo by Unknown Author is licensed under CC BY-SA hillelwayne.com

Slide 40

Slide 40 text

@hillelogram hillelwayne.com

Slide 41

Slide 41 text

@hillelogram hillelwayne.com

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

Mypy @hillelogram hillelwayne.com

Slide 44

Slide 44 text

def add(x: int, y: int) -> int: @hillelogram hillelwayne.com

Slide 45

Slide 45 text

def add(x: int, y: int) -> int: return str(x + y) "error: Incompatible return value type (got "str", expected "int")" @hillelogram hillelwayne.com

Slide 46

Slide 46 text

def add(x: int, y: int) -> int: return str(x + y) "error: Incompatible return value type (got "str", expected "int")" @hillelogram hillelwayne.com

Slide 47

Slide 47 text

def add(x: int, y: int) -> int: return x + y def g(): add(1, "2") @hillelogram hillelwayne.com

Slide 48

Slide 48 text

"Types will save us!" hillelwayne.com @hillelogram

Slide 49

Slide 49 text

def add(x: int, y: int) -> int: return 0 @hillelogram hillelwayne.com

Slide 50

Slide 50 text

def add(x, y): #requires assert isinstance(x, int) assert isinstance(y, int) z = x + y #ensures assert isinstance(z, int) return z @hillelogram hillelwayne.com

Slide 51

Slide 51 text

For a given list, return all-but-the first element in it. tail([9, 2, 1, "a"]) == [2, 1, "a"] @hillelogram hillelwayne.com

Slide 52

Slide 52 text

def tail(l: List[Any]) -> List[Any]: return [] @hillelogram hillelwayne.com

Slide 53

Slide 53 text

@hillelogram >>> tail([1,2,3]) [] hillelwayne.com

Slide 54

Slide 54 text

tail([9, 2, 1, "a"]) == [2, 1, "a"] [9] + [2, 1, "a"] == [9, 2, 1, "a"] @hillelogram hillelwayne.com

Slide 55

Slide 55 text

def tail(l: List[Any]) -> List[Any]: output = [] assert [l[0]] + output == l return output @hillelogram hillelwayne.com

Slide 56

Slide 56 text

@hillelogram >>> tail([1,2,3]) Traceback (most recent call last): File "", line 1, in File "", line 3, in tail AssertionError: Postcondition >>> hillelwayne.com

Slide 57

Slide 57 text

def tail(l: List[Any]) -> List[Any]: output = l[1:] assert [l[0]] + output == l return output @hillelogram hillelwayne.com

Slide 58

Slide 58 text

@hillelogram >>> tail([1,2,3]) [2, 3] >>> tail([]) Traceback (most recent call last): File "", line 1, in File "", line 3, in tail IndexError: list index out of range >>> hillelwayne.com

Slide 59

Slide 59 text

def tail(l: List[Any]) -> List[Any]: assert len(l) > 0 output = l[1:] assert [l[0]] + output == l return output @hillelogram hillelwayne.com

Slide 60

Slide 60 text

@hillelogram >>> tail([]) Traceback (most recent call last): File "", line 1, in File "", line 2, in tail AssertionError: Precondition >>> hillelwayne.com

Slide 61

Slide 61 text

Design By Contract hillelwayne.com @hillelogram

Slide 62

Slide 62 text

"Contracts" https://github.com/deadpixi/contracts hillelwayne.com @hillelogram

Slide 63

Slide 63 text

@require("l must not be empty", lambda args: len(args.l) > 0) def tail(l: List[Any]) -> List[Any]: assert len(l) > 0 output = l[1:] assert [l[0]] + output == l return output @hillelogram hillelwayne.com

Slide 64

Slide 64 text

@hillelogram @require("l must not be empty", lambda args: len(args.l) > 0) @ensure("result is tail of list", lambda args, result: [args.l[0]] + result == args.l) def tail(l: List[Any]) -> List[Any]: output = l[1:] assert [l[0]] + output == l return output hillelwayne.com

Slide 65

Slide 65 text

@hillelogram @require("l must not be empty", lambda a: len(a.l) > 0) @ensure("result is tail of list", lambda a, r: [a.l[0]] + r == a.l) def tail(l: List[Any]) -> List[Any]: return l[1:] hillelwayne.com

Slide 66

Slide 66 text

What’s our property test? @given(lists(integers(), 1)) def test_tail(l): tail(l) @hillelogram hillelwayne.com

Slide 67

Slide 67 text

"Full Specification" hillelwayne.com @hillelogram

Slide 68

Slide 68 text

def is_max_prod(a, r): pairs = combinations(a.l, 2) return all((r >= x * y for x, y in pairs)) @hillelogram hillelwayne.com

Slide 69

Slide 69 text

@require("2+ elements", lambda a: len(a.l) > 1) @ensure("is the max prod", is_max_prod) def max_product(l: List[int]) -> int: # efficient version @hillelogram hillelwayne.com

Slide 70

Slide 70 text

@given(lists(integers(), min_size=2)) def test_max_product(li): max_product(li) @hillelogram hillelwayne.com

Slide 71

Slide 71 text

Contracts + Property Tests = Integration Tests @hillelogram hillelwayne.com

Slide 72

Slide 72 text

@hillelogram hillelwayne.com

Slide 73

Slide 73 text

@hillelogram hillelwayne.com

Slide 74

Slide 74 text

@hillelogram hillelwayne.com

Slide 75

Slide 75 text

@hillelogram hillelwayne.com

Slide 76

Slide 76 text

@hillelogram hillelwayne.com

Slide 77

Slide 77 text

Can we go further? @hillelogram hillelwayne.com

Slide 78

Slide 78 text

Class Invariants @invariant("No overdrafts", lambda self: self.account >= 0) class BankAccount: def withdraw(self, val): ... def deposit(self, val): ... @hillelogram hillelwayne.com

Slide 79

Slide 79 text

Stateful Testing class NumberModifier(RuleBasedStateMachine): num = 0 @rule(x=integers()) def add_num(x): self.num += x @precondition(lambda self: self.num != 0) @rule() def reciprocal(self): self.num = 1 / self.num @hillelogram hillelwayne.com

Slide 80

Slide 80 text

Crazier Stuff • Test Generators (Clojure) • Contract Inheritance (Eiffel) • Global Contracts (Ada) • Static verification (Dafny) • ABC metatyping (Python???) @hillelogram hillelwayne.com

Slide 81

Slide 81 text

Conclusion @hillelogram Unit Tests => Property Tests Mypy => Contracts Integration Tests => Property Tests + Contracts hillelwayne.com

Slide 82

Slide 82 text

Hypothesis: http://hypothesis.works/ Contracts: https://github.com/deadpixi/contracts @hillelogram hillelwayne.com

Slide 83

Slide 83 text

WRITE UNIT TESTS hillelwayne.com @hillelogram

Slide 84

Slide 84 text

WRITE UNIT TESTS hillelwayne.com @hillelogram

Slide 85

Slide 85 text

WRITE UNIT TESTS hillelwayne.com @hillelogram

Slide 86

Slide 86 text

Formal Methods hillelwayne.com @hillelogram

Slide 87

Slide 87 text

Come to the formal methods open space! Saturday 2 PM Room 9 hillelwayne.com @hillelogram

Slide 88

Slide 88 text

Hillel Wayne hillelwayne.com @hillelogram