Slide 1

Slide 1 text

Property-Based Testing In Python Matt Bachmann @mattbachmann

Slide 2

Slide 2 text

Testing is Important Refactoring Design Regression Protection Faster Development

Slide 3

Slide 3 text

Testing is Hard Code Isolation Fixtures Indirect Value

Slide 4

Slide 4 text

Capture the Important Cases Minimize The Coding Overhead

Slide 5

Slide 5 text

Sorting a list of integers

Slide 6

Slide 6 text

def test_sort_empty(xs): assert quicksort([]) == [] def test_sorted(xs): assert quicksort([1, 2, 3]) == [1, 2, 3] def test_sort_unsorted(xs): assert quicksort([5, 4]) == [4, 5]

Slide 7

Slide 7 text

Property Based Testing ● Describe the arguments ● Describe the result ● Have the computer try to prove your code wrong

Slide 8

Slide 8 text

● The Arguments ○ List of Integers ● Properties of the Result ○ List ○ All elements preserved ○ Results are in ascending order

Slide 9

Slide 9 text

Hypothesis Inspired by Haskell’s QuickCheck Fantastic Docs Offers training/contracting http://hypothesis.works/

Slide 10

Slide 10 text

@given(st.lists(st.integers())) def test_sort(xs): sorted_xs = quicksort(xs) assert isinstance(sorted_xs, list) assert Counter(xs) == Counter(sorted_xs) assert all( x <= y for x, y in zip(sorted_xs, sorted_xs[1:]) )

Slide 11

Slide 11 text

@given(st.lists(st.integers())) def test_sort(xs): sorted_xs = quicksort(xs) assert isinstance(sorted_xs, list) assert Counter(xs) == Counter(sorted_xs) assert all( x <= y for x, y in zip(sorted_xs, sorted_xs[1:]) )

Slide 12

Slide 12 text

@given(st.lists(st.integers())) def test_sort(xs): sorted_xs = quicksort(xs) assert isinstance(sorted_xs, list) assert Counter(xs) == Counter(sorted_xs) assert all( x <= y for x, y in zip(sorted_xs, sorted_xs[1:]) )

Slide 13

Slide 13 text

@given(st.lists(st.integers())) def test_sort(xs): sorted_xs = quicksort(xs) assert isinstance(sorted_xs, list) assert Counter(xs) == Counter(sorted_xs) assert all( x <= y for x, y in zip(sorted_xs, sorted_xs[1:]) )

Slide 14

Slide 14 text

@given(st.lists(st.integers())) def test_sort(xs): sorted_xs = quicksort(xs) assert isinstance(sorted_xs, list) assert Counter(xs) == Counter(sorted_xs) assert all( x <= y for x, y in zip(sorted_xs, sorted_xs[1:]) )

Slide 15

Slide 15 text

@given(st.lists(st.integers())) def test_sort(xs): sorted_xs = quicksort(xs) assert isinstance(sorted_xs, list) assert Counter(xs) == Counter(sorted_xs) assert all( x <= y for x, y in zip(sorted_xs, sorted_xs[1:]) )

Slide 16

Slide 16 text

@given(st.lists(st.integers())) def test_sort(xs): sorted_xs = quicksort(xs) assert isinstance(sorted_xs, list) assert Counter(xs) == Counter(sorted_xs) assert all( x <= y for x, y in zip(sorted_xs, sorted_xs[1:]) )

Slide 17

Slide 17 text

@given(st.lists(st.integers())) test_sort([]) test_sort([0]) test_sort([93932932923, 82883982983838]) test_sort([9999,77,2,3,3,100,3,39,3993])

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

[-3223, -99999999999, 32]

Slide 20

Slide 20 text

[-3223, -99999999999, 32] [-3223, -99999999999]

Slide 21

Slide 21 text

[-3223, -99999999999, 32] [-3223, -99999999999] [1, 0]

Slide 22

Slide 22 text

@given

Slide 23

Slide 23 text

Strategies booleans, floats, strings, complex_numbers

Slide 24

Slide 24 text

Strategies booleans, floats, strings, complex_numbers dictionaries, tuples, lists, sets, builds

Slide 25

Slide 25 text

Strategies booleans, floats, strings, complex_numbers dictionaries, tuples, lists, sets, builds one_of, sampled_from, recursive

Slide 26

Slide 26 text

@given( st.builds( Dog, breed=st.text(), name=st.text(), height=st.floats(), weight=st.floats() ) )

Slide 27

Slide 27 text

@given( st.builds( Dog, breed=st.sampled_from(KNOWN_BREEDS), name=st.text(), height=st.floats(), weight=st.floats() ) )

Slide 28

Slide 28 text

@given( st.builds( Dog, breed=st.sampled_from(KNOWN_BREEDS), name=st.text(min_size=5), height=st.floats(), weight=st.floats() ) )

Slide 29

Slide 29 text

@given( st.builds( Dog, breed=st.sampled_from(KNOWN_BREEDS), name=st.text(min_size=5), height=st.floats( min_value=1, max_value=6, allow_nan=False, allow_infinity=False ), weight=st.floats( min_value=1, max_value=300, allow_nan=False, allow_infinity=False ), ) )

Slide 30

Slide 30 text

@given( st.builds( Dog, breed=st.sampled_from(KNOWN_BREEDS), name=st.text(min_size=5), height=st.floats( min_value=1, max_value=6, allow_nan=False, allow_infinity=False ), weight=st.floats( min_value=1, max_value=300, allow_nan=False, allow_infinity=False ), ) ) @example(Dog(breed="Labrador", name="Spot", height=2.1, weight=70))

Slide 31

Slide 31 text

Potentially infinite cases Nothing hardcoded Very little code

Slide 32

Slide 32 text

https://www.flickr.com/photos/t0fugurl/2507049701 Picture (without crossout) by Leanne Poon https://creativecommons.org/licenses/by-nc-nd/2.0/

Slide 33

Slide 33 text

Pattern 1: The code should not explode

Slide 34

Slide 34 text

https://upload.wikimedia. org/wikipedia/commons/3/39/Chip-pan- fire.jpg

Slide 35

Slide 35 text

/batputer/criminals/aliases/{id}? sort={sort}&max_results={max} ● JSON response ● Expected response codes ○ 200 - OK ○ 401 - Forbidden ○ 400 - Invalid data ○ 404 - Not Found

Slide 36

Slide 36 text

Slide 37

Slide 37 text

https://upload.wikimedia. org/wikipedia/commons/3/39/Chip-pan- fire.jpg

Slide 38

Slide 38 text

@given(st.integers(), st.text(), st.integers())) def test_no_explosion(id, sort, max): response = requests.get( BATPUTER_URL.format(id, sort, max) ) assert response and response.json() assert (response.status_code in [200, 401, 400, 404])

Slide 39

Slide 39 text

@given(st.integers(), st.text(), st.integers())) def test_no_explosion(id, sort, max): response = requests.get( BATPUTER_URL.format(id, sort, max) ) assert response and response.json() assert (response.status_code in [200, 401, 400, 404])

Slide 40

Slide 40 text

@given(st.integers(), st.text(), st.integers())) def test_no_explosion(id, sort, max): response = requests.get( BATPUTER_URL.format(id, sort, max) ) assert response and response.json() assert (response.status_code in [200, 401, 400, 404])

Slide 41

Slide 41 text

@given(st.integers(), st.text(), st.integers())) def test_no_explosion(id, sort, max): response = requests.get( BATPUTER_URL.format(id, sort, max) ) assert response and response.json() assert (response.status_code in [200, 401, 400, 404])

Slide 42

Slide 42 text

Pattern 2: Reversible Operations

Slide 43

Slide 43 text

Encoding Undo Operations Serialization

Slide 44

Slide 44 text

Encoding Undo Operations Serialization

Slide 45

Slide 45 text

class HistoricalEvent(object): def __init__(self, id, desc, time): self.id = id self.desc = desc self.time = time

Slide 46

Slide 46 text

class EventEncoder(json.JSONEncoder): def default(self, obj): if isinstance(obj, HistoricalEvent): return { "id": obj.id "description": obj.description "event_time":obj.event_time.isoformat() } return json.JSONEncoder.default(self, obj)

Slide 47

Slide 47 text

def fromJson(json_str): dct = json.loads(json_str) return HistoricalEvent( dct['id'], dct['description'], dateutil.parser.parse( dct['event_time'] ) )

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

@given(st.integers(), st.text(), datetimes(timezones=['UTC'])) def test_to_from_json(id, desc, date): original_event = HistoricalEvent(id, desc, date) encoded_event = json.dumps(event, cls=EventEncoder) decoded_event = json.loads( encoded_event, object_hook=HistoricalEvent.fromJson ) assert decoded_event == original_event

Slide 50

Slide 50 text

@given(st.integers(), st.text(), datetimes(timezones=['UTC'])) def test_to_from_json(id, desc, date): original_event = HistoricalEvent(id, desc, date) encoded_event = json.dumps(event, cls=EventEncoder) decoded_event = json.loads( encoded_event, object_hook=HistoricalEvent.fromJson ) assert decoded_event == original_event

Slide 51

Slide 51 text

@given(st.integers(), st.text(), datetimes(timezones=['UTC'])) def test_to_from_json(id, desc, date): original_event = HistoricalEvent(id, desc, date) encoded_event = json.dumps(event, cls=EventEncoder) decoded_event = json.loads( encoded_event, object_hook=HistoricalEvent.fromJson ) assert decoded_event == original_event

Slide 52

Slide 52 text

@given(st.integers(), st.text(), datetimes(timezones=['UTC'])) def test_to_from_json(id, desc, date): original_event = HistoricalEvent(id, desc, date) encoded_event = json.dumps(event, cls=EventEncoder) decoded_event = json.loads( encoded_event, object_hook=HistoricalEvent.fromJson ) assert decoded_event == original_event

Slide 53

Slide 53 text

@given(st.integers(), st.text(), datetimes(timezones=['UTC'])) def test_to_from_json(id, desc, date): original_event = HistoricalEvent(id, desc, date) encoded_event = json.dumps(event, cls=EventEncoder) decoded_event = json.loads( encoded_event, object_hook=HistoricalEvent.fromJson ) assert decoded_event == original_event

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

Pattern 3: Testing Oracle

Slide 56

Slide 56 text

Optimizing Refactoring Emulating

Slide 57

Slide 57 text

https://www.flickr.com/photos/versageek/493800514 versageek https://creativecommons.org/licenses/by-sa/2.0/ Leave It Alone

Slide 58

Slide 58 text

https://www.flickr.com/photos/versageek/493800514 versageek https://creativecommons.org/licenses/by-sa/2.0/ https://upload.wikimedia. org/wikipedia/commons/3/36/Under_Floor_Cable_ Runs_Tee.jpg =

Slide 59

Slide 59 text

@given(st.integers(), st.text()) def test_against_legacy(arg1, arg2): assert ( new_hotness(arg1, arg2) == legacy_system(arg1, arg2) )

Slide 60

Slide 60 text

Compare Against Brute Force

Slide 61

Slide 61 text

@given(st.lists(st.integers())) def test_against_brute_force(input): assert ( easy_but_inefficent(input) == optimized(input) )

Slide 62

Slide 62 text

Pattern 4: Stateful Testing

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

No content

Slide 65

Slide 65 text

No content

Slide 66

Slide 66 text

Stateful Tests ● Define a state ● What operations can happen in what conditions? ● How do operations affect the state? ● What must be true for each step?

Slide 67

Slide 67 text

https://upload.wikimedia.org/wikipedia/commons/8/83/Jan_Fabre's_%22Searching_for_Utopia%22.jpg FIND. ME. BUGS.

Slide 68

Slide 68 text

https://commons.wikimedia.org/wiki/File:Max-Heap.svg http://hypothesis.works/articles/rule-based-stateful-testing/

Slide 69

Slide 69 text

http://hypothesis.works/articles/rule-based-stateful-testing/ __init__ push pop merge

Slide 70

Slide 70 text

http://hypothesis.works/articles/rule-based-stateful-testing/ Integers Heaps

Slide 71

Slide 71 text

http://hypothesis.works/articles/rule-based-stateful-testing/ __init__ Integers Heaps

Slide 72

Slide 72 text

http://hypothesis.works/articles/rule-based-stateful-testing/ push Integers Heaps

Slide 73

Slide 73 text

http://hypothesis.works/articles/rule-based-stateful-testing/ merge Integers Heaps

Slide 74

Slide 74 text

http://hypothesis.works/articles/rule-based-stateful-testing/ merge Integers Heaps

Slide 75

Slide 75 text

http://hypothesis.works/articles/rule-based-stateful-testing/ pop Integers Heaps assert result actually max

Slide 76

Slide 76 text

http://hypothesis.works/articles/rule-based-stateful-testing/ class HeapMachine(RuleBasedStateMachine): Heaps = Bundle('heaps') @rule(target=Heaps) def new_heap(self): return Heap() @rule(heap=Heaps, value=integers()) def heap_push(self, heap, value): push(heap, value)

Slide 77

Slide 77 text

http://hypothesis.works/articles/rule-based-stateful-testing/ class HeapMachine(RuleBasedStateMachine): Heaps = Bundle('heaps') @rule(target=Heaps) def new_heap(self): return Heap() @rule(heap=Heaps, value=integers()) def heap_push(self, heap, value): push(heap, value)

Slide 78

Slide 78 text

class HeapMachine(RuleBasedStateMachine): Heaps = Bundle('heaps') @rule(target=Heaps) def new_heap(self): return Heap() @rule(heap=Heaps, value=integers()) def heap_push(self, heap, value): push(heap, value) http://hypothesis.works/articles/rule-based-stateful-testing/

Slide 79

Slide 79 text

class HeapMachine(RuleBasedStateMachine): Heaps = Bundle('heaps') @rule(target=Heaps) def new_heap(self): return Heap() @rule(heap=Heaps, value=integers()) def heap_push(self, heap, value): push(heap, value) http://hypothesis.works/articles/rule-based-stateful-testing/

Slide 80

Slide 80 text

http://hypothesis.works/articles/rule-based-stateful-testing/ … @rule(target=Heaps, heap1=Heaps, heap2=Heaps) def merge(self, heap1, heap2): return heap_merge(heap1, heap2) @rule(heap=Heaps.filter(bool)) def pop(self, heap): correct = max(list(heap)) result = heap_pop(heap) assert correct == result

Slide 81

Slide 81 text

http://hypothesis.works/articles/rule-based-stateful-testing/ … @rule(target=Heaps, heap1=Heaps, heap2=Heaps) def merge(self, heap1, heap2): return heap_merge(heap1, heap2) @rule(heap=Heaps.filter(bool)) def pop(self, heap): correct = max(list(heap)) result = heap_pop(heap) assert correct == result

Slide 82

Slide 82 text

https://upload.wikimedia.org/wikipedia/commons/8/83/Jan_Fabre's_%22Searching_for_Utopia%22.jpg FIND. ME. BUGS.

Slide 83

Slide 83 text

http://hypothesis.works/articles/rule-based-stateful-testing/ @rule(heap=Heaps.filter(bool)) def pop(self, heap): correct = max(list(heap)) result = heap_pop(heap) > assert correct == result E AssertionError: assert 1 == 0

Slide 84

Slide 84 text

http://hypothesis.works/articles/rule-based-stateful-testing/ v1 = new_heap() push(heap=v1, value=0) push(heap=v1, value=1) push(heap=v1, value=1) v2 = merge(heap2=v1, heap1=v1) pop(heap=v2) pop(heap=v2)

Slide 85

Slide 85 text

No content

Slide 86

Slide 86 text

Property Based Testing ● Describe the arguments ● Describe the result ● Have the computer try to prove your code wrong

Slide 87

Slide 87 text

Your Turn ● Download Hypothesis ● Use it ● Share how you used it ● FIND MORE PATTERNS

Slide 88

Slide 88 text

More about Hypothesis ● http://hypothesis.works/ More On Property Based Testing In General ● http://www.quviq.com/ ● https://fsharpforfunandprofit.com/posts/property-based-testing-2/ Testing Eventual Consistency with RIAK ● https://www.youtube.com/watch?v=x9mW54GJpG0

Slide 89

Slide 89 text

Thanks! Matt Bachmann https://github.com/Bachmann1234 Twitter: @mattbachmann Slides: goo.gl/g0mGgU