PyCon 2016
May 29, 2016
710

# Matt Bachmann - Better Testing With Less Code: Property Based Testing With Python

Standard unit tests have developers test specific inputs and outputs. This works, but often what breaks code are the cases we did not think about. Property based testing has developers define properties of output and has the computer explore the possible inputs to verify these properties. This talk will introduce property based testing and provide real world examples and patterns.

https://us.pycon.org/2016/schedule/presentation/1927/

May 29, 2016

## Transcript

1. Property-Based
Testing In
Python
Matt Bachmann @mattbachmann

2. Testing is Important
Refactoring
Design
Regression Protection
Faster Development

3. Testing is Hard
Code
Isolation
Fixtures
Indirect Value

4. Capture the
Important Cases
Minimize The

5. Sorting a list of
integers

6. def test_sort_empty(xs):
assert quicksort([]) == []
def test_sorted(xs):
assert quicksort([1, 2, 3]) == [1, 2, 3]
def test_sort_unsorted(xs):
assert quicksort([5, 4]) == [4, 5]

7. Property Based Testing
● Describe the arguments
● Describe the result
● Have the computer try to
prove your code wrong

8. ● The Arguments
○ List of Integers
● Properties of the Result
○ List
○ All elements preserved
○ Results are in ascending
order

9. Hypothesis
QuickCheck
Fantastic Docs
Offers training/contracting
http://hypothesis.works/

10. @given(st.lists(st.integers()))
def test_sort(xs):
sorted_xs = quicksort(xs)
assert isinstance(sorted_xs, list)
assert Counter(xs) == Counter(sorted_xs)
assert all(
x <= y for x, y in
zip(sorted_xs, sorted_xs[1:])
)

11. @given(st.lists(st.integers()))
def test_sort(xs):
sorted_xs = quicksort(xs)
assert isinstance(sorted_xs, list)
assert Counter(xs) == Counter(sorted_xs)
assert all(
x <= y for x, y in
zip(sorted_xs, sorted_xs[1:])
)

12. @given(st.lists(st.integers()))
def test_sort(xs):
sorted_xs = quicksort(xs)
assert isinstance(sorted_xs, list)
assert Counter(xs) == Counter(sorted_xs)
assert all(
x <= y for x, y in
zip(sorted_xs, sorted_xs[1:])
)

13. @given(st.lists(st.integers()))
def test_sort(xs):
sorted_xs = quicksort(xs)
assert isinstance(sorted_xs, list)
assert Counter(xs) == Counter(sorted_xs)
assert all(
x <= y for x, y in
zip(sorted_xs, sorted_xs[1:])
)

14. @given(st.lists(st.integers()))
def test_sort(xs):
sorted_xs = quicksort(xs)
assert isinstance(sorted_xs, list)
assert Counter(xs) == Counter(sorted_xs)
assert all(
x <= y for x, y in
zip(sorted_xs, sorted_xs[1:])
)

15. @given(st.lists(st.integers()))
def test_sort(xs):
sorted_xs = quicksort(xs)
assert isinstance(sorted_xs, list)
assert Counter(xs) == Counter(sorted_xs)
assert all(
x <= y for x, y in
zip(sorted_xs, sorted_xs[1:])
)

16. @given(st.lists(st.integers()))
def test_sort(xs):
sorted_xs = quicksort(xs)
assert isinstance(sorted_xs, list)
assert Counter(xs) == Counter(sorted_xs)
assert all(
x <= y for x, y in
zip(sorted_xs, sorted_xs[1:])
)

17. @given(st.lists(st.integers()))
test_sort([])
test_sort([0])
test_sort([93932932923, 82883982983838])
test_sort([9999,77,2,3,3,100,3,39,3993])

18. [-3223, -99999999999, 32]

19. [-3223, -99999999999, 32]
[-3223, -99999999999]

20. [-3223, -99999999999, 32]
[-3223, -99999999999]
[1, 0]

21. @given

22. Strategies
booleans, floats, strings, complex_numbers

23. Strategies
booleans, floats, strings, complex_numbers
dictionaries, tuples, lists, sets, builds

24. Strategies
booleans, floats, strings, complex_numbers
dictionaries, tuples, lists, sets, builds
one_of, sampled_from, recursive

25. @given(
st.builds(
Dog,
breed=st.text(),
name=st.text(),
height=st.floats(),
weight=st.floats()
)
)

26. @given(
st.builds(
Dog,
breed=st.sampled_from(KNOWN_BREEDS),
name=st.text(),
height=st.floats(),
weight=st.floats()
)
)

27. @given(
st.builds(
Dog,
breed=st.sampled_from(KNOWN_BREEDS),
name=st.text(min_size=5),
height=st.floats(),
weight=st.floats()
)
)

28. @given(
st.builds(
Dog,
breed=st.sampled_from(KNOWN_BREEDS),
name=st.text(min_size=5),
height=st.floats(
min_value=1, max_value=6, allow_nan=False, allow_infinity=False
),
weight=st.floats(
min_value=1, max_value=300, allow_nan=False, allow_infinity=False
),
)
)

29. @given(
st.builds(
Dog,
breed=st.sampled_from(KNOWN_BREEDS),
name=st.text(min_size=5),
height=st.floats(
min_value=1, max_value=6, allow_nan=False, allow_infinity=False
),
weight=st.floats(
min_value=1, max_value=300, allow_nan=False, allow_infinity=False
),
)
)
@example(Dog(breed="Labrador", name="Spot", height=2.1, weight=70))

30. Potentially infinite cases
Nothing hardcoded
Very little code

31. https://www.flickr.com/photos/t0fugurl/2507049701
Picture (without crossout) by Leanne Poon

32. Pattern 1:
The code should not
explode

org/wikipedia/commons/3/39/Chip-pan-
fire.jpg

34. /batputer/criminals/aliases/{id}?
sort={sort}&max_results={max}
● JSON response
● Expected response codes
○ 200 - OK
○ 401 - Forbidden
○ 400 - Invalid data

org/wikipedia/commons/3/39/Chip-pan-
fire.jpg

36. @given(st.integers(),
st.text(),
st.integers()))
def test_no_explosion(id, sort, max):
response = requests.get(
BATPUTER_URL.format(id, sort, max)
)
assert response and response.json()
assert (response.status_code in
[200, 401, 400, 404])

37. @given(st.integers(),
st.text(),
st.integers()))
def test_no_explosion(id, sort, max):
response = requests.get(
BATPUTER_URL.format(id, sort, max)
)
assert response and response.json()
assert (response.status_code in
[200, 401, 400, 404])

38. @given(st.integers(),
st.text(),
st.integers()))
def test_no_explosion(id, sort, max):
response = requests.get(
BATPUTER_URL.format(id, sort, max)
)
assert response and response.json()
assert (response.status_code in
[200, 401, 400, 404])

39. @given(st.integers(),
st.text(),
st.integers()))
def test_no_explosion(id, sort, max):
response = requests.get(
BATPUTER_URL.format(id, sort, max)
)
assert response and response.json()
assert (response.status_code in
[200, 401, 400, 404])

40. Pattern 2:
Reversible
Operations

41. Encoding
Undo Operations
Serialization

42. Encoding
Undo Operations
Serialization

43. class HistoricalEvent(object):
def __init__(self, id, desc, time):
self.id = id
self.desc = desc
self.time = time

44. class EventEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, HistoricalEvent):
return {
"id": obj.id
"description": obj.description
"event_time":obj.event_time.isoformat()
}
return json.JSONEncoder.default(self, obj)

45. def fromJson(json_str):
return HistoricalEvent(
dct['id'],
dct['description'],
dateutil.parser.parse(
dct['event_time']
)
)

46. @given(st.integers(), st.text(), datetimes(timezones=['UTC']))
def test_to_from_json(id, desc, date):
original_event = HistoricalEvent(id, desc, date)
encoded_event = json.dumps(event, cls=EventEncoder)
encoded_event,
object_hook=HistoricalEvent.fromJson
)
assert decoded_event == original_event

47. @given(st.integers(), st.text(), datetimes(timezones=['UTC']))
def test_to_from_json(id, desc, date):
original_event = HistoricalEvent(id, desc, date)
encoded_event = json.dumps(event, cls=EventEncoder)
encoded_event,
object_hook=HistoricalEvent.fromJson
)
assert decoded_event == original_event

48. @given(st.integers(), st.text(), datetimes(timezones=['UTC']))
def test_to_from_json(id, desc, date):
original_event = HistoricalEvent(id, desc, date)
encoded_event = json.dumps(event, cls=EventEncoder)
encoded_event,
object_hook=HistoricalEvent.fromJson
)
assert decoded_event == original_event

49. @given(st.integers(), st.text(), datetimes(timezones=['UTC']))
def test_to_from_json(id, desc, date):
original_event = HistoricalEvent(id, desc, date)
encoded_event = json.dumps(event, cls=EventEncoder)
encoded_event,
object_hook=HistoricalEvent.fromJson
)
assert decoded_event == original_event

50. @given(st.integers(), st.text(), datetimes(timezones=['UTC']))
def test_to_from_json(id, desc, date):
original_event = HistoricalEvent(id, desc, date)
encoded_event = json.dumps(event, cls=EventEncoder)
encoded_event,
object_hook=HistoricalEvent.fromJson
)
assert decoded_event == original_event

51. Pattern 3:
Testing Oracle

52. Optimizing
Refactoring
Emulating

53. https://www.flickr.com/photos/versageek/493800514
versageek
Leave It
Alone

54. https://www.flickr.com/photos/versageek/493800514
versageek
org/wikipedia/commons/3/36/Under_Floor_Cable_
Runs_Tee.jpg
=

55. @given(st.integers(), st.text())
def test_against_legacy(arg1, arg2):
assert (
new_hotness(arg1, arg2)
==
legacy_system(arg1, arg2)
)

56. Compare
Against Brute
Force

57. @given(st.lists(st.integers()))
def test_against_brute_force(input):
assert (
easy_but_inefficent(input)
==
optimized(input)
)

58. Pattern 4:
Stateful Testing

59. Stateful Tests
● Define a state
● What operations can happen in
what conditions?
● How do operations affect the
state?
● What must be true for each step?

FIND. ME. BUGS.

61. https://commons.wikimedia.org/wiki/File:Max-Heap.svg
http://hypothesis.works/articles/rule-based-stateful-testing/

62. http://hypothesis.works/articles/rule-based-stateful-testing/
__init__
push
pop
merge

63. http://hypothesis.works/articles/rule-based-stateful-testing/
Integers
Heaps

64. http://hypothesis.works/articles/rule-based-stateful-testing/
__init__ Integers
Heaps

65. http://hypothesis.works/articles/rule-based-stateful-testing/
push Integers
Heaps

66. http://hypothesis.works/articles/rule-based-stateful-testing/
merge Integers
Heaps

67. http://hypothesis.works/articles/rule-based-stateful-testing/
merge Integers
Heaps

68. http://hypothesis.works/articles/rule-based-stateful-testing/
pop Integers
Heaps
assert result actually max

69. http://hypothesis.works/articles/rule-based-stateful-testing/
class HeapMachine(RuleBasedStateMachine):
Heaps = Bundle('heaps')
@rule(target=Heaps)
def new_heap(self):
return Heap()
@rule(heap=Heaps, value=integers())
def heap_push(self, heap, value):
push(heap, value)

70. http://hypothesis.works/articles/rule-based-stateful-testing/
class HeapMachine(RuleBasedStateMachine):
Heaps = Bundle('heaps')
@rule(target=Heaps)
def new_heap(self):
return Heap()
@rule(heap=Heaps, value=integers())
def heap_push(self, heap, value):
push(heap, value)

71. class HeapMachine(RuleBasedStateMachine):
Heaps = Bundle('heaps')
@rule(target=Heaps)
def new_heap(self):
return Heap()
@rule(heap=Heaps, value=integers())
def heap_push(self, heap, value):
push(heap, value)
http://hypothesis.works/articles/rule-based-stateful-testing/

72. class HeapMachine(RuleBasedStateMachine):
Heaps = Bundle('heaps')
@rule(target=Heaps)
def new_heap(self):
return Heap()
@rule(heap=Heaps, value=integers())
def heap_push(self, heap, value):
push(heap, value)
http://hypothesis.works/articles/rule-based-stateful-testing/

73. http://hypothesis.works/articles/rule-based-stateful-testing/

@rule(target=Heaps, heap1=Heaps, heap2=Heaps)
def merge(self, heap1, heap2):
return heap_merge(heap1, heap2)
@rule(heap=Heaps.filter(bool))
def pop(self, heap):
correct = max(list(heap))
result = heap_pop(heap)
assert correct == result

74. http://hypothesis.works/articles/rule-based-stateful-testing/

@rule(target=Heaps, heap1=Heaps, heap2=Heaps)
def merge(self, heap1, heap2):
return heap_merge(heap1, heap2)
@rule(heap=Heaps.filter(bool))
def pop(self, heap):
correct = max(list(heap))
result = heap_pop(heap)
assert correct == result

FIND. ME. BUGS.

76. http://hypothesis.works/articles/rule-based-stateful-testing/
@rule(heap=Heaps.filter(bool))
def pop(self, heap):
correct = max(list(heap))
result = heap_pop(heap)
> assert correct == result
E AssertionError: assert 1 == 0

77. http://hypothesis.works/articles/rule-based-stateful-testing/
v1 = new_heap()
push(heap=v1, value=0)
push(heap=v1, value=1)
push(heap=v1, value=1)
v2 = merge(heap2=v1, heap1=v1)
pop(heap=v2)
pop(heap=v2)

78. Property Based Testing
● Describe the arguments
● Describe the result
● Have the computer try to
prove your code wrong

● Use it
● Share how you used it
● FIND MORE PATTERNS

80. More about Hypothesis
● http://hypothesis.works/
More On Property Based Testing In General
● http://www.quviq.com/
● https://fsharpforfunandprofit.com/posts/property-based-testing-2/
Testing Eventual Consistency with RIAK