Matt Bachmann - Better Testing With Less Code: Property Based Testing With Python

Standard unit tests have developers test specific inputs and outputs. This works, but often what breaks code are the cases we did not think about. Property based testing has developers define properties of output and has the computer explore the possible inputs to verify these properties. This talk will introduce property based testing and provide real world examples and patterns.

https://us.pycon.org/2016/schedule/presentation/1927/

Transcript

1. Property-Based
   Testing In
   Python
   Matt Bachmann @mattbachmann
   

   View Slide

2. Testing is Important
   Refactoring
   Design
   Regression Protection
   Faster Development
   

   View Slide

3. Testing is Hard
   Code
   Isolation
   Fixtures
   Indirect Value
   

   View Slide

4. Capture the
   Important Cases
   Minimize The
   Coding Overhead
   

   View Slide

5. Sorting a list of
   integers
   

   View Slide

6. def test_sort_empty(xs):
   assert quicksort([]) == []
   def test_sorted(xs):
   assert quicksort([1, 2, 3]) == [1, 2, 3]
   def test_sort_unsorted(xs):
   assert quicksort([5, 4]) == [4, 5]
   

   View Slide

7. Property Based Testing
   ● Describe the arguments
   ● Describe the result
   ● Have the computer try to
   prove your code wrong
   

   View Slide

8. ● The Arguments
   ○ List of Integers
   ● Properties of the Result
   ○ List
   ○ All elements preserved
   ○ Results are in ascending
   order
   

   View Slide

9. Hypothesis
   Inspired by Haskell’s
   QuickCheck
   Fantastic Docs
   Offers training/contracting
   http://hypothesis.works/
   

   View Slide

10. @given(st.lists(st.integers()))
    def test_sort(xs):
    sorted_xs = quicksort(xs)
    assert isinstance(sorted_xs, list)
    assert Counter(xs) == Counter(sorted_xs)
    assert all(
    x <= y for x, y in
    zip(sorted_xs, sorted_xs[1:])
    )
    

    View Slide

11. @given(st.lists(st.integers()))
    def test_sort(xs):
    sorted_xs = quicksort(xs)
    assert isinstance(sorted_xs, list)
    assert Counter(xs) == Counter(sorted_xs)
    assert all(
    x <= y for x, y in
    zip(sorted_xs, sorted_xs[1:])
    )
    

    View Slide

12. @given(st.lists(st.integers()))
    def test_sort(xs):
    sorted_xs = quicksort(xs)
    assert isinstance(sorted_xs, list)
    assert Counter(xs) == Counter(sorted_xs)
    assert all(
    x <= y for x, y in
    zip(sorted_xs, sorted_xs[1:])
    )
    

    View Slide

13. @given(st.lists(st.integers()))
    def test_sort(xs):
    sorted_xs = quicksort(xs)
    assert isinstance(sorted_xs, list)
    assert Counter(xs) == Counter(sorted_xs)
    assert all(
    x <= y for x, y in
    zip(sorted_xs, sorted_xs[1:])
    )
    

    View Slide

14. @given(st.lists(st.integers()))
    def test_sort(xs):
    sorted_xs = quicksort(xs)
    assert isinstance(sorted_xs, list)
    assert Counter(xs) == Counter(sorted_xs)
    assert all(
    x <= y for x, y in
    zip(sorted_xs, sorted_xs[1:])
    )
    

    View Slide

15. @given(st.lists(st.integers()))
    def test_sort(xs):
    sorted_xs = quicksort(xs)
    assert isinstance(sorted_xs, list)
    assert Counter(xs) == Counter(sorted_xs)
    assert all(
    x <= y for x, y in
    zip(sorted_xs, sorted_xs[1:])
    )
    

    View Slide

16. @given(st.lists(st.integers()))
    def test_sort(xs):
    sorted_xs = quicksort(xs)
    assert isinstance(sorted_xs, list)
    assert Counter(xs) == Counter(sorted_xs)
    assert all(
    x <= y for x, y in
    zip(sorted_xs, sorted_xs[1:])
    )
    

    View Slide

17. @given(st.lists(st.integers()))
    test_sort([])
    test_sort([0])
    test_sort([93932932923, 82883982983838])
    test_sort([9999,77,2,3,3,100,3,39,3993])
    

    View Slide

18. View Slide

19. [-3223, -99999999999, 32]
    

    View Slide

20. [-3223, -99999999999, 32]
    [-3223, -99999999999]
    

    View Slide

21. [-3223, -99999999999, 32]
    [-3223, -99999999999]
    [1, 0]
    

    View Slide

22. @given
    

    View Slide

23. Strategies
    booleans, floats, strings, complex_numbers
    

    View Slide

24. Strategies
    booleans, floats, strings, complex_numbers
    dictionaries, tuples, lists, sets, builds
    

    View Slide

25. Strategies
    booleans, floats, strings, complex_numbers
    dictionaries, tuples, lists, sets, builds
    one_of, sampled_from, recursive
    

    View Slide

26. @given(
    st.builds(
    Dog,
    breed=st.text(),
    name=st.text(),
    height=st.floats(),
    weight=st.floats()
    )
    )
    

    View Slide

27. @given(
    st.builds(
    Dog,
    breed=st.sampled_from(KNOWN_BREEDS),
    name=st.text(),
    height=st.floats(),
    weight=st.floats()
    )
    )
    

    View Slide

28. @given(
    st.builds(
    Dog,
    breed=st.sampled_from(KNOWN_BREEDS),
    name=st.text(min_size=5),
    height=st.floats(),
    weight=st.floats()
    )
    )
    

    View Slide

29. @given(
    st.builds(
    Dog,
    breed=st.sampled_from(KNOWN_BREEDS),
    name=st.text(min_size=5),
    height=st.floats(
    min_value=1, max_value=6, allow_nan=False, allow_infinity=False
    ),
    weight=st.floats(
    min_value=1, max_value=300, allow_nan=False, allow_infinity=False
    ),
    )
    )
    

    View Slide

30. @given(
    st.builds(
    Dog,
    breed=st.sampled_from(KNOWN_BREEDS),
    name=st.text(min_size=5),
    height=st.floats(
    min_value=1, max_value=6, allow_nan=False, allow_infinity=False
    ),
    weight=st.floats(
    min_value=1, max_value=300, allow_nan=False, allow_infinity=False
    ),
    )
    )
    @example(Dog(breed="Labrador", name="Spot", height=2.1, weight=70))
    

    View Slide

31. Potentially infinite cases
    Nothing hardcoded
    Very little code
    

    View Slide

32. https://www.flickr.com/photos/t0fugurl/2507049701
    Picture (without crossout) by Leanne Poon
    https://creativecommons.org/licenses/by-nc-nd/2.0/
    

    View Slide

33. Pattern 1:
    The code should not
    explode
    

    View Slide

34. https://upload.wikimedia.
    org/wikipedia/commons/3/39/Chip-pan-
    fire.jpg
    

    View Slide

35. /batputer/criminals/aliases/{id}?
    sort={sort}&max_results={max}
    ● JSON response
    ● Expected response codes
    ○ 200 - OK
    ○ 401 - Forbidden
    ○ 400 - Invalid data
    ○ 404 - Not Found
    

    View Slide

36. ✓
    

    View Slide

37. https://upload.wikimedia.
    org/wikipedia/commons/3/39/Chip-pan-
    fire.jpg
    

    View Slide

38. @given(st.integers(),
    st.text(),
    st.integers()))
    def test_no_explosion(id, sort, max):
    response = requests.get(
    BATPUTER_URL.format(id, sort, max)
    )
    assert response and response.json()
    assert (response.status_code in
    [200, 401, 400, 404])
    

    View Slide

39. @given(st.integers(),
    st.text(),
    st.integers()))
    def test_no_explosion(id, sort, max):
    response = requests.get(
    BATPUTER_URL.format(id, sort, max)
    )
    assert response and response.json()
    assert (response.status_code in
    [200, 401, 400, 404])
    

    View Slide

40. @given(st.integers(),
    st.text(),
    st.integers()))
    def test_no_explosion(id, sort, max):
    response = requests.get(
    BATPUTER_URL.format(id, sort, max)
    )
    assert response and response.json()
    assert (response.status_code in
    [200, 401, 400, 404])
    

    View Slide

41. @given(st.integers(),
    st.text(),
    st.integers()))
    def test_no_explosion(id, sort, max):
    response = requests.get(
    BATPUTER_URL.format(id, sort, max)
    )
    assert response and response.json()
    assert (response.status_code in
    [200, 401, 400, 404])
    

    View Slide

42. Pattern 2:
    Reversible
    Operations
    

    View Slide

43. Encoding
    Undo Operations
    Serialization
    

    View Slide

44. Encoding
    Undo Operations
    Serialization
    

    View Slide

45. class HistoricalEvent(object):
    def __init__(self, id, desc, time):
    self.id = id
    self.desc = desc
    self.time = time
    

    View Slide

46. class EventEncoder(json.JSONEncoder):
    def default(self, obj):
    if isinstance(obj, HistoricalEvent):
    return {
    "id": obj.id
    "description": obj.description
    "event_time":obj.event_time.isoformat()
    }
    return json.JSONEncoder.default(self, obj)
    

    View Slide

47. def fromJson(json_str):
    dct = json.loads(json_str)
    return HistoricalEvent(
    dct['id'],
    dct['description'],
    dateutil.parser.parse(
    dct['event_time']
    )
    )
    

    View Slide

48. View Slide

49. @given(st.integers(), st.text(), datetimes(timezones=['UTC']))
    def test_to_from_json(id, desc, date):
    original_event = HistoricalEvent(id, desc, date)
    encoded_event = json.dumps(event, cls=EventEncoder)
    decoded_event = json.loads(
    encoded_event,
    object_hook=HistoricalEvent.fromJson
    )
    assert decoded_event == original_event
    

    View Slide

50. @given(st.integers(), st.text(), datetimes(timezones=['UTC']))
    def test_to_from_json(id, desc, date):
    original_event = HistoricalEvent(id, desc, date)
    encoded_event = json.dumps(event, cls=EventEncoder)
    decoded_event = json.loads(
    encoded_event,
    object_hook=HistoricalEvent.fromJson
    )
    assert decoded_event == original_event
    

    View Slide

51. @given(st.integers(), st.text(), datetimes(timezones=['UTC']))
    def test_to_from_json(id, desc, date):
    original_event = HistoricalEvent(id, desc, date)
    encoded_event = json.dumps(event, cls=EventEncoder)
    decoded_event = json.loads(
    encoded_event,
    object_hook=HistoricalEvent.fromJson
    )
    assert decoded_event == original_event
    

    View Slide

52. @given(st.integers(), st.text(), datetimes(timezones=['UTC']))
    def test_to_from_json(id, desc, date):
    original_event = HistoricalEvent(id, desc, date)
    encoded_event = json.dumps(event, cls=EventEncoder)
    decoded_event = json.loads(
    encoded_event,
    object_hook=HistoricalEvent.fromJson
    )
    assert decoded_event == original_event
    

    View Slide

53. @given(st.integers(), st.text(), datetimes(timezones=['UTC']))
    def test_to_from_json(id, desc, date):
    original_event = HistoricalEvent(id, desc, date)
    encoded_event = json.dumps(event, cls=EventEncoder)
    decoded_event = json.loads(
    encoded_event,
    object_hook=HistoricalEvent.fromJson
    )
    assert decoded_event == original_event
    

    View Slide

54. View Slide

55. Pattern 3:
    Testing Oracle
    

    View Slide

56. Optimizing
    Refactoring
    Emulating
    

    View Slide

57. https://www.flickr.com/photos/versageek/493800514
    versageek
    https://creativecommons.org/licenses/by-sa/2.0/
    Leave It
    Alone
    

    View Slide

58. https://www.flickr.com/photos/versageek/493800514
    versageek
    https://creativecommons.org/licenses/by-sa/2.0/
    https://upload.wikimedia.
    org/wikipedia/commons/3/36/Under_Floor_Cable_
    Runs_Tee.jpg
    =
    

    View Slide

59. @given(st.integers(), st.text())
    def test_against_legacy(arg1, arg2):
    assert (
    new_hotness(arg1, arg2)
    ==
    legacy_system(arg1, arg2)
    )
    

    View Slide

60. Compare
    Against Brute
    Force
    

    View Slide

61. @given(st.lists(st.integers()))
    def test_against_brute_force(input):
    assert (
    easy_but_inefficent(input)
    ==
    optimized(input)
    )
    

    View Slide

62. Pattern 4:
    Stateful Testing
    

    View Slide

63. View Slide

64. View Slide

65. View Slide

66. Stateful Tests
    ● Define a state
    ● What operations can happen in
    what conditions?
    ● How do operations affect the
    state?
    ● What must be true for each step?
    

    View Slide

67. https://upload.wikimedia.org/wikipedia/commons/8/83/Jan_Fabre's_%22Searching_for_Utopia%22.jpg
    FIND. ME. BUGS.
    

    View Slide

68. https://commons.wikimedia.org/wiki/File:Max-Heap.svg
    http://hypothesis.works/articles/rule-based-stateful-testing/
    

    View Slide

69. http://hypothesis.works/articles/rule-based-stateful-testing/
    __init__
    push
    pop
    merge
    

    View Slide

70. http://hypothesis.works/articles/rule-based-stateful-testing/
    Integers
    Heaps
    

    View Slide

71. http://hypothesis.works/articles/rule-based-stateful-testing/
    __init__ Integers
    Heaps
    

    View Slide

72. http://hypothesis.works/articles/rule-based-stateful-testing/
    push Integers
    Heaps
    

    View Slide

73. http://hypothesis.works/articles/rule-based-stateful-testing/
    merge Integers
    Heaps
    

    View Slide

74. http://hypothesis.works/articles/rule-based-stateful-testing/
    merge Integers
    Heaps
    

    View Slide

75. http://hypothesis.works/articles/rule-based-stateful-testing/
    pop Integers
    Heaps
    assert result actually max
    

    View Slide

76. http://hypothesis.works/articles/rule-based-stateful-testing/
    class HeapMachine(RuleBasedStateMachine):
    Heaps = Bundle('heaps')
    @rule(target=Heaps)
    def new_heap(self):
    return Heap()
    @rule(heap=Heaps, value=integers())
    def heap_push(self, heap, value):
    push(heap, value)
    

    View Slide

77. http://hypothesis.works/articles/rule-based-stateful-testing/
    class HeapMachine(RuleBasedStateMachine):
    Heaps = Bundle('heaps')
    @rule(target=Heaps)
    def new_heap(self):
    return Heap()
    @rule(heap=Heaps, value=integers())
    def heap_push(self, heap, value):
    push(heap, value)
    

    View Slide

78. class HeapMachine(RuleBasedStateMachine):
    Heaps = Bundle('heaps')
    @rule(target=Heaps)
    def new_heap(self):
    return Heap()
    @rule(heap=Heaps, value=integers())
    def heap_push(self, heap, value):
    push(heap, value)
    http://hypothesis.works/articles/rule-based-stateful-testing/
    

    View Slide

79. class HeapMachine(RuleBasedStateMachine):
    Heaps = Bundle('heaps')
    @rule(target=Heaps)
    def new_heap(self):
    return Heap()
    @rule(heap=Heaps, value=integers())
    def heap_push(self, heap, value):
    push(heap, value)
    http://hypothesis.works/articles/rule-based-stateful-testing/
    

    View Slide

80. http://hypothesis.works/articles/rule-based-stateful-testing/
    …
    @rule(target=Heaps, heap1=Heaps, heap2=Heaps)
    def merge(self, heap1, heap2):
    return heap_merge(heap1, heap2)
    @rule(heap=Heaps.filter(bool))
    def pop(self, heap):
    correct = max(list(heap))
    result = heap_pop(heap)
    assert correct == result
    

    View Slide

81. http://hypothesis.works/articles/rule-based-stateful-testing/
    …
    @rule(target=Heaps, heap1=Heaps, heap2=Heaps)
    def merge(self, heap1, heap2):
    return heap_merge(heap1, heap2)
    @rule(heap=Heaps.filter(bool))
    def pop(self, heap):
    correct = max(list(heap))
    result = heap_pop(heap)
    assert correct == result
    

    View Slide

82. https://upload.wikimedia.org/wikipedia/commons/8/83/Jan_Fabre's_%22Searching_for_Utopia%22.jpg
    FIND. ME. BUGS.
    

    View Slide

83. http://hypothesis.works/articles/rule-based-stateful-testing/
    @rule(heap=Heaps.filter(bool))
    def pop(self, heap):
    correct = max(list(heap))
    result = heap_pop(heap)
    > assert correct == result
    E AssertionError: assert 1 == 0
    

    View Slide

84. http://hypothesis.works/articles/rule-based-stateful-testing/
    v1 = new_heap()
    push(heap=v1, value=0)
    push(heap=v1, value=1)
    push(heap=v1, value=1)
    v2 = merge(heap2=v1, heap1=v1)
    pop(heap=v2)
    pop(heap=v2)
    

    View Slide

85. View Slide

86. Property Based Testing
    ● Describe the arguments
    ● Describe the result
    ● Have the computer try to
    prove your code wrong
    

    View Slide

87. Your Turn
    ● Download Hypothesis
    ● Use it
    ● Share how you used it
    ● FIND MORE PATTERNS
    

    View Slide

88. More about Hypothesis
    ● http://hypothesis.works/
    More On Property Based Testing In General
    ● http://www.quviq.com/
    ● https://fsharpforfunandprofit.com/posts/property-based-testing-2/
    Testing Eventual Consistency with RIAK
    ● https://www.youtube.com/watch?v=x9mW54GJpG0
    

    View Slide

89. Thanks!
    Matt Bachmann
    https://github.com/Bachmann1234
    Twitter: @mattbachmann
    Slides: goo.gl/g0mGgU
    

    View Slide