$30 off During Our Annual Pro Sale. View Details »

Matt Bachmann - Better Testing With Less Code: Property Based Testing With Python

Matt Bachmann - Better Testing With Less Code: Property Based Testing With Python

Standard unit tests have developers test specific inputs and outputs. This works, but often what breaks code are the cases we did not think about. Property based testing has developers define properties of output and has the computer explore the possible inputs to verify these properties. This talk will introduce property based testing and provide real world examples and patterns.

https://us.pycon.org/2016/schedule/presentation/1927/

PyCon 2016

May 29, 2016
Tweet

More Decks by PyCon 2016

Other Decks in Programming

Transcript

  1. Property-Based
    Testing In
    Python
    Matt Bachmann @mattbachmann

    View Slide

  2. Testing is Important
    Refactoring
    Design
    Regression Protection
    Faster Development

    View Slide

  3. Testing is Hard
    Code
    Isolation
    Fixtures
    Indirect Value

    View Slide

  4. Capture the
    Important Cases
    Minimize The
    Coding Overhead

    View Slide

  5. Sorting a list of
    integers

    View Slide

  6. def test_sort_empty(xs):
    assert quicksort([]) == []
    def test_sorted(xs):
    assert quicksort([1, 2, 3]) == [1, 2, 3]
    def test_sort_unsorted(xs):
    assert quicksort([5, 4]) == [4, 5]

    View Slide

  7. Property Based Testing
    ● Describe the arguments
    ● Describe the result
    ● Have the computer try to
    prove your code wrong

    View Slide

  8. ● The Arguments
    ○ List of Integers
    ● Properties of the Result
    ○ List
    ○ All elements preserved
    ○ Results are in ascending
    order

    View Slide

  9. Hypothesis
    Inspired by Haskell’s
    QuickCheck
    Fantastic Docs
    Offers training/contracting
    http://hypothesis.works/

    View Slide

  10. @given(st.lists(st.integers()))
    def test_sort(xs):
    sorted_xs = quicksort(xs)
    assert isinstance(sorted_xs, list)
    assert Counter(xs) == Counter(sorted_xs)
    assert all(
    x <= y for x, y in
    zip(sorted_xs, sorted_xs[1:])
    )

    View Slide

  11. @given(st.lists(st.integers()))
    def test_sort(xs):
    sorted_xs = quicksort(xs)
    assert isinstance(sorted_xs, list)
    assert Counter(xs) == Counter(sorted_xs)
    assert all(
    x <= y for x, y in
    zip(sorted_xs, sorted_xs[1:])
    )

    View Slide

  12. @given(st.lists(st.integers()))
    def test_sort(xs):
    sorted_xs = quicksort(xs)
    assert isinstance(sorted_xs, list)
    assert Counter(xs) == Counter(sorted_xs)
    assert all(
    x <= y for x, y in
    zip(sorted_xs, sorted_xs[1:])
    )

    View Slide

  13. @given(st.lists(st.integers()))
    def test_sort(xs):
    sorted_xs = quicksort(xs)
    assert isinstance(sorted_xs, list)
    assert Counter(xs) == Counter(sorted_xs)
    assert all(
    x <= y for x, y in
    zip(sorted_xs, sorted_xs[1:])
    )

    View Slide

  14. @given(st.lists(st.integers()))
    def test_sort(xs):
    sorted_xs = quicksort(xs)
    assert isinstance(sorted_xs, list)
    assert Counter(xs) == Counter(sorted_xs)
    assert all(
    x <= y for x, y in
    zip(sorted_xs, sorted_xs[1:])
    )

    View Slide

  15. @given(st.lists(st.integers()))
    def test_sort(xs):
    sorted_xs = quicksort(xs)
    assert isinstance(sorted_xs, list)
    assert Counter(xs) == Counter(sorted_xs)
    assert all(
    x <= y for x, y in
    zip(sorted_xs, sorted_xs[1:])
    )

    View Slide

  16. @given(st.lists(st.integers()))
    def test_sort(xs):
    sorted_xs = quicksort(xs)
    assert isinstance(sorted_xs, list)
    assert Counter(xs) == Counter(sorted_xs)
    assert all(
    x <= y for x, y in
    zip(sorted_xs, sorted_xs[1:])
    )

    View Slide

  17. @given(st.lists(st.integers()))
    test_sort([])
    test_sort([0])
    test_sort([93932932923, 82883982983838])
    test_sort([9999,77,2,3,3,100,3,39,3993])

    View Slide

  18. View Slide

  19. [-3223, -99999999999, 32]

    View Slide

  20. [-3223, -99999999999, 32]
    [-3223, -99999999999]

    View Slide

  21. [-3223, -99999999999, 32]
    [-3223, -99999999999]
    [1, 0]

    View Slide

  22. @given

    View Slide

  23. Strategies
    booleans, floats, strings, complex_numbers

    View Slide

  24. Strategies
    booleans, floats, strings, complex_numbers
    dictionaries, tuples, lists, sets, builds

    View Slide

  25. Strategies
    booleans, floats, strings, complex_numbers
    dictionaries, tuples, lists, sets, builds
    one_of, sampled_from, recursive

    View Slide

  26. @given(
    st.builds(
    Dog,
    breed=st.text(),
    name=st.text(),
    height=st.floats(),
    weight=st.floats()
    )
    )

    View Slide

  27. @given(
    st.builds(
    Dog,
    breed=st.sampled_from(KNOWN_BREEDS),
    name=st.text(),
    height=st.floats(),
    weight=st.floats()
    )
    )

    View Slide

  28. @given(
    st.builds(
    Dog,
    breed=st.sampled_from(KNOWN_BREEDS),
    name=st.text(min_size=5),
    height=st.floats(),
    weight=st.floats()
    )
    )

    View Slide

  29. @given(
    st.builds(
    Dog,
    breed=st.sampled_from(KNOWN_BREEDS),
    name=st.text(min_size=5),
    height=st.floats(
    min_value=1, max_value=6, allow_nan=False, allow_infinity=False
    ),
    weight=st.floats(
    min_value=1, max_value=300, allow_nan=False, allow_infinity=False
    ),
    )
    )

    View Slide

  30. @given(
    st.builds(
    Dog,
    breed=st.sampled_from(KNOWN_BREEDS),
    name=st.text(min_size=5),
    height=st.floats(
    min_value=1, max_value=6, allow_nan=False, allow_infinity=False
    ),
    weight=st.floats(
    min_value=1, max_value=300, allow_nan=False, allow_infinity=False
    ),
    )
    )
    @example(Dog(breed="Labrador", name="Spot", height=2.1, weight=70))

    View Slide

  31. Potentially infinite cases
    Nothing hardcoded
    Very little code

    View Slide

  32. https://www.flickr.com/photos/t0fugurl/2507049701
    Picture (without crossout) by Leanne Poon
    https://creativecommons.org/licenses/by-nc-nd/2.0/

    View Slide

  33. Pattern 1:
    The code should not
    explode

    View Slide

  34. https://upload.wikimedia.
    org/wikipedia/commons/3/39/Chip-pan-
    fire.jpg

    View Slide

  35. /batputer/criminals/aliases/{id}?
    sort={sort}&max_results={max}
    ● JSON response
    ● Expected response codes
    ○ 200 - OK
    ○ 401 - Forbidden
    ○ 400 - Invalid data
    ○ 404 - Not Found

    View Slide


  36. View Slide

  37. https://upload.wikimedia.
    org/wikipedia/commons/3/39/Chip-pan-
    fire.jpg

    View Slide

  38. @given(st.integers(),
    st.text(),
    st.integers()))
    def test_no_explosion(id, sort, max):
    response = requests.get(
    BATPUTER_URL.format(id, sort, max)
    )
    assert response and response.json()
    assert (response.status_code in
    [200, 401, 400, 404])

    View Slide

  39. @given(st.integers(),
    st.text(),
    st.integers()))
    def test_no_explosion(id, sort, max):
    response = requests.get(
    BATPUTER_URL.format(id, sort, max)
    )
    assert response and response.json()
    assert (response.status_code in
    [200, 401, 400, 404])

    View Slide

  40. @given(st.integers(),
    st.text(),
    st.integers()))
    def test_no_explosion(id, sort, max):
    response = requests.get(
    BATPUTER_URL.format(id, sort, max)
    )
    assert response and response.json()
    assert (response.status_code in
    [200, 401, 400, 404])

    View Slide

  41. @given(st.integers(),
    st.text(),
    st.integers()))
    def test_no_explosion(id, sort, max):
    response = requests.get(
    BATPUTER_URL.format(id, sort, max)
    )
    assert response and response.json()
    assert (response.status_code in
    [200, 401, 400, 404])

    View Slide

  42. Pattern 2:
    Reversible
    Operations

    View Slide

  43. Encoding
    Undo Operations
    Serialization

    View Slide

  44. Encoding
    Undo Operations
    Serialization

    View Slide

  45. class HistoricalEvent(object):
    def __init__(self, id, desc, time):
    self.id = id
    self.desc = desc
    self.time = time

    View Slide

  46. class EventEncoder(json.JSONEncoder):
    def default(self, obj):
    if isinstance(obj, HistoricalEvent):
    return {
    "id": obj.id
    "description": obj.description
    "event_time":obj.event_time.isoformat()
    }
    return json.JSONEncoder.default(self, obj)

    View Slide

  47. def fromJson(json_str):
    dct = json.loads(json_str)
    return HistoricalEvent(
    dct['id'],
    dct['description'],
    dateutil.parser.parse(
    dct['event_time']
    )
    )

    View Slide

  48. View Slide

  49. @given(st.integers(), st.text(), datetimes(timezones=['UTC']))
    def test_to_from_json(id, desc, date):
    original_event = HistoricalEvent(id, desc, date)
    encoded_event = json.dumps(event, cls=EventEncoder)
    decoded_event = json.loads(
    encoded_event,
    object_hook=HistoricalEvent.fromJson
    )
    assert decoded_event == original_event

    View Slide

  50. @given(st.integers(), st.text(), datetimes(timezones=['UTC']))
    def test_to_from_json(id, desc, date):
    original_event = HistoricalEvent(id, desc, date)
    encoded_event = json.dumps(event, cls=EventEncoder)
    decoded_event = json.loads(
    encoded_event,
    object_hook=HistoricalEvent.fromJson
    )
    assert decoded_event == original_event

    View Slide

  51. @given(st.integers(), st.text(), datetimes(timezones=['UTC']))
    def test_to_from_json(id, desc, date):
    original_event = HistoricalEvent(id, desc, date)
    encoded_event = json.dumps(event, cls=EventEncoder)
    decoded_event = json.loads(
    encoded_event,
    object_hook=HistoricalEvent.fromJson
    )
    assert decoded_event == original_event

    View Slide

  52. @given(st.integers(), st.text(), datetimes(timezones=['UTC']))
    def test_to_from_json(id, desc, date):
    original_event = HistoricalEvent(id, desc, date)
    encoded_event = json.dumps(event, cls=EventEncoder)
    decoded_event = json.loads(
    encoded_event,
    object_hook=HistoricalEvent.fromJson
    )
    assert decoded_event == original_event

    View Slide

  53. @given(st.integers(), st.text(), datetimes(timezones=['UTC']))
    def test_to_from_json(id, desc, date):
    original_event = HistoricalEvent(id, desc, date)
    encoded_event = json.dumps(event, cls=EventEncoder)
    decoded_event = json.loads(
    encoded_event,
    object_hook=HistoricalEvent.fromJson
    )
    assert decoded_event == original_event

    View Slide

  54. View Slide

  55. Pattern 3:
    Testing Oracle

    View Slide

  56. Optimizing
    Refactoring
    Emulating

    View Slide

  57. https://www.flickr.com/photos/versageek/493800514
    versageek
    https://creativecommons.org/licenses/by-sa/2.0/
    Leave It
    Alone

    View Slide

  58. https://www.flickr.com/photos/versageek/493800514
    versageek
    https://creativecommons.org/licenses/by-sa/2.0/
    https://upload.wikimedia.
    org/wikipedia/commons/3/36/Under_Floor_Cable_
    Runs_Tee.jpg
    =

    View Slide

  59. @given(st.integers(), st.text())
    def test_against_legacy(arg1, arg2):
    assert (
    new_hotness(arg1, arg2)
    ==
    legacy_system(arg1, arg2)
    )

    View Slide

  60. Compare
    Against Brute
    Force

    View Slide

  61. @given(st.lists(st.integers()))
    def test_against_brute_force(input):
    assert (
    easy_but_inefficent(input)
    ==
    optimized(input)
    )

    View Slide

  62. Pattern 4:
    Stateful Testing

    View Slide

  63. View Slide

  64. View Slide

  65. View Slide

  66. Stateful Tests
    ● Define a state
    ● What operations can happen in
    what conditions?
    ● How do operations affect the
    state?
    ● What must be true for each step?

    View Slide

  67. https://upload.wikimedia.org/wikipedia/commons/8/83/Jan_Fabre's_%22Searching_for_Utopia%22.jpg
    FIND. ME. BUGS.

    View Slide

  68. https://commons.wikimedia.org/wiki/File:Max-Heap.svg
    http://hypothesis.works/articles/rule-based-stateful-testing/

    View Slide

  69. http://hypothesis.works/articles/rule-based-stateful-testing/
    __init__
    push
    pop
    merge

    View Slide

  70. http://hypothesis.works/articles/rule-based-stateful-testing/
    Integers
    Heaps

    View Slide

  71. http://hypothesis.works/articles/rule-based-stateful-testing/
    __init__ Integers
    Heaps

    View Slide

  72. http://hypothesis.works/articles/rule-based-stateful-testing/
    push Integers
    Heaps

    View Slide

  73. http://hypothesis.works/articles/rule-based-stateful-testing/
    merge Integers
    Heaps

    View Slide

  74. http://hypothesis.works/articles/rule-based-stateful-testing/
    merge Integers
    Heaps

    View Slide

  75. http://hypothesis.works/articles/rule-based-stateful-testing/
    pop Integers
    Heaps
    assert result actually max

    View Slide

  76. http://hypothesis.works/articles/rule-based-stateful-testing/
    class HeapMachine(RuleBasedStateMachine):
    Heaps = Bundle('heaps')
    @rule(target=Heaps)
    def new_heap(self):
    return Heap()
    @rule(heap=Heaps, value=integers())
    def heap_push(self, heap, value):
    push(heap, value)

    View Slide

  77. http://hypothesis.works/articles/rule-based-stateful-testing/
    class HeapMachine(RuleBasedStateMachine):
    Heaps = Bundle('heaps')
    @rule(target=Heaps)
    def new_heap(self):
    return Heap()
    @rule(heap=Heaps, value=integers())
    def heap_push(self, heap, value):
    push(heap, value)

    View Slide

  78. class HeapMachine(RuleBasedStateMachine):
    Heaps = Bundle('heaps')
    @rule(target=Heaps)
    def new_heap(self):
    return Heap()
    @rule(heap=Heaps, value=integers())
    def heap_push(self, heap, value):
    push(heap, value)
    http://hypothesis.works/articles/rule-based-stateful-testing/

    View Slide

  79. class HeapMachine(RuleBasedStateMachine):
    Heaps = Bundle('heaps')
    @rule(target=Heaps)
    def new_heap(self):
    return Heap()
    @rule(heap=Heaps, value=integers())
    def heap_push(self, heap, value):
    push(heap, value)
    http://hypothesis.works/articles/rule-based-stateful-testing/

    View Slide

  80. http://hypothesis.works/articles/rule-based-stateful-testing/

    @rule(target=Heaps, heap1=Heaps, heap2=Heaps)
    def merge(self, heap1, heap2):
    return heap_merge(heap1, heap2)
    @rule(heap=Heaps.filter(bool))
    def pop(self, heap):
    correct = max(list(heap))
    result = heap_pop(heap)
    assert correct == result

    View Slide

  81. http://hypothesis.works/articles/rule-based-stateful-testing/

    @rule(target=Heaps, heap1=Heaps, heap2=Heaps)
    def merge(self, heap1, heap2):
    return heap_merge(heap1, heap2)
    @rule(heap=Heaps.filter(bool))
    def pop(self, heap):
    correct = max(list(heap))
    result = heap_pop(heap)
    assert correct == result

    View Slide

  82. https://upload.wikimedia.org/wikipedia/commons/8/83/Jan_Fabre's_%22Searching_for_Utopia%22.jpg
    FIND. ME. BUGS.

    View Slide

  83. http://hypothesis.works/articles/rule-based-stateful-testing/
    @rule(heap=Heaps.filter(bool))
    def pop(self, heap):
    correct = max(list(heap))
    result = heap_pop(heap)
    > assert correct == result
    E AssertionError: assert 1 == 0

    View Slide

  84. http://hypothesis.works/articles/rule-based-stateful-testing/
    v1 = new_heap()
    push(heap=v1, value=0)
    push(heap=v1, value=1)
    push(heap=v1, value=1)
    v2 = merge(heap2=v1, heap1=v1)
    pop(heap=v2)
    pop(heap=v2)

    View Slide

  85. View Slide

  86. Property Based Testing
    ● Describe the arguments
    ● Describe the result
    ● Have the computer try to
    prove your code wrong

    View Slide

  87. Your Turn
    ● Download Hypothesis
    ● Use it
    ● Share how you used it
    ● FIND MORE PATTERNS

    View Slide

  88. More about Hypothesis
    ● http://hypothesis.works/
    More On Property Based Testing In General
    ● http://www.quviq.com/
    ● https://fsharpforfunandprofit.com/posts/property-based-testing-2/
    Testing Eventual Consistency with RIAK
    ● https://www.youtube.com/watch?v=x9mW54GJpG0

    View Slide

  89. Thanks!
    Matt Bachmann
    https://github.com/Bachmann1234
    Twitter: @mattbachmann
    Slides: goo.gl/g0mGgU

    View Slide