Escape from automanual testing with Hypothesis!

Transcript

1. Escape from automanual testing with Hypothesis! Zac Hatfield-Dodds
2. Your first property-based test a worked example and discussion 2
3. 3 “The sum of a list of integers is greater than the largest element in the list” * of how to test, not what to test! A Simple Example* def test_sum_above_max_small(): xs = [1, 2, 3] assert sum(xs) > max(xs), ... def test_sum_above_max_large(): xs = [10, 20, 30] assert sum(xs) > max(xs), ... Escape from automanual testing with Hypothesis!
4. 4 “The sum of a list of integers is greater than the largest element in the list” A Simple Example import pytest @pytest.mark.parametrize('xs', [ [1, 2, 3], [10, 20, 30], ... ]) def test_sum_above_max(xs): assert sum(xs) > max(xs), ... Escape from automanual testing with Hypothesis!
5. 5 “The sum of a list of integers is greater than the largest element in the list” What do we learn? A Simple Example from hypothesis import given from hypothesis.strategies import lists, integers @given(lists(integers())) def test_sum_above_max(xs): assert sum(xs) > max(xs), ... Escape from automanual testing with Hypothesis!
6. 6 “The sum of a list of integers is greater than the largest element in the list” What do we learn? • Error to call max([]) A Simple Example from hypothesis import given from hypothesis.strategies import lists, integers @given(lists(integers())) def test_sum_above_max(xs): assert sum(xs) > max(xs), ... Falsifying example: test_sum_above_max(xs=[]) Traceback (most recent call last): ... ValueError: max() arg is an empty sequence Escape from automanual testing with Hypothesis!
7. 7 “The sum of a list of integers is greater than the largest element in the list” What do we learn? • Error to call max([]) A Simple Example from hypothesis import given from hypothesis.strategies import lists, integers @given(lists(integers(), min_size=1)) def test_sum_above_max(xs): assert sum(xs) > max(xs), ... Escape from automanual testing with Hypothesis!
8. 8 “The sum of a list of integers is greater than the largest element in the list” What do we learn? • Error to call max([]) • Need greater or equal for 1-lists A Simple Example from hypothesis import given from hypothesis.strategies import lists, integers @given(lists(integers(), min_size=1)) def test_sum_above_max(xs): assert sum(xs) > max(xs), ... Falsifying example: test_sum_above_max(xs=[0]) Traceback (most recent call last): ... AssertionError: xs=[0], sum(xs)=0, max(xs)=0 Escape from automanual testing with Hypothesis!
9. 9 “The sum of a list of integers is greater than the largest element in the list” What do we learn? • Error to call max([]) • Need greater or equal for 1-lists Escape from automanual testing with Hypothesis! A Simple Example from hypothesis import given from hypothesis.strategies import lists, integers @given(lists(integers(), min_size=1)) def test_sum_above_max(xs): assert sum(xs) >= max(xs), ...
10. 10 “The sum of a list of integers is greater than the largest element in the list” What do we learn? • Error to call max([]) • Need greater or equal for 1-lists • Can’t have negative integers A Simple Example from hypothesis import given from hypothesis.strategies import lists, integers @given(lists(integers(), min_size=1)) def test_sum_above_max(xs): assert sum(xs) >= max(xs), ... Falsifying example: test_sum_above_max(xs=[0, -1]) Traceback (most recent call last): ... AssertionError: xs=[0, -1], sum(xs)=-1, max(xs)=0 Escape from automanual testing with Hypothesis!
11. 11 “The sum of a list of integers is greater than the largest element in the list” What do we learn? • Error to call max([]) • Need greater or equal for 1-lists • Can’t have negative integers And this version works! A Simple Example from hypothesis import given from hypothesis.strategies import lists, integers @given(lists(integers(min_value=0), min_size=1)) def test_sum_above_max(xs): assert sum(xs) >= max(xs), ... Escape from automanual testing with Hypothesis!
12. Describing inputs using, defining, and composing strategies 12
13. First-party support for common types • None, bool, numbers, strings, times… • min_value and max_value args • type-specific args floats(allow_infinity=False) times(..., timezones=...) Collections are composed: • From elements and sizes • From keys or indices fancier options are later in the talk 13 See https://hypothesis.readthedocs.io/en/latest/data.html Values and Collections Escape from automanual testing with Hypothesis!
14. 14 .map(f) - applies function f to example - minimises before mapping .filter(f) - retry unless f(ex) truthy - mostly for edge cases Transforming Strategies Escape from automanual testing with Hypothesis! s = integers() s.map(str) # strings of digits s.map(lambda x: x * 2) # even integers s.filter(lambda x: x % 2) # odd ints, slowly # Lists with >= 2 unique numbers lists(s, 2).filter(lambda x: len(set(x)) >=2) See https://hypothesis.readthedocs.io/en/latest/data.html
15. 15 Recursive data • Several ways to recur, e.g.: Interactive data • Run part of a test, then get more input • Useful with complex dataflow Custom strategies • Similar to interactive data in tests Advanced Options strat = deferred( lambda: integers() | lists(strat)) @given(data()) def test_something(data): i = data.draw(integers(...)) @composite def str_and_index(draw, min_size): s = draw(text(min_size=min_size)) i = draw(integers(min_size, len(s)) return (s, i) See https://hypothesis.readthedocs.io/en/latest/data.html Escape from automanual testing with Hypothesis!
16. 16 • Strings matching a regular expression • Function arguments from type hints • Objects from Attrs classes • Values for Numpy datatype • Data from a Django model • Many, many 3rd-party extensions Strong data validation makes testing easier! Inferring Data from Code Escape from automanual testing with Hypothesis!
17. Advanced Usage configuration and performance, and shrinking 17
18. 18 • Verbosity and debug info • How many inputs to test • Cache location • Deterministic mode • and much more! Configure in code or from pytest Define and load custom profiles Configuring Hypothesis from hypothesis import given, settings, database from hypothesis.strategies import none settings.database = database.ExampleDatabase( "/path/to/cache/dir/" # or ":memory:" ) @settings(derandomize=True, max_examples=1000)) @given(...) def test_with_settings(x): ... See https://hypothesis.readthedocs.io/en/latest/settings.html Escape from automanual testing with Hypothesis!
19. Hypothesis is fast! > pytest --hypothesis-show-statistics - mean time per example - % time generating data (~0 to ~20%) - define custom events Hypothesis runs a lot of tests! If your test is slow, running it hundreds of times will be really slow. (but you need fewer tests) Hypothesis runs your code under coverage - ~2x to ~5x slowdown - still improves bugs per unit time - can be disabled in pathological cases 19 Performance tips Escape from automanual testing with Hypothesis!
20. Integrate shrinking with generation! • Start with an arbitrary bytestream • Interpret as value according to strategy • Shrink bytestream, not value • More efficient than shrinking values • Track lots of metadata on targets • Use heuristics for better coverage 20 Secrets of Shrinking Escape from automanual testing with Hypothesis!
21. Integrate shrinking with generation! • Start with an arbitrary bytestream • Interpret as value according to strategy • Shrink bytestream, not value 21 Secrets of Shrinking Escape from automanual testing with Hypothesis! lists(booleans(), min_size=3) stream = [0, 1, 0, 1, 1, 0, ... out = [False, True, False, True]
22. Shrink bitstream in “shortlex” order • Start by deleting later elements • Then reduce bytes • Real shrinker is smarter Let’s shrink out, with any(out) == True 22 Secrets of Shrinking Escape from automanual testing with Hypothesis! lists(booleans(), min_size=3) stream = [0, 1, 0, 1, 1] out = [False, True, False, True]
23. Shrink bitstream in “shortlex” order • Start by deleting later elements • Then reduce bytes • Real shrinker is smarter Let’s shrink out, with any(out) == True 23 Secrets of Shrinking Escape from automanual testing with Hypothesis! lists(booleans(), min_size=3) stream = [0, 1, 0, 1, 0] out = [False, True, False, False]
24. Shrink bitstream in “shortlex” order • Start by deleting later elements • Then reduce bytes • Real shrinker is smarter Let’s shrink out, with any(out) == True 24 Secrets of Shrinking Escape from automanual testing with Hypothesis! lists(booleans(), min_size=3) stream = [0, 1, 0, 0, 0] out = [False, True, False]
25. Shrink bitstream in “shortlex” order • Start by deleting later elements • Then reduce bytes • Real shrinker is smarter Let’s shrink out, with any(out) == True 25 Secrets of Shrinking Escape from automanual testing with Hypothesis! lists(booleans(), min_size=3) stream = [0, 1, 0] out = [False, True, False]
26. Shrink bitstream in “shortlex” order • Start by deleting later elements • Then reduce bytes • Real shrinker is smarter Let’s shrink out, with any(out) == True 26 Secrets of Shrinking Escape from automanual testing with Hypothesis! lists(booleans(), min_size=3) stream = [0, 1, 1] out = [False, True, True]
27. Shrink bitstream in “shortlex” order • Start by deleting later elements • Then reduce bytes • Real shrinker is smarter Let’s shrink out, with any(out) == True 27 Secrets of Shrinking Escape from automanual testing with Hypothesis! lists(booleans(), min_size=3) stream = [0, 0, 1] out = [False, False, True]
28. 28 - Mozilla Public License - Contributors mentored! - Training available - Sponsor new features Q&A time Escape from automanual testing with Hypothesis! Zac Hatfield-Dodds
29. Choosing Properties 29
30. 30 The Test Oracle Compare to another version: new_hotness(x) == legacy(x) fancy_algorithm(x) == brute_force(x) foo(x, threads=10) == foo(x, threads=1) Escape from automanual testing with Hypothesis! See https://fsharpforfunandprofit.com/posts/property-based-testing-2/
31. 31 There and back again “inverse functions” • add / subtract • json.dumps / json.loads or just related: • set_x / get_x • list.append / list.index Escape from automanual testing with Hypothesis! See https://fsharpforfunandprofit.com/posts/property-based-testing-2/
32. 32 Hard to prove, easy to verify Find prime factors, then… • multiply factors to get input Tokenise a string, then… • check each token is valid • concatenate to get input Escape from automanual testing with Hypothesis! See https://fsharpforfunandprofit.com/posts/property-based-testing-2/
33. 33 Some things never change aka “invariants” e.g. sorting a list does not change the element counts (half of the definition of sorting!) Escape from automanual testing with Hypothesis! See https://fsharpforfunandprofit.com/posts/property-based-testing-2/
34. 34 You can stop now “idempotence” • set(x) == set(set(x)) Good for filtering, databases, etc. Escape from automanual testing with Hypothesis! See https://fsharpforfunandprofit.com/posts/property-based-testing-2/
35. 35 - Mozilla Public License - Contributors mentored! - Training available - Sponsor new features Final Q&A time Escape from automanual testing with Hypothesis! Zac Hatfield-Dodds