Escape from automanual testing with Hypothesis! Zac Hatfield-Dodds 

Your first property-based test a worked example and discussion 2 

3 “The sum of a list of integers is greater than the largest element in the list” * of how to test, not what to test! A Simple Example* def test_sum_above_max_small(): xs = [1, 2, 3] assert sum(xs) > max(xs), ... def test_sum_above_max_large(): xs = [10, 20, 30] assert sum(xs) > max(xs), ... Escape from automanual testing with Hypothesis! 

4 “The sum of a list of integers is greater than the largest element in the list” A Simple Example import pytest @pytest.mark.parametrize('xs', [ [1, 2, 3], [10, 20, 30], ... ]) def test_sum_above_max(xs): assert sum(xs) > max(xs), ... Escape from automanual testing with Hypothesis! 

5 “The sum of a list of integers is greater than the largest element in the list” What do we learn? A Simple Example from hypothesis import given from hypothesis.strategies import lists, integers @given(lists(integers())) def test_sum_above_max(xs): assert sum(xs) > max(xs), ... Escape from automanual testing with Hypothesis! 

6 “The sum of a list of integers is greater than the largest element in the list” What do we learn? • Error to call max([]) A Simple Example from hypothesis import given from hypothesis.strategies import lists, integers @given(lists(integers())) def test_sum_above_max(xs): assert sum(xs) > max(xs), ... Falsifying example: test_sum_above_max(xs=[]) Traceback (most recent call last): ... ValueError: max() arg is an empty sequence Escape from automanual testing with Hypothesis! 

7 “The sum of a list of integers is greater than the largest element in the list” What do we learn? • Error to call max([]) A Simple Example from hypothesis import given from hypothesis.strategies import lists, integers @given(lists(integers(), min_size=1)) def test_sum_above_max(xs): assert sum(xs) > max(xs), ... Escape from automanual testing with Hypothesis! 

8 “The sum of a list of integers is greater than the largest element in the list” What do we learn? • Error to call max([]) • Need greater or equal for 1-lists A Simple Example from hypothesis import given from hypothesis.strategies import lists, integers @given(lists(integers(), min_size=1)) def test_sum_above_max(xs): assert sum(xs) > max(xs), ... Falsifying example: test_sum_above_max(xs=[0]) Traceback (most recent call last): ... AssertionError: xs=[0], sum(xs)=0, max(xs)=0 Escape from automanual testing with Hypothesis! 

9 “The sum of a list of integers is greater than the largest element in the list” What do we learn? • Error to call max([]) • Need greater or equal for 1-lists Escape from automanual testing with Hypothesis! A Simple Example from hypothesis import given from hypothesis.strategies import lists, integers @given(lists(integers(), min_size=1)) def test_sum_above_max(xs): assert sum(xs) >= max(xs), ... 

10 “The sum of a list of integers is greater than the largest element in the list” What do we learn? • Error to call max([]) • Need greater or equal for 1-lists • Can’t have negative integers A Simple Example from hypothesis import given from hypothesis.strategies import lists, integers @given(lists(integers(), min_size=1)) def test_sum_above_max(xs): assert sum(xs) >= max(xs), ... Falsifying example: test_sum_above_max(xs=[0, -1]) Traceback (most recent call last): ... AssertionError: xs=[0, -1], sum(xs)=-1, max(xs)=0 Escape from automanual testing with Hypothesis! 

11 “The sum of a list of integers is greater than the largest element in the list” What do we learn? • Error to call max([]) • Need greater or equal for 1-lists • Can’t have negative integers And this version works! A Simple Example from hypothesis import given from hypothesis.strategies import lists, integers @given(lists(integers(min_value=0), min_size=1)) def test_sum_above_max(xs): assert sum(xs) >= max(xs), ... Escape from automanual testing with Hypothesis! 

Describing inputs using, defining, and composing strategies 12 

First-party support for common types • None, bool, numbers, strings, times… • min_value and max_value args • type-specific args floats(allow_infinity=False) times(..., timezones=...) Collections are composed: • From elements and sizes • From keys or indices fancier options are later in the talk 13 See https://hypothesis.readthedocs.io/en/latest/data.html Values and Collections Escape from automanual testing with Hypothesis! 

14 .map(f) - applies function f to example - minimises before mapping .filter(f) - retry unless f(ex) truthy - mostly for edge cases Transforming Strategies Escape from automanual testing with Hypothesis! s = integers() s.map(str) # strings of digits s.map(lambda x: x * 2) # even integers s.filter(lambda x: x % 2) # odd ints, slowly # Lists with >= 2 unique numbers lists(s, 2).filter(lambda x: len(set(x)) >=2) See https://hypothesis.readthedocs.io/en/latest/data.html 

15 Recursive data • Several ways to recur, e.g.: Interactive data • Run part of a test, then get more input • Useful with complex dataflow Custom strategies • Similar to interactive data in tests Advanced Options strat = deferred( lambda: integers() | lists(strat)) @given(data()) def test_something(data): i = data.draw(integers(...)) @composite def str_and_index(draw, min_size): s = draw(text(min_size=min_size)) i = draw(integers(min_size, len(s)) return (s, i) See https://hypothesis.readthedocs.io/en/latest/data.html Escape from automanual testing with Hypothesis! 

16 • Strings matching a regular expression • Function arguments from type hints • Objects from Attrs classes • Values for Numpy datatype • Data from a Django model • Many, many 3rd-party extensions Strong data validation makes testing easier! Inferring Data from Code Escape from automanual testing with Hypothesis! 

Advanced Usage configuration and performance, and shrinking 17 

18 • Verbosity and debug info • How many inputs to test • Cache location • Deterministic mode • and much more! Configure in code or from pytest Define and load custom profiles Configuring Hypothesis from hypothesis import given, settings, database from hypothesis.strategies import none settings.database = database.ExampleDatabase( "/path/to/cache/dir/" # or ":memory:" ) @settings(derandomize=True, max_examples=1000)) @given(...) def test_with_settings(x): ... See https://hypothesis.readthedocs.io/en/latest/settings.html Escape from automanual testing with Hypothesis! 

Hypothesis is fast! > pytest --hypothesis-show-statistics - mean time per example - % time generating data (~0 to ~20%) - define custom events Hypothesis runs a lot of tests! If your test is slow, running it hundreds of times will be really slow. (but you need fewer tests) Hypothesis runs your code under coverage - ~2x to ~5x slowdown - still improves bugs per unit time - can be disabled in pathological cases 19 Performance tips Escape from automanual testing with Hypothesis! 

Integrate shrinking with generation! • Start with an arbitrary bytestream • Interpret as value according to strategy • Shrink bytestream, not value • More efficient than shrinking values • Track lots of metadata on targets • Use heuristics for better coverage 20 Secrets of Shrinking Escape from automanual testing with Hypothesis! 

Integrate shrinking with generation! • Start with an arbitrary bytestream • Interpret as value according to strategy • Shrink bytestream, not value 21 Secrets of Shrinking Escape from automanual testing with Hypothesis! lists(booleans(), min_size=3) stream = [0, 1, 0, 1, 1, 0, ... out = [False, True, False, True] 

Shrink bitstream in “shortlex” order • Start by deleting later elements • Then reduce bytes • Real shrinker is smarter Let’s shrink out, with any(out) == True 22 Secrets of Shrinking Escape from automanual testing with Hypothesis! lists(booleans(), min_size=3) stream = [0, 1, 0, 1, 1] out = [False, True, False, True] 

Shrink bitstream in “shortlex” order • Start by deleting later elements • Then reduce bytes • Real shrinker is smarter Let’s shrink out, with any(out) == True 23 Secrets of Shrinking Escape from automanual testing with Hypothesis! lists(booleans(), min_size=3) stream = [0, 1, 0, 1, 0] out = [False, True, False, False] 

Shrink bitstream in “shortlex” order • Start by deleting later elements • Then reduce bytes • Real shrinker is smarter Let’s shrink out, with any(out) == True 24 Secrets of Shrinking Escape from automanual testing with Hypothesis! lists(booleans(), min_size=3) stream = [0, 1, 0, 0, 0] out = [False, True, False] 

Shrink bitstream in “shortlex” order • Start by deleting later elements • Then reduce bytes • Real shrinker is smarter Let’s shrink out, with any(out) == True 25 Secrets of Shrinking Escape from automanual testing with Hypothesis! lists(booleans(), min_size=3) stream = [0, 1, 0] out = [False, True, False] 

Shrink bitstream in “shortlex” order • Start by deleting later elements • Then reduce bytes • Real shrinker is smarter Let’s shrink out, with any(out) == True 26 Secrets of Shrinking Escape from automanual testing with Hypothesis! lists(booleans(), min_size=3) stream = [0, 1, 1] out = [False, True, True] 

Shrink bitstream in “shortlex” order • Start by deleting later elements • Then reduce bytes • Real shrinker is smarter Let’s shrink out, with any(out) == True 27 Secrets of Shrinking Escape from automanual testing with Hypothesis! lists(booleans(), min_size=3) stream = [0, 0, 1] out = [False, False, True] 

28 - Mozilla Public License - Contributors mentored! - Training available - Sponsor new features Q&A time Escape from automanual testing with Hypothesis! Zac Hatfield-Dodds 

Choosing Properties 29 

30 The Test Oracle Compare to another version: new_hotness(x) == legacy(x) fancy_algorithm(x) == brute_force(x) foo(x, threads=10) == foo(x, threads=1) Escape from automanual testing with Hypothesis! See https://fsharpforfunandprofit.com/posts/property-based-testing-2/ 

31 There and back again “inverse functions” • add / subtract • json.dumps / json.loads or just related: • set_x / get_x • list.append / list.index Escape from automanual testing with Hypothesis! See https://fsharpforfunandprofit.com/posts/property-based-testing-2/ 

32 Hard to prove, easy to verify Find prime factors, then… • multiply factors to get input Tokenise a string, then… • check each token is valid • concatenate to get input Escape from automanual testing with Hypothesis! See https://fsharpforfunandprofit.com/posts/property-based-testing-2/ 

33 Some things never change aka “invariants” e.g. sorting a list does not change the element counts (half of the definition of sorting!) Escape from automanual testing with Hypothesis! See https://fsharpforfunandprofit.com/posts/property-based-testing-2/ 

34 You can stop now “idempotence” • set(x) == set(set(x)) Good for filtering, databases, etc. Escape from automanual testing with Hypothesis! See https://fsharpforfunandprofit.com/posts/property-based-testing-2/ 

35 - Mozilla Public License - Contributors mentored! - Training available - Sponsor new features Final Q&A time Escape from automanual testing with Hypothesis! Zac Hatfield-Dodds