Parametrized Testing

Multiplying Your Testing Effectiveness with Parametrized Testing.
Exploring 3 methods to create parametrized tests with pytest.
Also when and why to use each method.

Transcript

1. Multiply your Testing E ectiveness with Parameterized Testing Brian Okken @brianokken Code and markdown for slides github.com/okken/presentation_parametrization Slides on slideshare slideshare.net/testandcode 1 / 25
2. Work Podcasts Book Brian Okken 2 / 25
3. Outline Value of Tests Parametrize vs Parameterize Examples: without Parametrization Function Parametrization Fixture Parametrization pytest_generate_tests() Choosing a Technique Combining Techniques Resources 3 / 25
4. Value of Tests Automated tests give us confidence. What does confidence look like? 4 / 25
5. Value of Tests Automated tests give us confidence. What does confidence look like? A passing test suite means: I didn't break anything that used to work. Future changes won’t break current features. The code is ready for users. I can refactor until I'm proud of the code. Code reviews can focus on team understanding and ownership. Only works if: New features are tested with new tests. Tests are easy and fast to write. 5 / 25
6. Parametrize vs Parameterize parameter + ize parameterize (US) parametrize (UK) pytest uses parametrize, the UK spelling. I've tried to get them to change it. They don't want to. I've gotten over it. 6 / 25
7. Something to Test triangles.py: def triangle_type(a, b, c): """ Given three angles, return 'obtuse', 'acute', 'right', or 'invalid'. """ angles = (a, b, c) if (sum(angles) == 180 and all([0 < a < 180 for a in angles])): if any([a > 90 for a in angles]): return "obtuse" if all([a < 90 for a in angles]): return "acute" if 90 in angles: return "right" return "invalid" Obtuse , Acute , Right 7 / 25
8. without Parametrization test_1_without_param.py: from triangle import triangle_type def test_obtuse(): assert triangle_type(100, 40, 40) == "obtuse" def test_acute(): assert triangle_type(60, 60, 60) == "acute" def test_right(): assert triangle_type(90, 60, 30) == "right" def test_invalid(): assert triangle_type(0, 0, 0) == "invalid" Obtuse , Acute , Right 8 / 25
9. Function Parametrization test_2_func_param.py: import pytest from triangle import triangle_type many_triangles = [ (100, 40, 40, "obtuse"), ( 60, 60, 60, "acute"), ( 90, 60, 30, "right"), ( 0, 0, 0, "invalid"), ] @pytest.mark.parametrize('a, b, c, expected', many_triangles) def test_type(a, b, c, expected): assert triangle_type(a, b, c) == expected 9 / 25
10. output without (venv) $ pytest -v test_1_without_param.py ===================== test session starts ====================== collected 4 items test_1_without_param.py::test_obtuse PASSED [ 25%] test_1_without_param.py::test_acute PASSED [ 50%] test_1_without_param.py::test_right PASSED [ 75%] test_1_without_param.py::test_invalid PASSED [100%] ====================== 4 passed in 0.02s ======================= 10 / 25
11. output with (venv) $ pytest -v test_2_func_param.py ===================== test session starts ====================== collected 4 items test_2_func_param.py::test_type[100-40-40-obtuse] PASSED [ 25%] test_2_func_param.py::test_type[60-60-60-acute] PASSED [ 50%] test_2_func_param.py::test_type[90-60-30-right] PASSED [ 75%] test_2_func_param.py::test_type[0-0-0-invalid] PASSED [100%] ====================== 4 passed in 0.02s ======================= 11 / 25
12. You can still run just one (venv) $ pytest -v 'test_2_func_param.py::test_type[0-0-0-invalid]' ===================== test session starts ====================== collected 1 item test_2_func_param.py::test_type[0-0-0-invalid] PASSED [100%] ====================== 1 passed in 0.02s ======================= This uses the node id. 12 / 25
13. or more (venv) $ pytest -v -k 60 test_2_func_param.py ===================== test session starts ====================== collected 4 items / 2 deselected / 2 selected test_2_func_param.py::test_type[60-60-60-acute] PASSED [ 50%] test_2_func_param.py::test_type[90-60-30-right] PASSED [100%] =============== 2 passed, 2 deselected in 0.02s ================ An example with -k to pick all tests with 60 degree angles. 13 / 25
14. Once again, this is Function Parametrization test_2_func_param.py: import pytest from triangle import triangle_type many_triangles = [ (100, 40, 40, "obtuse"), ( 60, 60, 60, "acute"), ( 90, 60, 30, "right"), ( 0, 0, 0, "invalid"), ] @pytest.mark.parametrize('a, b, c, expected', many_triangles) def test_type(a, b, c, expected): assert triangle_type(a, b, c) == expected 14 / 25
15. Fixture Parametrization test_3_fixture_param.py: import pytest from triangle import triangle_type many_triangles = [ (100, 40, 40, "obtuse"), ( 60, 60, 60, "acute"), ( 90, 60, 30, "right"), ( 0, 0, 0, "invalid"), ] @pytest.fixture(params=many_triangles) def a_triangle(request): return request.param def test_type(a_triangle): a, b, c, expected = a_triangle assert triangle_type(a, b, c) == expected 15 / 25
16. output (venv) $ pytest -v test_3_fixture_param.py ======================= test session starts ======================= collected 4 items test_3_fixture_param.py::test_type[a_triangle0] PASSED [ 25%] test_3_fixture_param.py::test_type[a_triangle1] PASSED [ 50%] test_3_fixture_param.py::test_type[a_triangle2] PASSED [ 75%] test_3_fixture_param.py::test_type[a_triangle3] PASSED [100%] ======================== 4 passed in 0.02s ======================== 16 / 25
17. adding an id function test_4_fixture_param.py: import pytest from triangle import triangle_type many_triangles = [ (100, 40, 40, "obtuse"), ( 60, 60, 60, "acute"), ( 90, 60, 30, "right"), ( 0, 0, 0, "invalid"), ] def idfn(a_triangle): a, b, c, expected = a_triangle return f'{a}_{b}_{c}_{expected}' @pytest.fixture(params=many_triangles, ids=idfn) def a_triangle(request): return request.param def test_type(a_triangle): a, b, c, expected = a_triangle assert triangle_type(a, b, c) == expected 17 / 25
18. output (venv) $ pytest -v test_4_fixture_param.py ======================= test session starts ======================= collected 4 items test_4_fixture_param.py::test_type[100_40_40_obtuse] PASSED [ 25%] test_4_fixture_param.py::test_type[60_60_60_acute] PASSED [ 50%] test_4_fixture_param.py::test_type[90_60_30_right] PASSED [ 75%] test_4_fixture_param.py::test_type[0_0_0_invalid] PASSED [100%] ======================== 4 passed in 0.02s ======================== 18 / 25
19. pytest_generate_tests() test_5_gen.py: from triangle import triangle_type many_triangles = [ (100, 40, 40, "obtuse"), ( 60, 60, 60, "acute"), ( 90, 60, 30, "right"), ( 0, 0, 0, "invalid"), ] def idfn(a_triangle): a, b, c, expected = a_triangle return f'{a}_{b}_{c}_{expected}' def pytest_generate_tests(metafunc): if "a_triangle" in metafunc.fixturenames: metafunc.parametrize("a_triangle", many_triangles, ids=idfn) def test_type(a_triangle): a, b, c, expected = a_triangle assert triangle_type(a, b, c) == expected 19 / 25
20. output (venv) $ pytest -v test_5_gen.py =================== test session starts ==================== platform darwin -- Python 3.7.3, pytest-5.2.1, py-1.8.0, pluggy-0.13.0 -- /Users/o cachedir: .pytest_cache rootdir: /Users/okken/projects/presentation_parametrization/code collected 4 items test_5_gen.py::test_type[100_40_40_obtuse] PASSED [ 25%] test_5_gen.py::test_type[60_60_60_acute] PASSED [ 50%] test_5_gen.py::test_type[90_60_30_right] PASSED [ 75%] test_5_gen.py::test_type[0_0_0_invalid] PASSED [100%] ==================== 4 passed in 0.03s ===================== 20 / 25
21. Choosing a Technique Guidelines 1. function parametrization use this if you can 2. fixture parametrization if doing work to set up each fixture value if cycling through pre-conditions if running multiple test against the same set of "setup states" 3. pytest_generate_tests() if you need to build up the list at runtime if list is based on passed in parameters or external resources for sparse matrix sets 21 / 25
22. more test cases test_6_more.py: ... triangles = [ ( 1, 1, 178, "obtuse"), # big angles ( 91, 44, 45, "obtuse"), # just over 90 (0.01, 0.01, 179.98, "obtuse"), # decimals (90, 60, 30, "right"), # check 90 for each angle (10, 90, 80, "right"), (85, 5, 90, "right"), (89, 89, 2, "acute"), # just under 90 (60, 60, 60, "acute"), (0, 0, 0, "invalid"), # zeros (61, 60, 60, "invalid"), # sum > 180 (90, 91, -1, "invalid"), # negative numbers ] @pytest.mark.parametrize('a, b, c, expected', triangles) def test_type(a, b, c, expected): assert triangle_type(a, b, c) == expected 22 / 25
23. Combining Techniques You can have multiple parametrizations for a test function. can have multiple @pytest.mark.parametrize() decorators on a test function. can parameterize multipe fixtures per test can use pytest_generate_tests() to parametrize multiple parameters can use a combination of techniques can blow up into lots and lots of test cases very fast 23 / 25
24. Not covered, intentionally Use with caution. indirect Have a list of parameters defined at the test function that gets passed to a fixture. Kind of a hybrid between function and fixture parametrization. subtests Sort of related, but not really. You can check multiple things within a test. If you care about test case counts, pass, fail, etc, then don't use subtests. 24 / 25
25. Python Testing with pytest The fastest way to get super productive with pytest pytest docs on parametrization, in general function parametrization fixture parametrization pytest_generate_tests podcasts Test & Code Python Bytes Talk Python slack community: Test & Code Slack Twitter: @brianokken, @testandcode This code, and markdown for slides, on github Presentation is at slideshare.net/testandcode Resources 25 / 25