Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Parametrized Testing

Avatar for okken okken
October 08, 2019

Parametrized Testing

Multiplying Your Testing Effectiveness with Parametrized Testing.
Exploring 3 methods to create parametrized tests with pytest.
Also when and why to use each method.

Avatar for okken

okken

October 08, 2019
Tweet

Other Decks in Programming

Transcript

  1. Multiply your Testing E ectiveness with Parameterized Testing Brian Okken

    @brianokken Code and markdown for slides github.com/okken/presentation_parametrization Slides on slideshare slideshare.net/testandcode 1 / 25
  2. Outline Value of Tests Parametrize vs Parameterize Examples: without Parametrization

    Function Parametrization Fixture Parametrization pytest_generate_tests() Choosing a Technique Combining Techniques Resources 3 / 25
  3. Value of Tests Automated tests give us confidence. What does

    confidence look like? A passing test suite means: I didn't break anything that used to work. Future changes won’t break current features. The code is ready for users. I can refactor until I'm proud of the code. Code reviews can focus on team understanding and ownership. Only works if: New features are tested with new tests. Tests are easy and fast to write. 5 / 25
  4. Parametrize vs Parameterize parameter + ize parameterize (US) parametrize (UK)

    pytest uses parametrize, the UK spelling. I've tried to get them to change it. They don't want to. I've gotten over it. 6 / 25
  5. Something to Test triangles.py: def triangle_type(a, b, c): """ Given

    three angles, return 'obtuse', 'acute', 'right', or 'invalid'. """ angles = (a, b, c) if (sum(angles) == 180 and all([0 < a < 180 for a in angles])): if any([a > 90 for a in angles]): return "obtuse" if all([a < 90 for a in angles]): return "acute" if 90 in angles: return "right" return "invalid" Obtuse , Acute , Right 7 / 25
  6. without Parametrization test_1_without_param.py: from triangle import triangle_type def test_obtuse(): assert

    triangle_type(100, 40, 40) == "obtuse" def test_acute(): assert triangle_type(60, 60, 60) == "acute" def test_right(): assert triangle_type(90, 60, 30) == "right" def test_invalid(): assert triangle_type(0, 0, 0) == "invalid" Obtuse , Acute , Right 8 / 25
  7. Function Parametrization test_2_func_param.py: import pytest from triangle import triangle_type many_triangles

    = [ (100, 40, 40, "obtuse"), ( 60, 60, 60, "acute"), ( 90, 60, 30, "right"), ( 0, 0, 0, "invalid"), ] @pytest.mark.parametrize('a, b, c, expected', many_triangles) def test_type(a, b, c, expected): assert triangle_type(a, b, c) == expected 9 / 25
  8. output without (venv) $ pytest -v test_1_without_param.py ===================== test session

    starts ====================== collected 4 items test_1_without_param.py::test_obtuse PASSED [ 25%] test_1_without_param.py::test_acute PASSED [ 50%] test_1_without_param.py::test_right PASSED [ 75%] test_1_without_param.py::test_invalid PASSED [100%] ====================== 4 passed in 0.02s ======================= 10 / 25
  9. output with (venv) $ pytest -v test_2_func_param.py ===================== test session

    starts ====================== collected 4 items test_2_func_param.py::test_type[100-40-40-obtuse] PASSED [ 25%] test_2_func_param.py::test_type[60-60-60-acute] PASSED [ 50%] test_2_func_param.py::test_type[90-60-30-right] PASSED [ 75%] test_2_func_param.py::test_type[0-0-0-invalid] PASSED [100%] ====================== 4 passed in 0.02s ======================= 11 / 25
  10. You can still run just one (venv) $ pytest -v

    'test_2_func_param.py::test_type[0-0-0-invalid]' ===================== test session starts ====================== collected 1 item test_2_func_param.py::test_type[0-0-0-invalid] PASSED [100%] ====================== 1 passed in 0.02s ======================= This uses the node id. 12 / 25
  11. or more (venv) $ pytest -v -k 60 test_2_func_param.py =====================

    test session starts ====================== collected 4 items / 2 deselected / 2 selected test_2_func_param.py::test_type[60-60-60-acute] PASSED [ 50%] test_2_func_param.py::test_type[90-60-30-right] PASSED [100%] =============== 2 passed, 2 deselected in 0.02s ================ An example with -k to pick all tests with 60 degree angles. 13 / 25
  12. Once again, this is Function Parametrization test_2_func_param.py: import pytest from

    triangle import triangle_type many_triangles = [ (100, 40, 40, "obtuse"), ( 60, 60, 60, "acute"), ( 90, 60, 30, "right"), ( 0, 0, 0, "invalid"), ] @pytest.mark.parametrize('a, b, c, expected', many_triangles) def test_type(a, b, c, expected): assert triangle_type(a, b, c) == expected 14 / 25
  13. Fixture Parametrization test_3_fixture_param.py: import pytest from triangle import triangle_type many_triangles

    = [ (100, 40, 40, "obtuse"), ( 60, 60, 60, "acute"), ( 90, 60, 30, "right"), ( 0, 0, 0, "invalid"), ] @pytest.fixture(params=many_triangles) def a_triangle(request): return request.param def test_type(a_triangle): a, b, c, expected = a_triangle assert triangle_type(a, b, c) == expected 15 / 25
  14. output (venv) $ pytest -v test_3_fixture_param.py ======================= test session starts

    ======================= collected 4 items test_3_fixture_param.py::test_type[a_triangle0] PASSED [ 25%] test_3_fixture_param.py::test_type[a_triangle1] PASSED [ 50%] test_3_fixture_param.py::test_type[a_triangle2] PASSED [ 75%] test_3_fixture_param.py::test_type[a_triangle3] PASSED [100%] ======================== 4 passed in 0.02s ======================== 16 / 25
  15. adding an id function test_4_fixture_param.py: import pytest from triangle import

    triangle_type many_triangles = [ (100, 40, 40, "obtuse"), ( 60, 60, 60, "acute"), ( 90, 60, 30, "right"), ( 0, 0, 0, "invalid"), ] def idfn(a_triangle): a, b, c, expected = a_triangle return f'{a}_{b}_{c}_{expected}' @pytest.fixture(params=many_triangles, ids=idfn) def a_triangle(request): return request.param def test_type(a_triangle): a, b, c, expected = a_triangle assert triangle_type(a, b, c) == expected 17 / 25
  16. output (venv) $ pytest -v test_4_fixture_param.py ======================= test session starts

    ======================= collected 4 items test_4_fixture_param.py::test_type[100_40_40_obtuse] PASSED [ 25%] test_4_fixture_param.py::test_type[60_60_60_acute] PASSED [ 50%] test_4_fixture_param.py::test_type[90_60_30_right] PASSED [ 75%] test_4_fixture_param.py::test_type[0_0_0_invalid] PASSED [100%] ======================== 4 passed in 0.02s ======================== 18 / 25
  17. pytest_generate_tests() test_5_gen.py: from triangle import triangle_type many_triangles = [ (100,

    40, 40, "obtuse"), ( 60, 60, 60, "acute"), ( 90, 60, 30, "right"), ( 0, 0, 0, "invalid"), ] def idfn(a_triangle): a, b, c, expected = a_triangle return f'{a}_{b}_{c}_{expected}' def pytest_generate_tests(metafunc): if "a_triangle" in metafunc.fixturenames: metafunc.parametrize("a_triangle", many_triangles, ids=idfn) def test_type(a_triangle): a, b, c, expected = a_triangle assert triangle_type(a, b, c) == expected 19 / 25
  18. output (venv) $ pytest -v test_5_gen.py =================== test session starts

    ==================== platform darwin -- Python 3.7.3, pytest-5.2.1, py-1.8.0, pluggy-0.13.0 -- /Users/o cachedir: .pytest_cache rootdir: /Users/okken/projects/presentation_parametrization/code collected 4 items test_5_gen.py::test_type[100_40_40_obtuse] PASSED [ 25%] test_5_gen.py::test_type[60_60_60_acute] PASSED [ 50%] test_5_gen.py::test_type[90_60_30_right] PASSED [ 75%] test_5_gen.py::test_type[0_0_0_invalid] PASSED [100%] ==================== 4 passed in 0.03s ===================== 20 / 25
  19. Choosing a Technique Guidelines 1. function parametrization use this if

    you can 2. fixture parametrization if doing work to set up each fixture value if cycling through pre-conditions if running multiple test against the same set of "setup states" 3. pytest_generate_tests() if you need to build up the list at runtime if list is based on passed in parameters or external resources for sparse matrix sets 21 / 25
  20. more test cases test_6_more.py: ... triangles = [ ( 1,

    1, 178, "obtuse"), # big angles ( 91, 44, 45, "obtuse"), # just over 90 (0.01, 0.01, 179.98, "obtuse"), # decimals (90, 60, 30, "right"), # check 90 for each angle (10, 90, 80, "right"), (85, 5, 90, "right"), (89, 89, 2, "acute"), # just under 90 (60, 60, 60, "acute"), (0, 0, 0, "invalid"), # zeros (61, 60, 60, "invalid"), # sum > 180 (90, 91, -1, "invalid"), # negative numbers ] @pytest.mark.parametrize('a, b, c, expected', triangles) def test_type(a, b, c, expected): assert triangle_type(a, b, c) == expected 22 / 25
  21. Combining Techniques You can have multiple parametrizations for a test

    function. can have multiple @pytest.mark.parametrize() decorators on a test function. can parameterize multipe fixtures per test can use pytest_generate_tests() to parametrize multiple parameters can use a combination of techniques can blow up into lots and lots of test cases very fast 23 / 25
  22. Not covered, intentionally Use with caution. indirect Have a list

    of parameters defined at the test function that gets passed to a fixture. Kind of a hybrid between function and fixture parametrization. subtests Sort of related, but not really. You can check multiple things within a test. If you care about test case counts, pass, fail, etc, then don't use subtests. 24 / 25
  23. Python Testing with pytest The fastest way to get super

    productive with pytest pytest docs on parametrization, in general function parametrization fixture parametrization pytest_generate_tests podcasts Test & Code Python Bytes Talk Python slack community: Test & Code Slack Twitter: @brianokken, @testandcode This code, and markdown for slides, on github Presentation is at slideshare.net/testandcode Resources 25 / 25