Improve your testing with Pytest and Mock

Presented at PyConSG 2015

Transcript

1. Improve Your Testing With Pytest and Mock Pycon SG - @gabehollombe June 19, 2015
2. Hello!
3. Helps you write better programs “ ”
4. It's popular • PyPy - with over 21,000 tests • Sentry • tox • Cloudera • Spotify • many more
5. Pytest's Key Benefits • Popular • Concise • Helpful assertion messages • Powerful fixtures • Many plugins
6. Also… • runs on Posix/Windows, Python 2.6-3.4, PyPy • well tested with more than a thousand tests against itself • strict backward compatibility policy for safe upgrades • comprehensive online and PDF documentation • many third party plugins and builtin helpers • used in many small and large projects and organisations
7. Installation pip install -U pytest py.test —-version
8. Our First Test
9. def plusone(x): return x + 1 def test_answer(): assert plusone(3) == 5
10. py.test <some file> py.test <some dir> py.test # from within a dir Running py.test
11. $ py.test =========================== test session starts ============================ platform linux -- Python 3.4.0 -- py-1.4.26 -- pytest-2.6.4 collected 1 items test_sample.py F ================================= FAILURES ================================= _______________________________ test_answer ________________________________ def test_answer(): > assert plusone(3) == 5 E assert 4 == 5 E + where 4 = plusone(3) test_sample.py:5: AssertionError ========================= 1 failed in 0.01 seconds =========================
12. def test_answer(): > assert plusone(3) == 5 E assert 4 == 5 E + where 4 = plusone(3) test_sample.py:5: AssertionError
13. Test discovery • collection starts from the initial command line arguments which may be directories, filenames or test ids. • recurse into directories, unless they match norecursedirs • test_*.py or *_test.py files • Test prefixed test classes (without an __init__ method) • test_ prefixed test functions or methods
14. Context-Sensitive Comparisons
15. assert 'foo 1 bar' == 'foo 2 bar' E - foo 1 bar E ? ^ E + foo 2 bar E ? ^
16. assert 'foo\nspam\nbar' == 'foo\neggs\nbar' E foo E - spam E + eggs E bar
17. a = '1'*100 + 'a' + '2'*100 b = '1'*100 + 'b' + '2'*100 assert a == b E assert '111111111111...2222222222222' == '1111111111111...2222222222222' E Skipping 90 identical leading characters in diff, use -v to show E Skipping 91 identical trailing characters in diff, use -v to show E - 1111111111a222222222 E ? ^ E + 1111111111b222222222 E ? ^
18. assert set(['0', '1', '3', '8']) \ == \ set(['0', '3', '5', '8']) E Extra items in the left set: E '1' E Extra items in the right set: E '5'
19. assert {'a': 0, 'b': 1, 'c': 0} \ == \ {'a': 0, 'b': 2, 'd': 0} E Omitting 1 identical items, use -v to show E Differing items: E {'b': 1} != {'b': 2} E Left contains more items: E {'c': 0} E Right contains more items: E {'d': 0} E Use -v to get the full diff
20. Testing Exceptions
21. # content of test_sysexit.py import pytest def f(): raise SystemExit(1) def test_mytest(): with pytest.raises(SystemExit): f() # (this test will pass)
22. Classic xUnit
 Setup & Teardown
23. # Runs once for this whole module def setup_module(module): print "setup_module" # Runs once for this whole module def teardown_module(module): print "teardown_module"
24. # [setup / teardown module]
25. # [setup / teardown module] class TestDB: # Runs once for all the tests @classmethod def setup_class(cls): print "\tsetup_class" # Runs once for all the tests @classmethod def teardown_class(cls): print "\t teardown_class"
26. # [setup / teardown module] class TestDB: # [setup / teardown class]
27. # [setup / teardown module] class TestDB: # [setup / teardown class] # Runs once for each test in this class def setup_method(self, method): print "\t\t setup_method" # Runs once for each test in this class def teardown_method(self, method): print "\t\t teardown_method"
28. # [setup / teardown module] class TestDB: # [setup / teardown class] # [setup / teardown method]
29. # [setup / teardown module] class TestDB: # [setup / teardown class] # [setup / teardown method] def test_one(self): print "\t\t\t test_one" def test_two(self): print "\t\t\t test_two"
30. setup_module setup_class setup_method test_one teardown_method setup_method test_two teardown_method teardown_class teardown_module
31. Fixtures
32. Fixtures The purpose of test fixtures is to provide a fixed baseline upon which tests can reliably and repeatedly execute. In Pytest, fixtures have explicit names and are activated by declaring their use from test functions, modules, classes or whole projects (via dependency injection).
33. # content of ./test_fixture_example.py import pytest class Person: def greet(): return "Hello there!"
34. # content of ./test_fixture_example.py import pytest class Person: def greet(): return "Hello there!" @pytest.fixture def person(): return Person()
35. # content of ./test_fixture_example.py import pytest class Person: def greet(): return "Hello there!" @pytest.fixture def person(): return Person() def test_greet(person): greeting = person.greet() assert greeting == "Hi there!
36. • pytest sees that test_greet needs a function argument named person • A matching fixture function is discovered by looking for a fixture-marked function named person • person() is called to create an instance • test_greet(<Person instance>) is called What's Happening
37. # content of ./test_fixture_example.py import pytest class Person: def greet(): return "Hello there!" @pytest.fixture def person(): return Person() def test_greet(person): greeting = person.greet() assert greeting == "Hi there!
38. > assert greeting == "Hi there!" E assert 'Hello there!' == 'Hi there!' E - Hi there! E + Hello there!
39. • Can call other fixtures • Can be used to parameterize a test function against a collection of test values • Can be shared between test files via conftest.py • @pytest.fixture(autouse=True) acts as setup/teardown without explicit request by your test functions • You can even test your fixtures (they’re just functions) Fixtures are awesome
40. Verifying Calls
41. # content of mock_example.py: class DB: def __init__(self): pass def persist(self, person): pass
42. # content of mock_example.py: class DB: def __init__(self): pass def persist(self, person): pass class Person: def __init__(self, name, db): self.name = name self.db = db def save(self): self.db.persist(self)
43. # content of mock_test.py import pytest from mock import Mock from mock_example import Person, DB
44. # content of mock_test.py import pytest from mock import Mock from mock_example import Person, DB @pytest.fixture def mock_db(): return Mock(spec=DB) # specs to follow . . .
45. # content of mock_test.py (continued) def test_save_persists_to_db(mock_db): gabe = Person("Gabe", mock_db) gabe.save() mock_db.persist.assert_called_with(gabe) # Passes
46. # content of mock_test.py (continued) def test_fail_not_called(mock_db): mock_db.persist.assert_called_with() # E AssertionError: Expected call: persist() # E Not called
47. # content of mock_test.py (continued) def test_fail_called_with_other_arg(mock_db): mock_db.persist(1, 2, 3) mock_db.persist.assert_called_with() # E AssertionError: Expected call: persist() # E Actual call: persist(1, 2, 3)
48. # content of mock_test.py (continued) def test_any_call(mock_db): mock_db.persist(1) mock_db.persist(2) mock_db.persist.assert_any_call(1) # Passes
49. • assert_called_with only tracks the last call • if you call mock('bar') and then try mock.assert_any_call('foo'), it will fail, but it won’t tell you that mock('bar') was called • the docs are your friend Tiny gotchas (read the docs)
50. Stubbing Return Values
51. # content of forecaster.py import random class WeatherService: def barometer(self): # some unpredictable result here (live weather) random.choice(['rising', 'falling'])
52. # content of forecaster.py continued class Forecaster: def __init__(self, weather_service): self.weather_service = weather_service def forecast(self): reading = self.weather_service.barometer() forecasts = dict( rising='Going to rain', falling='Looks clear', ) return forecasts[reading]
53. import pytest from mock import Mock from forecaster import Forecaster, WeatherService @pytest.fixture def mock_ws(): return Mock(spec=WeatherService)
54. import pytest from mock import Mock from forecaster import Forecaster, WeatherService @pytest.fixture def mock_ws(): return Mock(spec=WeatherService) def test_rain_when_barometer_rising(mock_ws): forecaster = Forecaster(mock_ws) mock_ws.barometer.return_value = 'rising' assert forecaster.forecast() == 'Going to rain'
55. import pytest from mock import Mock from forecaster import Forecaster, WeatherService @pytest.fixture def mock_ws(): return Mock(spec=WeatherService) def test_rain_when_barometer_rising(mock_ws): forecaster = Forecaster(mock_ws) mock_ws.barometer.return_value = 'rising' assert forecaster.forecast() == 'Going to rain' def test_clear_when_barometer_falling(mock_ws): forecaster = Forecaster(mock_ws) mock_ws.barometer.return_value = 'falling' assert forecaster.forecast() == 'Looks clear'
56. @pytest.mark.parametrize("reading, expected_forecast", [ ('rising', 'Going to rain'), ('falling', 'Looks clear'), ]) def test_forecast(reading, expected_forecast, mock_ws): forecaster = Forecaster(mock_ws) mock_ws.barometer.return_value = reading assert forecaster.forecast() == expected_forecast Refactored to use a parametrized test
57. Monkeypatching
58. # content of weather_service.py import random class WeatherService: def barometer(self): # some unpredictable result here (live weather) random.choice(['rising', 'falling'])
59. # content of forecaster.py from weather_service import WeatherService class Forecaster: def __init__(self): self.weather_service = WeatherService() # . . . What if I can't inject a mock into my constructor?
60. monkeypatch to the rescue • it's a special predefined fixture • use monkeypatch.setattr() to go in to a module and patch a value • pytest removes the patch when the test function returns
61. def test_rain_when_barometer_rising(monkeypatch, mock_ws): WS = Mock(return_value=mock_ws) monkeypatch.setattr('forecaster.WeatherService', WS) forecaster = Forecaster() mock_ws.barometer.return_value = 'rising' assert forecaster.forecast() == 'Going to rain' Using the monkeypatch fixture
62. Plugins
63. pytest-allure-adaptor-1.5.4 pytest-ansible-1.1 pytest-autochecklog-0.1.2 pytest-bdd-2.6.1 pytest-beakerlib-0.5 pytest-beds-0.0.1 pytest-bench-0.3.0 pytest-benchmark-2.3.0 pytest-blockage-0.1 pytest-bpdb-0.1.4 pytest-browsermob-proxy-0.1 pytest-bugzilla-0.2 pytest-cache-1.0 pytest-cagoule-0.2.0 pytest-capturelog-0.7 pytest-catchlog-1.0 pytest-circleci-0.0.2 pytest-cloud-1.0.15 pytest-codecheckers-0.2 pytest-colordots-0.1 pytest-config-0.0.11 pytest-contextfixture-0.1.1 pytest-couchdbkit-0.5.1 pytest-cov-1.8.1 pytest-cpp-0.3.1 pytest-dbfixtures-0.9.0 pytest-dbus-notification-1.0.1 pytest-describe-0.10 pytest-diffeo-0.1.8.dev3 pytest-django-2.8.0 pytest-django-haystack-0.1.1 pytest-django-lite-0.1.1 pytest-django-sqlcount-0.1.0 pytest-echo-1.3 pytest-env-0.5.1 pytest-eradicate-0.0.2 pytest-figleaf-1.0 pytest-fixture-tools-1.0.0 pytest-flakes-0.2 pytest-flask-0.7.2 pytest-greendots-0.3 pytest-growl-0.2 pytest-httpbin-0.0.6 pytest-httpretty-0.2.0 pytest-incremental-0.3.0 pytest-instafail-0.3.0 pytest-ipdb-0.1-prerelease2 pytest-ipynb-0.1.1 pytest-jira-0.01 pytest-knows-0.1.5 pytest-konira-0.2 pytest-localserver-0.3.4 pytest-marker-bugzilla-0.06 pytest-markfiltration-0.8 pytest-marks-0.4 pytest-mock-0.4.0 pytest-monkeyplus-1.1.0 pytest-mozwebqa-1.5 pytest-multihost-0.6 pytest-oerp-0.2.0 pytest-oot-0.3.1 pytest-optional-0.0.2 pytest-ordering-0.3 pytest-osxnotify-0.1.5 pytest-paste-config-0.1 pytest-pep257-0.0.1 pytest-pep8-1.0.6 pytest-pipeline-0.1.0 pytest-poo-0.2 pytest-poo-fail-1.1 pytest-pycharm-0.3.0 pytest-pydev-0.1 pytest-pyq-1.1 pytest-pythonpath-0.6 pytest-qt-1.2.3 pytest-quickcheck-0.8.1 pytest-rage-0.1 pytest-raisesregexp-2.0 pytest-random-0.02 pytest-readme-1.0.0 pytest-regtest-0.5.1 pytest-remove-stale-bytecode-1.0 pytest-rerunfailures-0.05 pytest-runfailed-0.3 pytest-runner-2.3 pytest-services-1.0.4 pytest-sftpserver-1.1.0 pytest-smartcov-0.1 pytest-sourceorder-0.4 pytest-spec-0.2.24 pytest-splinter-1.2.10 pytest-stepwise-0.2 pytest-sugar-0.3.6 pytest-timeout-0.4 pytest-tornado-0.4.2 pytest-translations-0.2.0 pytest-twisted-1.5 pytest-unmarked-1.1 pytest-watch-2.0.0 pytest-xdist-1.11 pytest-xprocess-0.8 pytest-yamlwsgi-0.6 pytest-zap-0.2
64. Invoke the python debugger for each failure
 pytest <test_fie> --pdb
 
 Debug first failure, then stop running tests
 pytest <test_fie> -x --pdb
 
 Prefer ipdb? Use mverteuil/pytest-ipdb
 pip install \ 
 git+git://github.com/mverteuil/pytest-ipdb.git 
 pytest <test_fie> --ipdb

65. Time for one more tip?
66. import pytest class Developer: def __init__(self, favorite): self.loves=favorite def brag(self): return self.loves + " is the best!"
67. # class Developer . . .
68. # class Developer . . . @pytest.fixture def language(): return 'python' @pytest.fixture def developer(language): return Developer(language)
69. # class Developer . . . # fixtures . . .
70. # class Developer . . . # fixtures . . . def test_brag(developer): assert developer.brag() == "python is the best!"
71. # class Developer . . . # fixtures . . . # test_brag . . . What if we want to make this developer instance love ruby instead?
72. # class Developer . . . # fixtures . . . # test_brag . . . You can parametrize dependent fixtures!
73. # class Developer . . . # fixtures . . . # test_brag . . . @pytest.mark.parametrize('language', ['ruby']) def test_brag_works_for_ruby(developer): assert developer.brag() == "ruby is the best!"
74. # class Developer . . . @pytest.fixture def language(): return 'python' @pytest.fixture def developer(language): return Developer(language)
75. That's all for today.
76. Where to read more • pytest.org • www.voidspace.org.uk/python/mock
77. Questions?
78. Thank you! @gabehollombe gabe@neo.com