Testing Flask Applications

Testing Flask applications can be done in many ways. This presentation covers the usage of pytest and a few other libraries.

Transcript

1. Testing Flask Applications (with pytest and other things)
2. Matt Wright @mattupstate Engineer @ChatID • Python, Ruby, CoffeeScript • Flask, Chef, AngularJS Open Source • Flask-Security • Flask-Social • Flask-Mail + a few others
3. What is a test?
4. What is a test? The means by which the presence, quality, or genuineness of anything is determined; a means of trial [1]
6. Will it blend?
8. Yes! It Blends!
9. What is software testing?
10. What is software testing? Software testing can be stated as the process of validating and verifying that a computer program/application/product: • meets the requirements that guided its design and development • works as expected • can be implemented with the same characteristics • and satisfies the needs of stakeholders. [2]
12. What is software testing?
13. What is software testing? The means by which essential or distinctive characteristics of a piece of software is determined.
15. Does addition work in Python 3?
16. # ~/test.py try: assert 1 + 1 == 2 print('Test passed!') except AssertionError: print('Test failed!')
17. barry @ paradise-cafe in ~ $ python3 test.py
18. barry @ paradise-cafe in ~ $ python3 test.py Test Passed!
19. Hell yes! Addition works in Python 3!
20. But why?!?!
21. 1. Proves that your code works But why?!?!
22. 1. Proves that your code works 2. Demonstrates thought process But why?!?!
23. 1. Proves that your code works 2. Demonstrates thought process 3. Refactoring becomes a lot easier But why?!?!
24. But why?!?! 1. Proves that your code works 2. Demonstrates thought process 3. Refactoring becomes a lot easier 4. Tests can serve as example code
25. But why?!?! 1. Proves that your code works 2. Demonstrates thought process 3. Refactoring becomes a lot easier 4. Tests can serve as example code 5. A good challenge is fun!
26. Testing Flask Apps
27. Disclaimer!
28. Getting Started
29. barry @ pc in ~/myapp $ mkvirtualenv myapp -p `which python3` … barry @ pc in ~/myapp workon:myapp $ pip install flask …
30. # ~/myapp/app.py from flask import Flask app = Flask(__name__) @app.route('/') def index(): return 'Hello, world!' @app.route('/contact') def contact(): return 'barry@manilow.com'
31. Now What?
32. Werkzeug! http://werkzeug.pocoo.org/docs/test/ also http://flask.pocoo.org/docs/testing/
33. # ~/myapp/test_index.py from app import app app.testing = True client = app.test_client() res = client.get('/') try: assert res.data == b'Hello, world!' print('Test passed!') except AssertionError: print('Test failed!')
34. barry @ pc in ~/myapp workon:myapp $ python test_index.py Test passed!
35. unittest
36. # ~/myapp/test_index.py import unittest from app import app app.testing = True class IndexTestCase(unittest.TestCase): def test_index(self): client = app.test_client() res = client.get('/') self.assertEqual(res.data, b'Hello, world!') if __name__ == '__main__': unittest.main()
37. barry @ pc in ~/myapp workon:myapp $ python test_index.py
38. barry @ pc in ~/myapp workon:myapp $ python test_index.py . -------------------------------------- Ran 1 test in 0.020s OK
39. # ~/myapp/test_contact.py import unittest from app import app app.testing = True class ContactTestCase(unittest.TestCase): def test_contact(self): client = app.test_client() res = client.get('/contact') self.assertEqual(res.data, b'barry@manilow.com') if __name__ == '__main__': unittest.main()
40. barry @ pc in ~/myapp workon:myapp $ python test_contact.py . -------------------------------------- Ran 1 test in 0.020s OK
41. # ~/myapp/test_suite.py import unittest from test_index import IndexTestCase from test_contact import ContactTestCase if __name__ == '__main__': unittest.main()
42. barry @ pc in ~/myapp workon:myapp $ python test_suite.py . -------------------------------------- Ran 2 tests in 0.016s OK
43. pytest!
44. 1. Easy to get started pytest
45. barry @ pc in ~/myapp workon:myapp $ pip install pytest
46. barry @ pc in ~/myapp workon:myapp $ pip install pytest barry @ pc in ~/myapp workon:myapp $ py.test
47. barry @ pc in ~/myapp workon:myapp $ pip install pytest barry @ pc in ~/myapp workon:myapp $ py.test =================== test session starts ==================== platform darwin -- Python 3.3.3 -- py-1.4.20 -- pytest-2.5.2 collected 4 items test_contact.py . test_index.py . test_suite.py .. ================ 4 passed in 0.20 seconds ==================
48. barry @ pc in ~/myapp workon:myapp $ rm test_suite.py
49. barry @ pc in ~/myapp workon:myapp $ rm test_suite.py barry @ pc in ~/myapp workon:myapp $ py.test
50. barry @ pc in ~/myapp workon:myapp $ rm test_suite.py barry @ pc in ~/myapp workon:myapp $ py.test =================== test session starts ==================== platform darwin -- Python 3.3.3 -- py-1.4.20 -- pytest-2.5.2 collected 2 items test_contact.py . test_index.py . ================ 2 passed in 0.10 seconds ==================
51. 1. Easy to get started 2. Asserting with the assert statement! pytest
52. # ~/myapp/test_index.py import unittest from app import app class IndexTestCase(unittest.TestCase): def test_index(self): client = app.test_client() res = client.get('/') self.assertEqual(res.data, b'Hello, world!') if __name__ == '__main__': unittest.main()
53. # ~/myapp/test_index.py import unittest from app import app class IndexTestCase(unittest.TestCase): def test_index(self): client = app.test_client() res = client.get('/') assert res.data == b'Hello, world!' if __name__ == '__main__': unittest.main()
54. 1. Easy to get started 2. Asserting with the assert statement! 3. Helpful traceback and failing assertion reporting pytest
55. # ~/myapp/test_index.py import unittest from app import app class IndexTestCase(unittest.TestCase): def test_index(self): client = app.test_client() res = client.get('/') self.assertEqual(res.data, b'Hello, moon!') if __name__ == '__main__': unittest.main()
56. barry @ pc in ~/myapp workon:myapp $ py.test ========================= test session starts ========================== platform darwin -- Python 3.3.3 -- py-1.4.20 -- pytest-2.5.2 collected 2 items test_contact.py . test_index.py F [CONTINUED...]
57. =============================== FAILURES =============================== _______________________ IndexTestCase.test_index _______________________ self = <test_index.IndexTestCase testMethod=test_index> def test_index(self): client = app.test_client() res = client.get('/') > assert res.data == b'Hello, world!a' E AssertionError: b'Hello, world!' != b'Hello, moon!' test_index.py:9: AssertionError ================== 1 failed, 1 passed in 0.06 seconds ==================
58. # ~/myapp/test_index.py import unittest from app import app class IndexTestCase(unittest.TestCase): def test_index(self): client = app.test_client() res = client.get('/') assert res.data == b'Hello, moon!' if __name__ == '__main__': unittest.main()
59. barry @ pc in ~/myapp workon:myapp $ py.test ========================= test session starts ========================== platform darwin -- Python 3.3.3 -- py-1.4.20 -- pytest-2.5.2 collected 2 items test_contact.py . test_index.py F [CONTINUED...]
60. =============================== FAILURES =============================== _______________________ IndexTestCase.test_index _______________________ self = <test_index.IndexTestCase testMethod=test_index> def test_index(self): client = app.test_client() res = client.get('/') > assert res.data == b'Hello, world!a' E AssertionError: assert b'Hello, world!' == b'Hello, moon!' E At index 7 diff: 119 != 109 E Left contains more items, first extra item: 33 test_index.py:9: AssertionError ================== 1 failed, 1 passed in 0.06 seconds ==================
61. 1. Easy to get started 2. Asserting with the assert statement! 3. Helpful traceback and failing assertion reporting 4. Print debugging and the capturing of standard output during test execution pytest
62. # ~/myapp/test_index.py import unittest from app import app class IndexTestCase(unittest.TestCase): def test_index(self): client = app.test_client() res = client.get('/') print('a debug message') assert res.data == b'Hello, moon!' if __name__ == '__main__': unittest.main()
63. ... def test_index(self): client = app.test_client() res = client.get('/') > assert res.data == b'Hello, world!a' E AssertionError: assert b'Hello, world!' == b'Hello, moon!' E At index 7 diff: 119 != 109 E Left contains more items, first extra item: 33 test_index.py:9: AssertionError --------------------------- Captured stdout ---------------------------- a debug message ================== 1 failed, 1 passed in 0.06 seconds ==================
64. But wait! There’s more!
65. Just say NO to boilerplate
66. # ~/myapp/test_index.py import unittest from app import app class IndexTestCase(unittest.TestCase): def test_index(self): client = app.test_client() res = client.get('/') assert res.data == b'Hello, world!' if __name__ == '__main__': unittest.main()
67. # ~/myapp/test_index.py from app import app app.testing = True def test_index(): client = app.test_client() res = client.get('/') assert res.data == b'Hello, world!'
68. # ~/myapp/test_contact.py import unittest from app import app class ContactTestCase(unittest.TestCase): def test_contact(self): client = app.test_client() res = client.get('/contact') assert res.data == b'barry@manilow.com' if __name__ == '__main__': unittest.main()
69. # ~/myapp/test_contact.py from app import app app.testing = True def test_contact(): client = app.test_client() res = client.get('/contact') assert res.data == b'barry@manilow.com'
70. Fixtures
71. # ~/myapp/conftest.py import pytest from app import app as _app @pytest.fixture() def app(): _app.testing = True return _app @pytest.fixture() def client(app): return app.test_client()
72. # ~/myapp/test_index.py def test_index(client): res = client.get('/') assert res.data == b'Hello, world!' # ~/myapp/test_contact.py def test_contact(client): res = client.get('/contact') assert res.data == b'barry@manilow.com'
73. Fixture Scope
74. # ~/myapp/conftest.py import pytest from app import app as _app @pytest.fixture(scope='session') def app(): app.testing = True return _app @pytest.fixture(scope='session') def client(app): return app.test_client()
75. Protip: See what fixtures are available
76. barry @ pc in ~/myapp workon:myapp $ py.test --fixtures
77. barry @ pc in ~/myapp workon:myapp $ py.test --fixtures ======================== test session starts ========================== platform darwin -- Python 3.3.3 -- py-1.4.20 -- pytest-2.5.2 collected 2 items capsys enables capturing of writes to sys.stdout/sys.stderr and makes captured output available via ``capsys.readouterr()`` method calls which return a ``(out, err)`` tuple. capfd enables capturing of writes to file descriptors 1 and 2 and makes captured output available via ``capsys.readouterr()`` method calls which return a ``(out, err)`` tuple.
78. tmpdir return a temporary directory path object which is unique to each test function invocation, created as a sub directory of the base temporary directory. The returned object is a `py.path.local`_ path object. -------------------- fixtures defined from conftest -------------------- app conftest.py:5: no docstring available client conftest.py:9: no docstring available =========================== in 0.02 seconds ==========================
79. Project Layout
80. barry @ pc in ~/myapp workon:myapp $ tree . ├── app.py └── tests ├── conftest.py ├── test_contact.py └── test_index.py 1 directory, 4 files
81. barry @ pc in ~/myapp workon:myapp $ py.test [BLAH BLAH BLAH] _path/local.py", line 620, in pyimport __import__(modname) File "/Users/matt/myapp/tests/conftest.py", line 2, in <module> from app import app as _app ImportError: No module named 'app'
82. # ~/myapp/setup.py from setuptools import setup setup( name='My App', version='0.1.0', py_modules=['app'], install_requires=['Flask==0.10.1'] tests_require=['pytest==2.5.2'] )
83. # ~/myapp/setup.py from setuptools import setup, find_packages setup( name='My App', version='0.1.0', packages=find_packages(), install_requires=['Flask==0.10.1'] tests_require=['pytest==2.5.2'] )
84. barry @ pc in ~/myapp workon:myapp $ pip install -e .
85. barry @ pc in ~/myapp workon:myapp $ pip install -e . barry @ pc in ~/myapp workon:myapp $ py.test ========================= test session starts ========================== platform darwin -- Python 3.3.3 -- py-1.4.20 -- pytest-2.5.2 collected 2 items tests/test_contact.py . tests/test_index.py . ======================= 2 passed in 0.04 seconds =======================
86. Testing Forms (application/x-www-form-urlencoded)
87. # ~/myapp/app.py from flask import request @app.route('/contact', methods=['POST']) def send_contact(): msg = request.form.get('msg', None) if msg is not None: return 'Message sent!' else: return 'Invalid message', 400
88. # ~/myapp/tests/test_contact.py def test_send_contact(client): data = {'msg': 'Hi, Barry!'} res = client.post('/contact', data=data) assert res.status_code == 200 assert res.data == b'Message sent!' def test_send_invalid_contact(client): res = client.post('/contact', data={}) assert res.status_code == 400 assert res.data == b'Invalid message'
89. Testing Forms (application/json)
90. # ~/myapp/app.py from flask import request, jsonify @app.route('/contact.json', methods=['POST']) def send_contact_json(): json = request.get_json() msg = json.get('msg', None) if msg is not None: return jsonify({'success': True}) else: return jsonify({'success': False}), 400
91. # ~/myapp/tests/test_contact.py from flask import json JSON_MEDIA_TYPE = 'application/json' def test_send_contact_json(client): data = json.dumps({'msg': 'Hi, Barry!'}) res = client.post('/contact.json', data=data, content_type=JSON_MEDIA_TYPE) assert res.status_code == 200 assert res.content_type == JSON_MEDIA_TYPE jdata = json.loads(res.data) assert jdata['success'] is True
92. Dealing with a Database
93. SQLAlchemy (the easy way)
94. # ~/myapp/app.py from flask_sqlalchemy import SQLAlchemy db = SQLAlchemy() def create_app(database_uri=None): database_uri = database_uri or 'sqlite:////myapp.db' app = Flask(__name__) app.config['SQLALCHEMY_DATABASE_URI'] = database_uri db.init_app(app) app.register_blueprint(my_blueprint) return app
95. # ~/myapp/tests/conftest.py import tempfile, pytest from app import create_app, db @pytest.fixture() def app(tmpdir): f, path = tempfile.mkstemp(suffix='.db', dir=str(tmpdir)) app = create_app(database_uri='sqlite:///' + path) with app.app_context(): db.create_all() return app
96. http://alexmic.net/flask-sqlalchemy-pytest/
97. Test Data (with factory-boy)
98. barry @ pc in ~/myapp workon:myapp $ pip install factory-boy
99. # ~/myapp/app.py class User(db.Model): id = db.Column(db.Integer, primary_key=True) name = db.Column(db.String(255)) email = db.Column(db.String(255)) password = db.Column(db.String(255))
100. # ~/myapp/app.py from factory import Sequence from factory.alchemy import SQLAlchemyModelFactory class ModelFactory(SQLAlchemyModelFactory): FACTORY_SESSION = db.session ABSTRACT_FACTORY = True class UserModelFactory(ModelFactory): FACTORY_FOR = User name = Sequence(lambda n: 'User %s' % n) email = Sequence(lambda n: 'user%s@manilow.com' % n) password = 'password'
101. # ~/myapp/tests/conftest.py import pytest from app import UserModelFactory, db @pytest.fixture() def user(app): with app.app_context() user = UserModelFactory() db.session.commit() return user
102. # ~/myapp/tests/test_users.py def test_update_user(client, user): res = client.post('/users/%s' % user.id, data={ 'name': 'Barry' }) assert res.status_code = 400
103. Testing ReST APIs
104. JSON Schema http://json-schema.org/
105. # ~/myapp/contact_schema.json { "type": "object", "properties": { "success": { "type": "boolean" } }, "additionalProperties": false }
106. barry @ pc in ~/myapp workon:myapp $ pip install jsonschema
107. # ~/myapp/app.py import os, jsonschema as _jsonschema from flask import json class jsonschema: def __init__(self, name): file_name = '%s.json' % name base_dir = os.path.join(os.path.dirname(os.path.realpath(__file__))) with open(os.path.join(base_dir, file_name)) as f: schema = json.load(f) self._schema = schema def __eq__(self, jdata): return _jsonschema.validate(jdata, self._schema) is None
108. # ~/myapp/tests/test_contact.py from flask import json from app import jsonschema JSON_MEDIA_TYPE = 'application/json' def test_send_contact_json(client): data = json.dumps({'msg': 'Hi, Barry!'}) res = client.post('/contact.json', data=data, content_type=JSON_MEDIA_TYPE) assert res.status_code == 200 assert res.content_type == JSON_MEDIA_TYPE assert json.loads(res.data) == jsonschema('contact_schema')
109. pytest Extensions
110. Parting Advice
111. Imagine the desired API Parting Advice
112. Imagine the desired API Avoid hypothetical edge cases Parting Advice
113. Parting Advice Imagine the desired API Avoid hypothetical edge cases Cover edge cases as they appear
114. Thank You! mattupstate.com gittip.com/mattupstate github.com/mattupstate twitter.com/mattupstate
115. Annotations [1] http://dictionary.reference.com/ [2] http://en.wikipedia.org/wiki/Software_testing