Slide 1
Slide 1 text
Testing
Flask Applications
(with pytest and other things)
Slide 2
Slide 2 text
Matt Wright @mattupstate
Engineer @ChatID
● Python, Ruby, CoffeeScript
● Flask, Chef, AngularJS
Open Source
● Flask-Security
● Flask-Social
● Flask-Mail + a few others
Slide 3
Slide 3 text
What is a test?
Slide 4
Slide 4 text
What is a test?
The means by which the
presence, quality, or genuineness
of anything is determined;
a means of trial [1]
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
Will it
blend?
Slide 7
Slide 7 text
No content
Slide 8
Slide 8 text
Yes! It
Blends!
Slide 9
Slide 9 text
What is software testing?
Slide 10
Slide 10 text
What is software testing?
Software testing can be stated as the process of validating
and verifying that a computer
program/application/product:
● meets the requirements that guided its design and
development
● works as expected
● can be implemented with the same characteristics
● and satisfies the needs of stakeholders. [2]
Slide 11
Slide 11 text
No content
Slide 12
Slide 12 text
What is software testing?
Slide 13
Slide 13 text
What is software testing?
The means by which essential or
distinctive characteristics of a piece of
software is determined.
Slide 14
Slide 14 text
No content
Slide 15
Slide 15 text
Does addition
work in
Python 3?
Slide 16
Slide 16 text
# ~/test.py
try:
assert 1 + 1 == 2
print('Test passed!')
except AssertionError:
print('Test failed!')
Slide 17
Slide 17 text
barry @ paradise-cafe in ~
$ python3 test.py
Slide 18
Slide 18 text
barry @ paradise-cafe in ~
$ python3 test.py
Test Passed!
Slide 19
Slide 19 text
Hell yes!
Addition works
in Python 3!
Slide 20
Slide 20 text
But why?!?!
Slide 21
Slide 21 text
1. Proves that your code works
But why?!?!
Slide 22
Slide 22 text
1. Proves that your code works
2. Demonstrates thought process
But why?!?!
Slide 23
Slide 23 text
1. Proves that your code works
2. Demonstrates thought process
3. Refactoring becomes a lot easier
But why?!?!
Slide 24
Slide 24 text
But why?!?!
1. Proves that your code works
2. Demonstrates thought process
3. Refactoring becomes a lot easier
4. Tests can serve as example code
Slide 25
Slide 25 text
But why?!?!
1. Proves that your code works
2. Demonstrates thought process
3. Refactoring becomes a lot easier
4. Tests can serve as example code
5. A good challenge is fun!
Slide 26
Slide 26 text
Testing Flask Apps
Slide 27
Slide 27 text
Disclaimer!
Slide 28
Slide 28 text
Getting Started
Slide 29
Slide 29 text
barry @ pc in ~/myapp
$ mkvirtualenv myapp -p `which python3`
…
barry @ pc in ~/myapp workon:myapp
$ pip install flask
…
Slide 30
Slide 30 text
# ~/myapp/app.py
from flask import Flask
app = Flask(__name__)
@app.route('/')
def index():
return 'Hello, world!'
@app.route('/contact')
def contact():
return '[email protected]'
Slide 31
Slide 31 text
Now What?
Slide 32
Slide 32 text
Werkzeug!
http://werkzeug.pocoo.org/docs/test/
also
http://flask.pocoo.org/docs/testing/
Slide 33
Slide 33 text
# ~/myapp/test_index.py
from app import app
app.testing = True
client = app.test_client()
res = client.get('/')
try:
assert res.data == b'Hello, world!'
print('Test passed!')
except AssertionError:
print('Test failed!')
Slide 34
Slide 34 text
barry @ pc in ~/myapp workon:myapp
$ python test_index.py
Test passed!
Slide 35
Slide 35 text
unittest
Slide 36
Slide 36 text
# ~/myapp/test_index.py
import unittest
from app import app
app.testing = True
class IndexTestCase(unittest.TestCase):
def test_index(self):
client = app.test_client()
res = client.get('/')
self.assertEqual(res.data, b'Hello, world!')
if __name__ == '__main__':
unittest.main()
Slide 37
Slide 37 text
barry @ pc in ~/myapp workon:myapp
$ python test_index.py
Slide 38
Slide 38 text
barry @ pc in ~/myapp workon:myapp
$ python test_index.py
.
--------------------------------------
Ran 1 test in 0.020s
OK
Slide 39
Slide 39 text
# ~/myapp/test_contact.py
import unittest
from app import app
app.testing = True
class ContactTestCase(unittest.TestCase):
def test_contact(self):
client = app.test_client()
res = client.get('/contact')
self.assertEqual(res.data, b'[email protected]')
if __name__ == '__main__':
unittest.main()
Slide 40
Slide 40 text
barry @ pc in ~/myapp workon:myapp
$ python test_contact.py
.
--------------------------------------
Ran 1 test in 0.020s
OK
Slide 41
Slide 41 text
# ~/myapp/test_suite.py
import unittest
from test_index import IndexTestCase
from test_contact import ContactTestCase
if __name__ == '__main__':
unittest.main()
Slide 42
Slide 42 text
barry @ pc in ~/myapp workon:myapp
$ python test_suite.py
.
--------------------------------------
Ran 2 tests in 0.016s
OK
Slide 43
Slide 43 text
pytest!
Slide 44
Slide 44 text
1. Easy to get started
pytest
Slide 45
Slide 45 text
barry @ pc in ~/myapp workon:myapp
$ pip install pytest
Slide 46
Slide 46 text
barry @ pc in ~/myapp workon:myapp
$ pip install pytest
barry @ pc in ~/myapp workon:myapp
$ py.test
Slide 47
Slide 47 text
barry @ pc in ~/myapp workon:myapp
$ pip install pytest
barry @ pc in ~/myapp workon:myapp
$ py.test
=================== test session starts ====================
platform darwin -- Python 3.3.3 -- py-1.4.20 -- pytest-2.5.2
collected 4 items
test_contact.py .
test_index.py .
test_suite.py ..
================ 4 passed in 0.20 seconds ==================
Slide 48
Slide 48 text
barry @ pc in ~/myapp workon:myapp
$ rm test_suite.py
Slide 49
Slide 49 text
barry @ pc in ~/myapp workon:myapp
$ rm test_suite.py
barry @ pc in ~/myapp workon:myapp
$ py.test
Slide 50
Slide 50 text
barry @ pc in ~/myapp workon:myapp
$ rm test_suite.py
barry @ pc in ~/myapp workon:myapp
$ py.test
=================== test session starts ====================
platform darwin -- Python 3.3.3 -- py-1.4.20 -- pytest-2.5.2
collected 2 items
test_contact.py .
test_index.py .
================ 2 passed in 0.10 seconds ==================
Slide 51
Slide 51 text
1. Easy to get started
2. Asserting with the assert statement!
pytest
Slide 52
Slide 52 text
# ~/myapp/test_index.py
import unittest
from app import app
class IndexTestCase(unittest.TestCase):
def test_index(self):
client = app.test_client()
res = client.get('/')
self.assertEqual(res.data, b'Hello, world!')
if __name__ == '__main__':
unittest.main()
Slide 53
Slide 53 text
# ~/myapp/test_index.py
import unittest
from app import app
class IndexTestCase(unittest.TestCase):
def test_index(self):
client = app.test_client()
res = client.get('/')
assert res.data == b'Hello, world!'
if __name__ == '__main__':
unittest.main()
Slide 54
Slide 54 text
1. Easy to get started
2. Asserting with the assert statement!
3. Helpful traceback and failing assertion
reporting
pytest
Slide 55
Slide 55 text
# ~/myapp/test_index.py
import unittest
from app import app
class IndexTestCase(unittest.TestCase):
def test_index(self):
client = app.test_client()
res = client.get('/')
self.assertEqual(res.data, b'Hello, moon!')
if __name__ == '__main__':
unittest.main()
Slide 56
Slide 56 text
barry @ pc in ~/myapp workon:myapp
$ py.test
========================= test session starts ==========================
platform darwin -- Python 3.3.3 -- py-1.4.20 -- pytest-2.5.2
collected 2 items
test_contact.py .
test_index.py F
[CONTINUED...]
Slide 57
Slide 57 text
=============================== FAILURES ===============================
_______________________ IndexTestCase.test_index _______________________
self =
def test_index(self):
client = app.test_client()
res = client.get('/')
> assert res.data == b'Hello, world!a'
E AssertionError: b'Hello, world!' != b'Hello, moon!'
test_index.py:9: AssertionError
================== 1 failed, 1 passed in 0.06 seconds ==================
Slide 58
Slide 58 text
# ~/myapp/test_index.py
import unittest
from app import app
class IndexTestCase(unittest.TestCase):
def test_index(self):
client = app.test_client()
res = client.get('/')
assert res.data == b'Hello, moon!'
if __name__ == '__main__':
unittest.main()
Slide 59
Slide 59 text
barry @ pc in ~/myapp workon:myapp
$ py.test
========================= test session starts ==========================
platform darwin -- Python 3.3.3 -- py-1.4.20 -- pytest-2.5.2
collected 2 items
test_contact.py .
test_index.py F
[CONTINUED...]
Slide 60
Slide 60 text
=============================== FAILURES ===============================
_______________________ IndexTestCase.test_index _______________________
self =
def test_index(self):
client = app.test_client()
res = client.get('/')
> assert res.data == b'Hello, world!a'
E AssertionError: assert b'Hello, world!' == b'Hello, moon!'
E At index 7 diff: 119 != 109
E Left contains more items, first extra item: 33
test_index.py:9: AssertionError
================== 1 failed, 1 passed in 0.06 seconds ==================
Slide 61
Slide 61 text
1. Easy to get started
2. Asserting with the assert statement!
3. Helpful traceback and failing assertion
reporting
4. Print debugging and the capturing of
standard output during test execution
pytest
Slide 62
Slide 62 text
# ~/myapp/test_index.py
import unittest
from app import app
class IndexTestCase(unittest.TestCase):
def test_index(self):
client = app.test_client()
res = client.get('/')
print('a debug message')
assert res.data == b'Hello, moon!'
if __name__ == '__main__':
unittest.main()
Slide 63
Slide 63 text
...
def test_index(self):
client = app.test_client()
res = client.get('/')
> assert res.data == b'Hello, world!a'
E AssertionError: assert b'Hello, world!' == b'Hello, moon!'
E At index 7 diff: 119 != 109
E Left contains more items, first extra item: 33
test_index.py:9: AssertionError
--------------------------- Captured stdout ----------------------------
a debug message
================== 1 failed, 1 passed in 0.06 seconds ==================
Slide 64
Slide 64 text
But wait! There’s more!
Slide 65
Slide 65 text
Just say NO
to boilerplate
Slide 66
Slide 66 text
# ~/myapp/test_index.py
import unittest
from app import app
class IndexTestCase(unittest.TestCase):
def test_index(self):
client = app.test_client()
res = client.get('/')
assert res.data == b'Hello, world!'
if __name__ == '__main__':
unittest.main()
Slide 67
Slide 67 text
# ~/myapp/test_index.py
from app import app
app.testing = True
def test_index():
client = app.test_client()
res = client.get('/')
assert res.data == b'Hello, world!'
Slide 68
Slide 68 text
# ~/myapp/test_contact.py
import unittest
from app import app
class ContactTestCase(unittest.TestCase):
def test_contact(self):
client = app.test_client()
res = client.get('/contact')
assert res.data == b'[email protected]'
if __name__ == '__main__':
unittest.main()
Slide 69
Slide 69 text
# ~/myapp/test_contact.py
from app import app
app.testing = True
def test_contact():
client = app.test_client()
res = client.get('/contact')
assert res.data == b'[email protected]'
Slide 70
Slide 70 text
Fixtures
Slide 71
Slide 71 text
# ~/myapp/conftest.py
import pytest
from app import app as _app
@pytest.fixture()
def app():
_app.testing = True
return _app
@pytest.fixture()
def client(app):
return app.test_client()
Slide 72
Slide 72 text
# ~/myapp/test_index.py
def test_index(client):
res = client.get('/')
assert res.data == b'Hello, world!'
# ~/myapp/test_contact.py
def test_contact(client):
res = client.get('/contact')
assert res.data == b'[email protected]'
Slide 73
Slide 73 text
Fixture Scope
Slide 74
Slide 74 text
# ~/myapp/conftest.py
import pytest
from app import app as _app
@pytest.fixture(scope='session')
def app():
app.testing = True
return _app
@pytest.fixture(scope='session')
def client(app):
return app.test_client()
Slide 75
Slide 75 text
Protip:
See what fixtures
are available
Slide 76
Slide 76 text
barry @ pc in ~/myapp workon:myapp
$ py.test --fixtures
Slide 77
Slide 77 text
barry @ pc in ~/myapp workon:myapp
$ py.test --fixtures
======================== test session starts ==========================
platform darwin -- Python 3.3.3 -- py-1.4.20 -- pytest-2.5.2
collected 2 items
capsys
enables capturing of writes to sys.stdout/sys.stderr and makes
captured output available via ``capsys.readouterr()`` method calls
which return a ``(out, err)`` tuple.
capfd
enables capturing of writes to file descriptors 1 and 2 and makes
captured output available via ``capsys.readouterr()`` method calls
which return a ``(out, err)`` tuple.
Slide 78
Slide 78 text
tmpdir
return a temporary directory path object
which is unique to each test function invocation,
created as a sub directory of the base temporary
directory. The returned object is a `py.path.local`_
path object.
-------------------- fixtures defined from conftest --------------------
app
conftest.py:5: no docstring available
client
conftest.py:9: no docstring available
=========================== in 0.02 seconds ==========================
Slide 79
Slide 79 text
Project Layout
Slide 80
Slide 80 text
barry @ pc in ~/myapp workon:myapp
$ tree
.
├── app.py
└── tests
├── conftest.py
├── test_contact.py
└── test_index.py
1 directory, 4 files
Slide 81
Slide 81 text
barry @ pc in ~/myapp workon:myapp
$ py.test
[BLAH BLAH BLAH]
_path/local.py", line 620, in pyimport
__import__(modname)
File "/Users/matt/myapp/tests/conftest.py", line 2, in
from app import app as _app
ImportError: No module named 'app'
Slide 82
Slide 82 text
# ~/myapp/setup.py
from setuptools import setup
setup(
name='My App',
version='0.1.0',
py_modules=['app'],
install_requires=['Flask==0.10.1']
tests_require=['pytest==2.5.2']
)
Slide 83
Slide 83 text
# ~/myapp/setup.py
from setuptools import setup, find_packages
setup(
name='My App',
version='0.1.0',
packages=find_packages(),
install_requires=['Flask==0.10.1']
tests_require=['pytest==2.5.2']
)
Slide 84
Slide 84 text
barry @ pc in ~/myapp workon:myapp
$ pip install -e .
Slide 85
Slide 85 text
barry @ pc in ~/myapp workon:myapp
$ pip install -e .
barry @ pc in ~/myapp workon:myapp
$ py.test
========================= test session starts ==========================
platform darwin -- Python 3.3.3 -- py-1.4.20 -- pytest-2.5.2
collected 2 items
tests/test_contact.py .
tests/test_index.py .
======================= 2 passed in 0.04 seconds =======================
Slide 86
Slide 86 text
Testing Forms
(application/x-www-form-urlencoded)
Slide 87
Slide 87 text
# ~/myapp/app.py
from flask import request
@app.route('/contact', methods=['POST'])
def send_contact():
msg = request.form.get('msg', None)
if msg is not None:
return 'Message sent!'
else:
return 'Invalid message', 400
Slide 88
Slide 88 text
# ~/myapp/tests/test_contact.py
def test_send_contact(client):
data = {'msg': 'Hi, Barry!'}
res = client.post('/contact', data=data)
assert res.status_code == 200
assert res.data == b'Message sent!'
def test_send_invalid_contact(client):
res = client.post('/contact', data={})
assert res.status_code == 400
assert res.data == b'Invalid message'
Slide 89
Slide 89 text
Testing Forms
(application/json)
Slide 90
Slide 90 text
# ~/myapp/app.py
from flask import request, jsonify
@app.route('/contact.json', methods=['POST'])
def send_contact_json():
json = request.get_json()
msg = json.get('msg', None)
if msg is not None:
return jsonify({'success': True})
else:
return jsonify({'success': False}), 400
Slide 91
Slide 91 text
# ~/myapp/tests/test_contact.py
from flask import json
JSON_MEDIA_TYPE = 'application/json'
def test_send_contact_json(client):
data = json.dumps({'msg': 'Hi, Barry!'})
res = client.post('/contact.json', data=data,
content_type=JSON_MEDIA_TYPE)
assert res.status_code == 200
assert res.content_type == JSON_MEDIA_TYPE
jdata = json.loads(res.data)
assert jdata['success'] is True
Slide 92
Slide 92 text
Dealing with
a Database
Slide 93
Slide 93 text
SQLAlchemy
(the easy way)
Slide 94
Slide 94 text
# ~/myapp/app.py
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
def create_app(database_uri=None):
database_uri = database_uri or 'sqlite:////myapp.db'
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = database_uri
db.init_app(app)
app.register_blueprint(my_blueprint)
return app
Slide 95
Slide 95 text
# ~/myapp/tests/conftest.py
import tempfile, pytest
from app import create_app, db
@pytest.fixture()
def app(tmpdir):
f, path = tempfile.mkstemp(suffix='.db', dir=str(tmpdir))
app = create_app(database_uri='sqlite:///' + path)
with app.app_context():
db.create_all()
return app
Slide 96
Slide 96 text
http://alexmic.net/flask-sqlalchemy-pytest/
Slide 97
Slide 97 text
Test Data
(with factory-boy)
Slide 98
Slide 98 text
barry @ pc in ~/myapp workon:myapp
$ pip install factory-boy
Slide 99
Slide 99 text
# ~/myapp/app.py
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(255))
email = db.Column(db.String(255))
password = db.Column(db.String(255))
Slide 100
Slide 100 text
# ~/myapp/app.py
from factory import Sequence
from factory.alchemy import SQLAlchemyModelFactory
class ModelFactory(SQLAlchemyModelFactory):
FACTORY_SESSION = db.session
ABSTRACT_FACTORY = True
class UserModelFactory(ModelFactory):
FACTORY_FOR = User
name = Sequence(lambda n: 'User %s' % n)
email = Sequence(lambda n: 'user%[email protected]' % n)
password = 'password'
Slide 101
Slide 101 text
# ~/myapp/tests/conftest.py
import pytest
from app import UserModelFactory, db
@pytest.fixture()
def user(app):
with app.app_context()
user = UserModelFactory()
db.session.commit()
return user
Slide 102
Slide 102 text
# ~/myapp/tests/test_users.py
def test_update_user(client, user):
res = client.post('/users/%s' % user.id, data={
'name': 'Barry'
})
assert res.status_code = 400
Slide 103
Slide 103 text
Testing ReST APIs
Slide 104
Slide 104 text
JSON Schema
http://json-schema.org/
Slide 105
Slide 105 text
# ~/myapp/contact_schema.json
{
"type": "object",
"properties": {
"success": {
"type": "boolean"
}
},
"additionalProperties": false
}
Slide 106
Slide 106 text
barry @ pc in ~/myapp workon:myapp
$ pip install jsonschema
Slide 107
Slide 107 text
# ~/myapp/app.py
import os, jsonschema as _jsonschema
from flask import json
class jsonschema:
def __init__(self, name):
file_name = '%s.json' % name
base_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)))
with open(os.path.join(base_dir, file_name)) as f:
schema = json.load(f)
self._schema = schema
def __eq__(self, jdata):
return _jsonschema.validate(jdata, self._schema) is None
Slide 108
Slide 108 text
# ~/myapp/tests/test_contact.py
from flask import json
from app import jsonschema
JSON_MEDIA_TYPE = 'application/json'
def test_send_contact_json(client):
data = json.dumps({'msg': 'Hi, Barry!'})
res = client.post('/contact.json', data=data,
content_type=JSON_MEDIA_TYPE)
assert res.status_code == 200
assert res.content_type == JSON_MEDIA_TYPE
assert json.loads(res.data) == jsonschema('contact_schema')
Slide 109
Slide 109 text
pytest
Extensions
Slide 110
Slide 110 text
Parting Advice
Slide 111
Slide 111 text
Imagine the desired API
Parting Advice
Slide 112
Slide 112 text
Imagine the desired API
Avoid hypothetical edge cases
Parting Advice
Slide 113
Slide 113 text
Parting Advice
Imagine the desired API
Avoid hypothetical edge cases
Cover edge cases as they appear
Slide 114
Slide 114 text
Thank You!
mattupstate.com
gittip.com/mattupstate
github.com/mattupstate
twitter.com/mattupstate
Slide 115
Slide 115 text
Annotations
[1] http://dictionary.reference.com/
[2] http://en.wikipedia.org/wiki/Software_testing