Testing Flask Applications with pytest

Testing a Flask application helps ensure that your app will work as expected for your end users. This talk details how to create and run tests with pytest, utilize fixtures to initialize the state of the Flask app, and check the coverage of the tests using coverage.py.

Transcript

1. Testing
   Flask Applications
   with pytest
   Patrick Kennedy
   1
   

   View Slide

2. Outline
   ● What should be tested in a Flask application?
   ● Writing Flask-specific tests with pytest
   ● Running tests with pytest
   ● Fixtures
   ● Coverage
   2
   

   View Slide

3. Introduction
   3
   

   View Slide

4. Why Write Tests?
   4
   ● Testing helps ensure that your app will work as expected for your
   end users
   ○ Fundamental part of the Test-Driven Development (TDD) process
   ● Testing should be combined with a Continuous Integration (CI)
   process
   ○ Tests are automatically run on code changes
   ○ Helps catch problems before end users see them in production
   More about TDD: https://testdriven.io/test-driven-development/
   

   View Slide

5. What Should be Tested in a Flask App?
   ● Unit tests:
   ○ Focus on testing small units of code in isolation
   ○ Examples:
   ■ Database models
   ■ Utility functions called by view functions
   ● Functional tests:
   ○ Focus on how the view functions operate under different conditions
   ○ Examples:
   ■ Nominal conditions (GET, POST, etc.)
   ■ Invalid HTTP methods
   ■ Invalid input data
   5
   Unit Tests
   Functional Tests
   End-to-End
   Tests
   Info about testing pyramid:
   https://martinfowler.com/articles/
   practical-test-pyramid.html
   

   View Slide

6. What is pytest?
   ● pytest is a test framework for Python:
   ○ used to write, organize, and run test cases
   ○ provides a lot of flexibility for running the tests
   ● pytest satisfies the key aspects of a great test environment:
   ○ tests are fun to write
   ○ tests can be written quickly by using helper functions (fixtures)
   ○ tests can be executed with a single command
   ○ tests run quickly
   6
   pytest documentation: https://docs.pytest.org/en/latest/
   Installation:
   (venv) $ pip install pytest
   

   View Slide

7. Writing Tests with pytest
   7
   

   View Slide

8. Flask User Management Example
   8
   ● Example Flask application:
   ○ Homepage
   ○ Register new users
   ○ Log in/out users
   ○ User Profile
   Source code for examples: https://gitlab.com/patkennedy79/flask_user_management_example
   

   View Slide

9. ├── app.py
   ├── project
   │ ├── __init__.py
   │ ├── models.py
   │ └── ...blueprint folders...
   ├── requirements.txt
   ├── tests
   │ ├── conftest.py
   │ ├── functional
   │ │ ├── __init__.py
   │ │ ├── test_recipes.py
   │ │ └── test_users.py
   │ └── unit
   │ ├── __init__.py
   │ └── test_models.py
   └── venv
   ● tests folder contains all the
   tests for the project
   ● Tests are divided into different
   folders for:
   ○ functional tests
   ○ unit tests
   Organization
   9
   

   View Slide

10. Writing a Unit Test (1 / 2)
    from project import db
    from werkzeug.security import generate_password_hash, check_password_hash
    class User(db.Model):
    __tablename__ = 'users'
    id = db.Column(db.Integer, primary_key =True, autoincrement =True)
    email = db.Column(db.String, unique=True, nullable=False)
    password_hashed = db.Column(db.String( 128), nullable=False)
    def __init__(self, email: str, password_plaintext: str):
    """Create a new User object using the email address and hashing the
    plaintext password using Werkzeug.Security.
    """
    self.email = email
    self.password_hashed = self._generate_password_hash(password_plaintext)
    @staticmethod
    def _generate_password_hash (password_plaintext: str):
    return generate_password_hash(password_plaintext)
    10
    Can a User object be
    created successfully?
    

    View Slide

11. Writing a Unit Test (2 / 2)
    from project.models import User
    def test_new_user():
    """
    GIVEN a User model
    WHEN a new User is created
    THEN check the email and password_hashed fields are defined correctly
    """
    user = User('patkennedy79@gmail.com', 'FlaskIsAwesome')
    assert user.email == 'patkennedy79@gmail.com'
    assert user.password_hashed != 'FlaskIsAwesome'
    11
    Create a new `User` object
    Using `assert`, check the following:
    1. Check that the email is stored correctly
    2. Check that the hashed password does not
    equal the plaintext password
    tests/unit/test_models.py
    

    View Slide

12. ● Maintenance is a challenging aspect of software development
    ○ Documenting tests is often lacking or missing
    ● GIVEN / WHEN / THEN structure provides a common structure for
    describing tests
    ● GIVEN - what are the initial conditions for the test?
    ● WHEN - what is occurring that needs to be tested?
    ● THEN - what is the expected response?
    Documenting Tests
    12
    More about GIVEN/WHEN/THEN: https://martinfowler.com/bliki/GivenWhenThen.html
    

    View Slide

13. @recipes_blueprint.route ('/')
    def index():
    return render_template('recipes/index.html')
    Writing a Functional Test (1 / 2)
    13
    

    View Slide

14. Writing a Functional Test (2 / 2)
    14
    from project import create_app
    def test_home_page():
    """
    GIVEN a Flask application configured for testing
    WHEN the '/' page is requested (GET)
    THEN check that the response is valid
    """
    flask_app = create_app('flask_test.cfg')
    # Create a test client using the Flask application configured for testing
    with flask_app.test_client() as test_client:
    response = test_client.get('/')
    assert response.status_code == 200
    assert b’Flask User Management Example!’ in response.data
    assert b’Need an account?’ in response.data
    assert b’Existing user?’ in response.data
    Create a Flask application
    (assumes the Application
    Factory approach)
    Using `assert`, check the following:
    1. 200 (OK) status returned
    2. Expected text is in the response
    Utilize the `test_client` included with Flask
    to GET the homepage (‘/’ route)
    tests/functional/test_users.py
    

    View Slide

15. def test_home_page_post():
    """
    GIVEN a Flask application configured for testing
    WHEN the '/' page is is posted to (POST)
    THEN check that a '405' status code is returned
    """
    flask_app = create_app('flask_test.cfg')
    # Create a test client using the Flask application configured for testing
    with flask_app.test_client() as test_client:
    response = test_client.post('/')
    assert response.status_code == 405
    assert b’Flask User Management Example!’ not in response.data
    Writing a Functional Test - Off-Nominal
    15
    Using `assert`, check the following:
    1. 405 (Method Not Allowed) status returned
    2. Header text is NOT in the response
    Utilize the `test_client` included with Flask
    to POST to the homepage (‘/’ route)
    tests/functional/test_users.py
    

    View Slide

16. Running Tests with pytest
    16
    

    View Slide

17. $ python -m pytest
    ============================== test session starts ==============================
    tests/functional/test_recipes.py .... [ 26%]
    tests/functional/test_users.py ....... [ 73%]
    tests/unit/test_models.py .... [100%]
    ============================== 15 passed in 1.13s ===============================
    Running pytest (1 / 2)
    17
    ● Recommend running pytest through the Python interpreter
    ○ https://docs.pytest.org/en/6.2.x/usage.html#calling-pytest-through-python-m-pytest
    

    View Slide

18. Running pytest (2 / 2)
    ● `-v ` flag provides verbose output about the tests run:
    ● Specific types of tests can be run:
    ● Only run the failed tests from the last time pytest was run:
    18
    $ python -m pytest --last-failed
    $ python -m pytest tests/unit
    $ python -m pytest -v
    ========================== test session starts ================================
    tests/functional/test_recipes.py::test_home_page PASSED [ 6%]
    ...
    tests/unit/test_models.py::test_setting_password PASSED [ 93%]
    tests/unit/test_models.py::test_user_id PASSED [100%]
    ============================= 15 passed in 1.05s ==============================
    

    View Slide

19. Fixtures
    19
    

    View Slide

20. What is a Fixture?
    20
    ● Fixtures initialize tests to a known state:
    ○ Run tests in a predictable and repeatable manner
    ● Fixtures are defined as functions (with a descriptive name!)
    ● Multiple fixture can be run to initialize a test
    ○ Fixtures can even call other fixtures!
    ● Fixtures should be defined in `tests/conftest.py`
    

    View Slide

21. Scope of Fixtures
    21
    ● Scope defines WHEN a fixture will be run
    ● function - run once per test function (default scope)
    ● class - run once per test class
    ● module - run once per module (i.e. test file)
    ● session - run once per session (i.e. per call to pytest)
    

    View Slide

22. def test_new_user_with_fixture(new_user):
    """
    GIVEN a User model
    WHEN a new User is created
    THEN check the email and hashed_password are defined correctly
    """
    assert new_user.email == 'patkennedy79@gmail.com'
    assert new_user.hashed_password != 'FlaskIsAwesome'
    from project.models import User
    @pytest.fixture(scope='module')
    def new_user():
    user = User('patkennedy79@gmail.com', 'FlaskIsAwesome')
    return user
    Fixture to Help with Unit Testing
    22
    tests/unit/test_models.py
    tests/conftest.py
    

    View Slide

23. Fixture to Help with Functional Testing
    23
    from project import create_app
    @pytest.fixture(scope ='module')
    def test_client():
    flask_app = create_app( 'flask_test.cfg' )
    # Create a test client using the Flask application configured for testing
    with flask_app.test_client() as testing_client:
    yield testing_client # this is where the testing happens!
    def test_home_page_post_with_fixture (test_client):
    """
    GIVEN a Flask application
    WHEN the '/' page is is posted to (POST)
    THEN check that a '405' status code is returned
    """
    response = test_client .post('/')
    assert response.status_code == 405
    assert b’Flask User Management Example!’ not in response.data
    tests/functional/test_users.py
    tests/conftest.py
    

    View Slide

24. Fixtures in Action!
    24
    $ python -m pytest --setup-show tests/functional/test_users.py
    ================================== test session starts =================================
    tests/functional/test_recipes.py
    ...
    SETUP M test_client
    functional/test_users.py::test_home_page_with_fixture (fixtures used: test_client).
    ...
    TEARDOWN M test_client
    =================================== 4 passed in 0.18s ==================================
    Step 1 - test_client fixture is executed
    Step 3 - Execution returns to test_client fixture -> Done!
    Step 2 - test executes!
    

    View Slide

25. Coverage
    25
    

    View Slide

26. ● Code coverage is a metric of how much source code is executed when
    running the tests
    ● Two excellent Python packages:
    ○ coverage.py - tool for measuring code coverage of Python programs
    ○ pytest-cov - pytest plugin for running coverage
    Code Coverage - Introduction
    26
    Coverage.py (by Ned Batcheldor): https://coverage.readthedocs.io/
    Pytest-cov: https://pytest-cov.readthedocs.io/
    

    View Slide

27. Checking Coverage in pytest
    27
    $ python -m pytest --cov=project
    ================================= test session starts ==================================
    tests/functional/test_recipes.py .... [ 26%]
    tests/functional/test_users.py ....... [ 73%]
    tests/unit/test_models.py .... [100%]
    ---------- coverage: platform darwin, python 3.10.0-final-0 ----------
    Name Stmts Miss Cover
    -------------------------------------------------
    project/__init__.py 24 0 100%
    project/models.py 33 0 100%
    project/recipes/__init__.py 3 0 100%
    project/recipes/routes.py 5 0 100%
    project/users/__init__.py 3 0 100%
    project/users/forms.py 13 0 100%
    project/users/routes.py 47 0 100%
    -------------------------------------------------
    TOTAL 128 0 100%
    ================================ 15 passed in 1.36s ===================================
    Any lines in the source code that are
    not executed are identified here
    Specify the directory containing the source code
    

    View Slide

28. Conclusion
    28
    

    View Slide

29. Conclusion
    ● pytest helps simplify testing Flask applications
    ● Fixtures provide a consistent approach for initializing state for tests
    ● Coverage is easy to collect with coverage.py and pytest-cov
    29
    For more complex examples of fixtures and test functions:
    https://gitlab.com/patkennedy79/flask_user_management_example
    “Testing Flask Applications with pytest” blog post:
    https://testdriven.io/blog/flask-pytest/
    

    View Slide

30. Thank you!
    30
    Patrick Kennedy
    Email: patkennedy79@gmail.com
    Twitter: patkennedy79
    Personal Blog: https://www.patricksoftwareblog.com
    Learn Flask by Building and Deploying a Stock Portfolio App:
    https://testdriven.io/courses/learn-flask/
    

    View Slide