Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing in Python - The Big Picture

Testing in Python - The Big Picture

Any team developing and maintaining software - be it free and open source or commercial - employs one form of software testing or another. But what are the different kinds of tests in our tool boxes? And how are they best used? In this talk we'll take a look around and try to answer these questions.

First, we'll examine the basic concepts of testing: Everyone has probably at least heard about unit tests, but are they all you need? Performance tests can help you find out how well your product performs under load and detect bottle necks early on. Manual testing is often looked down upon, since it's not automated, but is it always a bad idea? And what even is mutation testing?

We'll also get to know a lot of the amazing testing tools from the Python ecosystem. Find out what the best test runner is (Spoiler alert: it's pytest). Learn how to make writing test more fun and less work using tools like mock, Faker and factory_boy. Measure the quality of your test suite using coverage.py.

But no tool is the right one for any situation. We'll also talk about when and how to use each of the tools, while debunking common misconceptions and demonstrating best practices.

Niklas Meinzer

October 24, 2018

More Decks by Niklas Meinzer

Other Decks in Programming


  1. Testing in Python
    The Big Picture
    Pycon.de, Karlsruhe 24.10.2018 Niklas Meinzer

    View Slide

  2. Who am I?

    View Slide

  3. Based in Freiburg

    View Slide

  4. 4 years ago ...

    View Slide

  5. Today?

    View Slide

  6. View Slide

  7. Why test?

    How to test?

    How am I doing?

    View Slide

  8. Why test?

    View Slide

  9. For your users/stakeholders

    Is the product working according to specs?

    Specialized tests:
    – Performance
    – Accessibility
    – User Experience

    View Slide

  10. For your colleagues

    Tests as documentation

    Easier code review

    Easier quality assurance

    View Slide

  11. For yourself

    Know when you’re done (TDD)

    Don’t be afraid to touch stuff

    Have a more complete picture
    of your product
    – Functionality
    – Performance
    – Dependencies

    View Slide

  12. How to test?

    View Slide

  13. Classification

    Functional tests

    – Speed Testing
    – Load Testing
    – Best practice / style
    – Security

    View Slide

  14. Unit Tests





    Run often

    Make sure the most basic parts work correctly

    View Slide

  15. Unit Tests - Example

    View Slide

  16. Tox – supercharged Unit Tests

    Standardize test execution

    Run test suites for multiple
    versions of Python

    View Slide

  17. Hypothesis

    Unittests need to be written by hand

    With hypothesis you can define ranges of input

    Hypothesis tries to find edge cases for you

    View Slide

  18. hypothesis example

    View Slide

  19. Supportive tools

    Hide parts of code

    Focus on the unit under test

    Easily create real (enough) test data

    Supports many locales

    View Slide

  20. Behavior driven development

    Python implementation: behave

    Write tests in a more human
    readable way

    Focus on what is done, instead of how it’s

    View Slide

  21. Behavior driven development

    View Slide

  22. A note about correctness

    Passing tests say nothing about correctness

    Correct code can have failing test suites

    Incorrect code can have passing test suites

    Test can be wrong

    Tests can be incomplete

    Tests can be improperly configured

    View Slide

  23. A note about correctness
    What your program should do
    What your test suite tests

    Tests are a collection of assumptions on the
    behavior of a program

    View Slide

  24. Test Suite Quality
    How can we improve the quality of a test suite?

    Use tools which make things easy!
    – Pytest parametrization
    – Hypothesis

    Analyze code coverage!

    View Slide

  25. Code Coverage

    Which part of the code got executed during a
    specific run, e.g. a test run

    Line coverage and branch coverage


    View Slide

  26. Code Coverage

    100% coverage means:
    – All code was run
    – NOT all code was tested

    But: Uncovered code was certainly not

    Include test code in coverage analysis
    – Catch shadowed tests

    View Slide

  27. Mutation Testing
    A way to test the specificity of a test suite

    A good test suite should break if the code changes.
    1) Change code
    2) Run tests
    3) Assume failure

    View Slide

  28. cosmic ray

    Python framework for mutation testing

    Operators (classes of mutations):
    – break/continue
    – Change constants
    – Flip booleans

    Modifies AST

    Run in celery

    View Slide

  29. Manual Testing (QA)

    Automation is not creative

    Humans can identify unrelated issues

    Some things are simply to hard to automate

    Maintain test protocols

    Test releases

    View Slide

  30. Talk to me!
    @[email protected]

    View Slide