Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing in Python - The Big Picture

Testing in Python - The Big Picture

Any team developing and maintaining software - be it free and open source or commercial - employs one form of software testing or another. But what are the different kinds of tests in our tool boxes? And how are they best used? In this talk we'll take a look around and try to answer these questions.

First, we'll examine the basic concepts of testing: Everyone has probably at least heard about unit tests, but are they all you need? Performance tests can help you find out how well your product performs under load and detect bottle necks early on. Manual testing is often looked down upon, since it's not automated, but is it always a bad idea? And what even is mutation testing?

We'll also get to know a lot of the amazing testing tools from the Python ecosystem. Find out what the best test runner is (Spoiler alert: it's pytest). Learn how to make writing test more fun and less work using tools like mock, Faker and factory_boy. Measure the quality of your test suite using coverage.py.

But no tool is the right one for any situation. We'll also talk about when and how to use each of the tools, while debunking common misconceptions and demonstrating best practices.

Niklas Meinzer

October 24, 2018

More Decks by Niklas Meinzer

Other Decks in Programming


  1. For your users/stakeholders • Is the product working according to

    specs? • Specialized tests: – Performance – Accessibility – User Experience
  2. For yourself • Know when you’re done (TDD) • Don’t

    be afraid to touch stuff • Have a more complete picture of your product – Functionality – Performance – Dependencies
  3. Classification • Functional tests • Other: – Speed Testing –

    Load Testing – Best practice / style – Security
  4. Unit Tests • unittest • pytest • Nose • Lightweight

    • Run often • Make sure the most basic parts work correctly
  5. Tox – supercharged Unit Tests • Standardize test execution •

    Run test suites for multiple versions of Python
  6. Hypothesis • Unittests need to be written by hand •

    With hypothesis you can define ranges of input • Hypothesis tries to find edge cases for you
  7. Supportive tools Mock • Hide parts of code • Focus

    on the unit under test Faker • Easily create real (enough) test data • Supports many locales
  8. Behavior driven development • Python implementation: behave • Write tests

    in a more human readable way • Focus on what is done, instead of how it’s done
  9. A note about correctness • Passing tests say nothing about

    correctness • Correct code can have failing test suites • Incorrect code can have passing test suites • Test can be wrong • Tests can be incomplete • Tests can be improperly configured
  10. A note about correctness What your program should do What

    your test suite tests • Tests are a collection of assumptions on the behavior of a program
  11. Test Suite Quality How can we improve the quality of

    a test suite? • Use tools which make things easy! – Pytest parametrization – Hypothesis • Analyze code coverage!
  12. Code Coverage • Which part of the code got executed

    during a specific run, e.g. a test run • Line coverage and branch coverage • coverage.py
  13. Code Coverage • 100% coverage means: – All code was

    run – NOT all code was tested • But: Uncovered code was certainly not tested • Include test code in coverage analysis – Catch shadowed tests
  14. Mutation Testing A way to test the specificity of a

    test suite Assumption: • A good test suite should break if the code changes. 1) Change code 2) Run tests 3) Assume failure
  15. cosmic ray • Python framework for mutation testing • Operators

    (classes of mutations): – break/continue – Change constants – Flip booleans • Modifies AST • Run in celery
  16. Manual Testing (QA) • Automation is not creative • Humans

    can identify unrelated issues • Some things are simply to hard to automate • Maintain test protocols • Test releases