Testing in Python - The Big Picture

Testing in Python - The Big Picture

Any team developing and maintaining software - be it free and open source or commercial - employs one form of software testing or another. But what are the different kinds of tests in our tool boxes? And how are they best used? In this talk we'll take a look around and try to answer these questions.

First, we'll examine the basic concepts of testing: Everyone has probably at least heard about unit tests, but are they all you need? Performance tests can help you find out how well your product performs under load and detect bottle necks early on. Manual testing is often looked down upon, since it's not automated, but is it always a bad idea? And what even is mutation testing?

We'll also get to know a lot of the amazing testing tools from the Python ecosystem. Find out what the best test runner is (Spoiler alert: it's pytest). Learn how to make writing test more fun and less work using tools like mock, Faker and factory_boy. Measure the quality of your test suite using coverage.py.

But no tool is the right one for any situation. We'll also talk about when and how to use each of the tools, while debunking common misconceptions and demonstrating best practices.


Niklas Meinzer

October 24, 2018


  1. Testing in Python The Big Picture Pycon.de, Karlsruhe 24.10.2018 Niklas

  2. Who am I? www.mps-med.de

  3. Based in Freiburg Pycon.de Me!

  4. 4 years ago ...

  5. Today?

  6. None
  7. • Why test? • How to test? • How am

    I doing?
  8. Why test?

  9. For your users/stakeholders • Is the product working according to

    specs? • Specialized tests: – Performance – Accessibility – User Experience
  10. For your colleagues • Tests as documentation • Easier code

    review • Easier quality assurance
  11. For yourself • Know when you’re done (TDD) • Don’t

    be afraid to touch stuff • Have a more complete picture of your product – Functionality – Performance – Dependencies
  12. How to test?

  13. Classification • Functional tests • Other: – Speed Testing –

    Load Testing – Best practice / style – Security
  14. Unit Tests • unittest • pytest • Nose • Lightweight

    • Run often • Make sure the most basic parts work correctly
  15. Unit Tests - Example

  16. Tox – supercharged Unit Tests • Standardize test execution •

    Run test suites for multiple versions of Python
  17. Hypothesis • Unittests need to be written by hand •

    With hypothesis you can define ranges of input • Hypothesis tries to find edge cases for you
  18. hypothesis example

  19. Supportive tools Mock • Hide parts of code • Focus

    on the unit under test Faker • Easily create real (enough) test data • Supports many locales
  20. Behavior driven development • Python implementation: behave • Write tests

    in a more human readable way • Focus on what is done, instead of how it’s done
  21. Behavior driven development

  22. A note about correctness • Passing tests say nothing about

    correctness • Correct code can have failing test suites • Incorrect code can have passing test suites • Test can be wrong • Tests can be incomplete • Tests can be improperly configured
  23. A note about correctness What your program should do What

    your test suite tests • Tests are a collection of assumptions on the behavior of a program
  24. Test Suite Quality How can we improve the quality of

    a test suite? • Use tools which make things easy! – Pytest parametrization – Hypothesis • Analyze code coverage!
  25. Code Coverage • Which part of the code got executed

    during a specific run, e.g. a test run • Line coverage and branch coverage • coverage.py
  26. Code Coverage • 100% coverage means: – All code was

    run – NOT all code was tested • But: Uncovered code was certainly not tested • Include test code in coverage analysis – Catch shadowed tests
  27. Mutation Testing A way to test the specificity of a

    test suite Assumption: • A good test suite should break if the code changes. 1) Change code 2) Run tests 3) Assume failure
  28. cosmic ray • Python framework for mutation testing • Operators

    (classes of mutations): – break/continue – Change constants – Flip booleans • Modifies AST • Run in celery
  29. Manual Testing (QA) • Automation is not creative • Humans

    can identify unrelated issues • Some things are simply to hard to automate • Maintain test protocols • Test releases
  30. Talk to me! @SevenOfNein@chaos.social @NiklasMM www.niklas-meinzer.de