Testing in Python - The Big Picture

Any team developing and maintaining software - be it free and open source or commercial - employs one form of software testing or another. But what are the different kinds of tests in our tool boxes? And how are they best used? In this talk we'll take a look around and try to answer these questions.

First, we'll examine the basic concepts of testing: Everyone has probably at least heard about unit tests, but are they all you need? Performance tests can help you find out how well your product performs under load and detect bottle necks early on. Manual testing is often looked down upon, since it's not automated, but is it always a bad idea? And what even is mutation testing?

We'll also get to know a lot of the amazing testing tools from the Python ecosystem. Find out what the best test runner is (Spoiler alert: it's pytest). Learn how to make writing test more fun and less work using tools like mock, Faker and factory_boy. Measure the quality of your test suite using coverage.py.

But no tool is the right one for any situation. We'll also talk about when and how to use each of the tools, while debunking common misconceptions and demonstrating best practices.

Transcript

1. Testing in Python The Big Picture Pycon.de, Karlsruhe 24.10.2018 Niklas Meinzer
2. Who am I? www.mps-med.de
3. Based in Freiburg Pycon.de Me!
4. 4 years ago ...
5. Today?
7. • Why test? • How to test? • How am I doing?
8. Why test?
9. For your users/stakeholders • Is the product working according to specs? • Specialized tests: – Performance – Accessibility – User Experience
10. For your colleagues • Tests as documentation • Easier code review • Easier quality assurance
11. For yourself • Know when you’re done (TDD) • Don’t be afraid to touch stuff • Have a more complete picture of your product – Functionality – Performance – Dependencies
12. How to test?
13. Classification • Functional tests • Other: – Speed Testing – Load Testing – Best practice / style – Security
14. Unit Tests • unittest • pytest • Nose • Lightweight • Run often • Make sure the most basic parts work correctly
15. Unit Tests - Example
16. Tox – supercharged Unit Tests • Standardize test execution • Run test suites for multiple versions of Python
17. Hypothesis • Unittests need to be written by hand • With hypothesis you can define ranges of input • Hypothesis tries to find edge cases for you
18. hypothesis example
19. Supportive tools Mock • Hide parts of code • Focus on the unit under test Faker • Easily create real (enough) test data • Supports many locales
20. Behavior driven development • Python implementation: behave • Write tests in a more human readable way • Focus on what is done, instead of how it’s done
21. Behavior driven development
22. A note about correctness • Passing tests say nothing about correctness • Correct code can have failing test suites • Incorrect code can have passing test suites • Test can be wrong • Tests can be incomplete • Tests can be improperly configured
23. A note about correctness What your program should do What your test suite tests • Tests are a collection of assumptions on the behavior of a program
24. Test Suite Quality How can we improve the quality of a test suite? • Use tools which make things easy! – Pytest parametrization – Hypothesis • Analyze code coverage!
25. Code Coverage • Which part of the code got executed during a specific run, e.g. a test run • Line coverage and branch coverage • coverage.py
26. Code Coverage • 100% coverage means: – All code was run – NOT all code was tested • But: Uncovered code was certainly not tested • Include test code in coverage analysis – Catch shadowed tests
27. Mutation Testing A way to test the specificity of a test suite Assumption: • A good test suite should break if the code changes. 1) Change code 2) Run tests 3) Assume failure
28. cosmic ray • Python framework for mutation testing • Operators (classes of mutations): – break/continue – Change constants – Flip booleans • Modifies AST • Run in celery
29. Manual Testing (QA) • Automation is not creative • Humans can identify unrelated issues • Some things are simply to hard to automate • Maintain test protocols • Test releases
30. Talk to me! @SevenOfNein@chaos.social @NiklasMM www.niklas-meinzer.de