Testing in Python
The Big Picture
Pycon.de, Karlsruhe 24.10.2018 Niklas Meinzer
Slide 2
Slide 2 text
Who am I?
www.mps-med.de
Slide 3
Slide 3 text
Based in Freiburg
Pycon.de
Me!
Slide 4
Slide 4 text
4 years ago ...
Slide 5
Slide 5 text
Today?
Slide 6
Slide 6 text
No content
Slide 7
Slide 7 text
●
Why test?
●
How to test?
●
How am I doing?
Slide 8
Slide 8 text
Why test?
Slide 9
Slide 9 text
For your users/stakeholders
●
Is the product working according to specs?
●
Specialized tests:
– Performance
– Accessibility
– User Experience
Slide 10
Slide 10 text
For your colleagues
●
Tests as documentation
●
Easier code review
●
Easier quality assurance
Slide 11
Slide 11 text
For yourself
●
Know when you’re done (TDD)
●
Don’t be afraid to touch stuff
●
Have a more complete picture
of your product
– Functionality
– Performance
– Dependencies
Slide 12
Slide 12 text
How to test?
Slide 13
Slide 13 text
Classification
●
Functional tests
●
Other:
– Speed Testing
– Load Testing
– Best practice / style
– Security
Slide 14
Slide 14 text
Unit Tests
●
unittest
●
pytest
●
Nose
●
Lightweight
●
Run often
●
Make sure the most basic parts work correctly
Slide 15
Slide 15 text
Unit Tests - Example
Slide 16
Slide 16 text
Tox – supercharged Unit Tests
●
Standardize test execution
●
Run test suites for multiple
versions of Python
Slide 17
Slide 17 text
Hypothesis
●
Unittests need to be written by hand
●
With hypothesis you can define ranges of input
●
Hypothesis tries to find edge cases for you
Slide 18
Slide 18 text
hypothesis example
Slide 19
Slide 19 text
Supportive tools
Mock
●
Hide parts of code
●
Focus on the unit under test
Faker
●
Easily create real (enough) test data
●
Supports many locales
Slide 20
Slide 20 text
Behavior driven development
●
Python implementation: behave
●
Write tests in a more human
readable way
●
Focus on what is done, instead of how it’s
done
Slide 21
Slide 21 text
Behavior driven development
Slide 22
Slide 22 text
A note about correctness
●
Passing tests say nothing about correctness
●
Correct code can have failing test suites
●
Incorrect code can have passing test suites
●
Test can be wrong
●
Tests can be incomplete
●
Tests can be improperly configured
Slide 23
Slide 23 text
A note about correctness
What your program should do
What your test suite tests
●
Tests are a collection of assumptions on the
behavior of a program
Slide 24
Slide 24 text
Test Suite Quality
How can we improve the quality of a test suite?
●
Use tools which make things easy!
– Pytest parametrization
– Hypothesis
●
Analyze code coverage!
Slide 25
Slide 25 text
Code Coverage
●
Which part of the code got executed during a
specific run, e.g. a test run
●
Line coverage and branch coverage
●
coverage.py
Slide 26
Slide 26 text
Code Coverage
●
100% coverage means:
– All code was run
– NOT all code was tested
●
But: Uncovered code was certainly not
tested
●
Include test code in coverage analysis
– Catch shadowed tests
Slide 27
Slide 27 text
Mutation Testing
A way to test the specificity of a test suite
Assumption:
●
A good test suite should break if the code changes.
1) Change code
2) Run tests
3) Assume failure
Slide 28
Slide 28 text
cosmic ray
●
Python framework for mutation testing
●
Operators (classes of mutations):
– break/continue
– Change constants
– Flip booleans
●
Modifies AST
●
Run in celery
Slide 29
Slide 29 text
Manual Testing (QA)
●
Automation is not creative
●
Humans can identify unrelated issues
●
Some things are simply to hard to automate
●
Maintain test protocols
●
Test releases