Testing in Python - The Big Picture

Any team developing and maintaining software - be it free and open source or commercial - employs one form of software testing or another. But what are the different kinds of tests in our tool boxes? And how are they best used? In this talk we'll take a look around and try to answer these questions.

First, we'll examine the basic concepts of testing: Everyone has probably at least heard about unit tests, but are they all you need? Performance tests can help you find out how well your product performs under load and detect bottle necks early on. Manual testing is often looked down upon, since it's not automated, but is it always a bad idea? And what even is mutation testing?

We'll also get to know a lot of the amazing testing tools from the Python ecosystem. Find out what the best test runner is (Spoiler alert: it's pytest). Learn how to make writing test more fun and less work using tools like mock, Faker and factory_boy. Measure the quality of your test suite using coverage.py.

But no tool is the right one for any situation. We'll also talk about when and how to use each of the tools, while debunking common misconceptions and demonstrating best practices.

Transcript

1. Testing in Python
   The Big Picture
   Pycon.de, Karlsruhe 24.10.2018 Niklas Meinzer
   

   View Slide

2. Who am I?
   www.mps-med.de
   

   View Slide

3. Based in Freiburg
   Pycon.de
   Me!
   

   View Slide

4. 4 years ago ...
   

   View Slide

5. Today?
   

   View Slide

6. View Slide

7. ●
   Why test?
   ●
   How to test?
   ●
   How am I doing?
   

   View Slide

8. Why test?
   

   View Slide

9. For your users/stakeholders
   ●
   Is the product working according to specs?
   ●
   Specialized tests:
   – Performance
   – Accessibility
   – User Experience
   

   View Slide

10. For your colleagues
    ●
    Tests as documentation
    ●
    Easier code review
    ●
    Easier quality assurance
    

    View Slide

11. For yourself
    ●
    Know when you’re done (TDD)
    ●
    Don’t be afraid to touch stuff
    ●
    Have a more complete picture
    of your product
    – Functionality
    – Performance
    – Dependencies
    

    View Slide

12. How to test?
    

    View Slide

13. Classification
    ●
    Functional tests
    ●
    Other:
    – Speed Testing
    – Load Testing
    – Best practice / style
    – Security
    

    View Slide

14. Unit Tests
    ●
    unittest
    ●
    pytest
    ●
    Nose
    ●
    Lightweight
    ●
    Run often
    ●
    Make sure the most basic parts work correctly
    

    View Slide

15. Unit Tests - Example
    

    View Slide

16. Tox – supercharged Unit Tests
    ●
    Standardize test execution
    ●
    Run test suites for multiple
    versions of Python
    

    View Slide

17. Hypothesis
    ●
    Unittests need to be written by hand
    ●
    With hypothesis you can define ranges of input
    ●
    Hypothesis tries to find edge cases for you
    

    View Slide

18. hypothesis example
    

    View Slide

19. Supportive tools
    Mock
    ●
    Hide parts of code
    ●
    Focus on the unit under test
    Faker
    ●
    Easily create real (enough) test data
    ●
    Supports many locales
    

    View Slide

20. Behavior driven development
    ●
    Python implementation: behave
    ●
    Write tests in a more human
    readable way
    ●
    Focus on what is done, instead of how it’s
    done
    

    View Slide

21. Behavior driven development
    

    View Slide

22. A note about correctness
    ●
    Passing tests say nothing about correctness
    ●
    Correct code can have failing test suites
    ●
    Incorrect code can have passing test suites
    ●
    Test can be wrong
    ●
    Tests can be incomplete
    ●
    Tests can be improperly configured
    

    View Slide

23. A note about correctness
    What your program should do
    What your test suite tests
    ●
    Tests are a collection of assumptions on the
    behavior of a program
    

    View Slide

24. Test Suite Quality
    How can we improve the quality of a test suite?
    ●
    Use tools which make things easy!
    – Pytest parametrization
    – Hypothesis
    ●
    Analyze code coverage!
    

    View Slide

25. Code Coverage
    ●
    Which part of the code got executed during a
    specific run, e.g. a test run
    ●
    Line coverage and branch coverage
    ●
    coverage.py
    

    View Slide

26. Code Coverage
    ●
    100% coverage means:
    – All code was run
    – NOT all code was tested
    ●
    But: Uncovered code was certainly not
    tested
    ●
    Include test code in coverage analysis
    – Catch shadowed tests
    

    View Slide

27. Mutation Testing
    A way to test the specificity of a test suite
    Assumption:
    ●
    A good test suite should break if the code changes.
    1) Change code
    2) Run tests
    3) Assume failure
    

    View Slide

28. cosmic ray
    ●
    Python framework for mutation testing
    ●
    Operators (classes of mutations):
    – break/continue
    – Change constants
    – Flip booleans
    ●
    Modifies AST
    ●
    Run in celery
    

    View Slide

29. Manual Testing (QA)
    ●
    Automation is not creative
    ●
    Humans can identify unrelated issues
    ●
    Some things are simply to hard to automate
    ●
    Maintain test protocols
    ●
    Test releases
    

    View Slide

30. Talk to me!
    @SevenOfNein@chaos.social
    @NiklasMM
    www.niklas-meinzer.de
    

    View Slide