Property-based Testing in Python with Hypothesis

A talk given at GeekCamp.id, 24 October 2015.

Examples used in the presentation available at https://github.com/michel-slm/talk--property-based-testing--hypothesis/

Transcript

1. Property-based Testing
   with Hypothesis
   Michel Alexandre Salim • michel@traveloka.com
   

   View Slide

2. Overview
   Why Test?
   From Examples to Properties
   Taking a Deeper Look
   

   View Slide

3. Why Test?
   

   View Slide

4. We are not perfect
   Mistakes in
   ● Implementation
   ● Assumptions
   ● Third-party code!
   Test as Documentation
   ● Requirements are vague
   ● Tests provide specification
   ● Regression tests
   

   View Slide

5. From Examples to Properties
   

   View Slide

6. All Swans are White ... or are they?
   

   View Slide

7. Property-based Testing
   Example-Based
   ● Specific scenarios
   ● Must think of boundary conditions
   ● White box (mostly) - refactoring?
   ● Deterministic
   Property-Based
   ● Higher-order properties
   ● Can help find boundary conditions
   ● Black box
   ● Non-deterministic
   

   View Slide

8. More non-deterministic tests
   Property-Based Testing
   Test cases are randomly generated (within given boundaries).
   Try to find counter-examples to properties.
   Fuzz Testing
   Program inputs are randomly generated. Try and cause crash.
   Mutation Testing
   Program code is mutated. Verify that test suite kills mutants!.
   

   View Slide

9. Taking a Deeper Look
   

   View Slide

10. Hypothesis in Context
    History
    ● 1999: QuickCheck (Haskell)
    ● 2007: ScalaCheck
    ● 2013: Hypothesis
    ● 2015: Hypothesis 1.0
    

    View Slide

11. Generate
    Random test inputs
    generated (+ manual
    examples)
    Filter
    Only keep the data
    with the desired
    properties
    Test
    Run the test for each
    random data set
    Simplify
    On failure, shrink the
    failing inputs to find
    simplest failing case
    Report
    Report result of test
    runs
    

    View Slide

12. Because a live demonstration is worth a thousand words...
    

    View Slide

13. Online Resources
    Hypothesis Documentation
    https://hypothesis.readthedocs.org/en/latest/
    pytest
    http://pytest.org/latest/
    Examples from this presentation
    https://github.com/michel-slm/talk--property-based-testing--hypothesis/
    

    View Slide

14. Questions?
    

    View Slide