We are not perfect Mistakes in ● Implementation ● Assumptions ● Third-party code! Test as Documentation ● Requirements are vague ● Tests provide specification ● Regression tests
Property-based Testing Example-Based ● Specific scenarios ● Must think of boundary conditions ● White box (mostly) - refactoring? ● Deterministic Property-Based ● Higher-order properties ● Can help find boundary conditions ● Black box ● Non-deterministic
More non-deterministic tests Property-Based Testing Test cases are randomly generated (within given boundaries). Try to find counter-examples to properties. Fuzz Testing Program inputs are randomly generated. Try and cause crash. Mutation Testing Program code is mutated. Verify that test suite kills mutants!.
Generate Random test inputs generated (+ manual examples) Filter Only keep the data with the desired properties Test Run the test for each random data set Simplify On failure, shrink the failing inputs to find simplest failing case Report Report result of test runs