Upgrade to Pro — share decks privately, control downloads, hide ads and more …

An introduction to property based testing

An introduction to property based testing

Video and more content at fsharpforfunandprofit.com/pbt

"The lazy programmer's guide to writing 1000's of tests: An introduction to property based testing"

We are all familiar with example-based testing, as typified by TDD and BDD. Property-based testing takes a very different approach, where a single test is run hundreds of times with randomly generated inputs.

Property-based testing is a great way to find edge cases, and also helps you to understand and document the behaviour of your code under all conditions.

This talk will introduce property-based testing and show how it works, and why you should consider adding it to your arsenal of testing tools.

Scott Wlaschin

June 10, 2015
Tweet

More Decks by Scott Wlaschin

Other Decks in Programming

Transcript

  1. The lazy programmer's guide to
    writing 1000's of tests
    An introduction to property based testing
    @ScottWlaschin
    fsharpforfunandprofit.com

    View full-size slide

  2. Part 1:
    In which I have a conversation with a
    remote developer
    This was a project from a long time ago,
    in a galaxy far far away

    View full-size slide

  3. For some reason
    we needed a
    custom "add"
    function

    View full-size slide

  4. ...some time later

    View full-size slide

  5. So I decide to start writing the
    unit tests myself

    View full-size slide

  6. []
    let ``When I add 1 + 3, I expect 4``()=
    let result = add 1 3
    Assert.AreEqual(4,result)
    []
    let ``When I add 2 + 2, I expect 4``()=
    let result = add 2 2
    Assert.AreEqual(4,result)


    First, I had a look at the existing tests...

    View full-size slide

  7. []
    let ``When I add -1 + 3, I expect 2``()=
    let result = add -1 3
    Assert.AreEqual(2,result)

    Ok, now for my first new test...

    View full-size slide

  8. let add x y =
    4
    wtf!
    Hmm.. let's look at the implementation...

    View full-size slide

  9. []
    let ``When I add 2 + 3, I expect 5``()=
    let result = add 2 3
    Assert.AreEqual(5,result)
    []
    let ``When I add 1 + 41, I expect 42``()=
    let result = add 1 41
    Assert.AreEqual(42,result)


    Time for some more tests...

    View full-size slide

  10. let add x y =
    match (x,y) with
    | (2,3) -> 5
    | (1,41) -> 42
    | (_,_) -> 4 // all other cases
    Let's just check the implementation again...

    View full-size slide

  11. Write the minimal code that will make the
    test pass
    At this point you need to write code that will
    successfully pass the test.
    The code written at this stage will not be 100%
    final, you will improve it later stages.
    Do not try to write the perfect code at this stage,
    just write code that will pass the test.
    From http://www.typemock.com/test-driven-development-tdd/
    TDD best practices

    View full-size slide

  12. []
    let ``When I add two numbers,
    I expect to get their sum``()=
    for (x,y,expected) in [
    (1,2,3);
    (2,2,4);
    (3,5,8);
    (27,15,42); ]
    let actual = add x y
    Assert.AreEqual(expected,actual)
    Another attempt at a test

    View full-size slide

  13. let add x y =
    match (x,y) with
    | (1,2) -> 3
    | (2,3) -> 5
    | (3,5) -> 8
    | (1,41) -> 42
    | (25,15) -> 42
    | (_,_) -> 4 // all other cases
    Let's check the implementation one more time....

    View full-size slide

  14. It dawned on me who I was
    dealing with...
    ...the legendary burned-out, always lazy and
    often malicious programmer called...

    View full-size slide

  15. The Enterprise
    Developer From Hell

    View full-size slide

  16. Rethinking the approach
    The EDFH will always make
    specific examples pass, no
    matter what I do...
    So let's not use
    specific examples!

    View full-size slide

  17. []
    let ``When I add two random numbers,
    I expect their sum to be correct``()=
    let x = randInt()
    let y = randInt()
    let expected = x + y
    let actual = add x y
    Assert.AreEqual(expected,actual)
    Let's use random numbers instead...

    View full-size slide

  18. []
    let ``When I add two random numbers (100 times),
    I expect their sum to be correct``()=
    for _ in [1..100] do
    let x = randInt()
    let y = randInt()
    let expected = x + y
    let actual = add x y
    Assert.AreEqual(expected,actual)
    Yea! Problem solved!
    And why not do it 100 times just to be sure...
    The EDFH can't beat this!

    View full-size slide

  19. []
    let ``When I add two random numbers (100 times),
    I expect their sum to be correct``()=
    for _ in [1..100] do
    let x = randInt()
    let y = randInt()
    let expected = x + y
    let actual = add x y
    Assert.AreEqual(expected,actual)
    Uh-oh!
    But if you can't test by using +, how CAN you test?
    We can't test "add" using +!

    View full-size slide

  20. Part II:
    Property based testing

    View full-size slide

  21. What are the "requirements" for
    the "add" function?
    It's hard to know where to get started, but one
    approach is to compare it with something different...
    How does "add" differ from "subtract", for example?

    View full-size slide

  22. []
    let ``When I add two numbers, the result
    should not depend on parameter order``()=
    for _ in [1..100] do
    let x = randInt()
    let y = randInt()
    let result1 = add x y
    let result2 = add y x
    Assert.AreEqual(result1,result2)
    reversed params
    So how does "add" differ from "subtract"?
    For "subtract", the order of the parameters makes a
    difference, while for "add" it doesn't.

    View full-size slide

  23. let add x y =
    x * y
    The EDFH responds with:

    TEST: ``When I add two numbers, the result
    should not depend on parameter order``

    View full-size slide

  24. Ok, what's the difference
    between add and multiply?
    Example: two "add 1"s is the same as one "add 2".

    View full-size slide

  25. []
    let ``Adding 1 twice is the same as adding 2``()=
    for _ in [1..100] do
    let x = randInt()
    let y = randInt()
    let result1 = x |> add 1 |> add 1
    let result2 = x |> add 2
    Assert.AreEqual(result1,result2)
    Test: two "add 1"s is the same as one "add 2".

    View full-size slide

  26. let add x y =
    x - y
    The EDFH responds with:


    TEST: ``When I add two numbers, the result
    should not depend on parameter order``
    TEST: ``Adding 1 twice is the same as adding 2``
    Ha! Gotcha, EDFH!
    But luckily we have the previous test as well!

    View full-size slide

  27. let add x y =
    0
    The EDFH responds with another implementation:

    TEST: ``When I add two numbers, the result
    should not depend on parameter order``
    TEST: ``Adding 1 twice is the same as adding 2``

    Aarrghh! Where did our approach go wrong?

    View full-size slide

  28. []
    let ``Adding zero is the same as doing nothing``()=
    for _ in [1..100] do
    let x = randInt()
    let result1 = x |> add 0
    let result2 = x
    Assert.AreEqual(result1,result2)
    Yes! Adding zero is the same as doing nothing
    We have to check that the result is somehow connected to the input.
    Is there a trivial property of add that we know the answer to
    without reimplementing our own version?

    View full-size slide

  29. Finally, the EDFH is defeated...

    TEST: ``When I add two numbers, the result
    should not depend on parameter order``
    TEST: ``Adding 1 twice is the same as adding 2``

    TEST: ``Adding zero is the same as doing nothing`` 
    If these are all true we
    MUST have a correct
    implementation*
    * not quite true

    View full-size slide

  30. let propertyCheck property =
    // property has type: int -> int -> bool
    for _ in [1..100] do
    let x = randInt()
    let y = randInt()
    let result = property x y
    Assert.IsTrue(result)
    Let's extract the shared code... Pass in a "property"
    Check the property is
    true for random inputs

    View full-size slide

  31. let commutativeProperty x y =
    let result1 = add x y
    let result2 = add y x
    result1 = result2
    And the tests now look like:
    []
    let ``When I add two numbers, the result
    should not depend on parameter order``()=
    propertyCheck commutativeProperty

    View full-size slide

  32. let adding1TwiceIsAdding2OnceProperty x _ =
    let result1 = x |> add 1 |> add 1
    let result2 = x |> add 2
    result1 = result2
    And the second property
    []
    let ``Adding 1 twice is the same as adding 2``()=
    propertyCheck adding1TwiceIsAdding2OnceProperty

    View full-size slide

  33. let identityProperty x _ =
    let result1 = x |> add 0
    result1 = x
    And the third property
    []
    let ``Adding zero is the same as doing nothing``()=
    propertyCheck identityProperty

    View full-size slide

  34. Testing with properties
    • The parameter order doesn't matter
    • Doing "add 1" twice is the same as
    doing "add 2" once
    • Adding zero does nothing
    These properties
    apply to ALL inputs
    So we have a very
    high confidence that
    the implementation is
    correct

    View full-size slide

  35. Testing with properties
    • "Commutativity" property
    • "Associativity" property
    • "Identity" property
    These properties
    define addition!
    The EDFH can't create an
    incorrect implementation!
    Bonus: By using specifications, we have
    understood the requirements in a deeper way.
    Specification

    View full-size slide

  36. Why bother with the EDFH?
    Surely such a malicious programmer is
    unrealistic and over-the-top?

    View full-size slide

  37. Evil
    Stupid
    Lazy
    In practice,
    no difference!

    View full-size slide

  38. In my career, I've always had to deal with one
    stupid person in particular 
    Me!
    When I look at my old code, I almost always see something wrong!
    I've often created flawed implementations, not out of evil
    intent, but out of unawareness and blindness
    The real EDFH!

    View full-size slide

  39. Part III:
    QuickCheck and its ilk
    Wouldn't it be nice to have a toolkit for doing this?
    The "QuickCheck" library was originally developed for Haskell by
    Koen Claessen and John Hughes, and has been ported to many
    other languages.

    View full-size slide

  40. QuickCheck
    Generator Shrinker
    Your Property Function that returns bool
    Checker API
    Pass to checker
    Generates
    random inputs
    Creates minimal
    failing input

    View full-size slide

  41. // correct implementation of add!
    let add x y = x + y
    let commutativeProperty x y =
    let result1 = add x y
    let result2 = add y x
    result1 = result2
    // check the property interactively
    Check.Quick commutativeProperty
    Using QuickCheck (FsCheck) looks like this:
    Ok, passed 100 tests.
    And get the output:

    View full-size slide

  42. Generators:
    making random inputs
    QuickCheck
    Generator Shrinker
    Checker API

    View full-size slide

  43. Generates ints
    "int" generator 0, 1, 3, -2, ... etc
    Generates strings
    "string" generator "", "eiX$a^", "U%0Ika&r", ... etc
    "bool" generator true, false, false, true, ... etc
    Generating primitive types
    Generates bools

    View full-size slide

  44. Generates pairs of ints
    "int*int" generator (0,0), (1,0), (2,0), (-1,1), (-1,2) ... etc
    Generates options
    "int option" generator Some 0, Some -1, None, Some -4; None ...
    "Color" generator Green 47, Red, Blue true, Green -12, ...
    Generating compound types
    type Color = Red | Green of int | Blue of bool
    Generates values of custom type
    Define custom type

    View full-size slide

  45. let commutativeProperty (x,y) =
    let result1 = add x y
    let result2 = add y x // reversed params
    result1 = result2
    (b) Appropriate generator will
    be automatically created
    int*int generator
    (0,0) (1,0) (2,0) (-1,1) (100,-99) ...
    (a) Checker detects that the
    input is a pair of ints
    Checker API
    (c) Valid values will be generated...
    (d) ...and passed to
    the property for
    evaluation
    How it works in practice

    View full-size slide

  46. Shrinking:
    dealing with failure
    QuickCheck
    Generator Shrinker
    Checker API

    View full-size slide

  47. let smallerThan81Property x =
    x < 81
    Property to test – we know it's gonna fail!
    "int" generator 0, 1, 3, -2, 34, -65, 100
    Fails at 100!
    So 100 fails, but knowing that is not very helpful
    How shrinking works
    Time to start shrinking!

    View full-size slide

  48. let smallerThan81Property x =
    x < 81
    Shrink again starting at 88
    How shrinking works
    Shrink list for 100 0, 50, 75, 88, 94, 97, 99
    Fails at 88!
    Generate a new
    sequence up to 100

    View full-size slide

  49. let smallerThan81Property x =
    x < 81
    Shrink again starting at 83
    How shrinking works
    Shrink list for 88 0, 44, 66, 77, 83, 86, 87
    Fails at 83!
    Generate a new
    sequence up to 88

    View full-size slide

  50. let smallerThan81Property x =
    x < 81
    Shrink again starting at 81
    How shrinking works
    Shrink list for 83 0, 42, 63, 73, 78, 81, 82
    Fails at 81!
    Generate a new
    sequence up to 83

    View full-size slide

  51. let smallerThan81Property x =
    x < 81
    Shrink has determined that 81 is
    the smallest failing input!
    How shrinking works
    Shrink list for 81 0, 41, 61, 71, 76, 79, 80
    All pass!
    Generate a new
    sequence up to 81

    View full-size slide

  52. Shrinking – final result
    Check.Quick smallerThan81Property
    // result: Falsifiable, after 23 tests (3 shrinks)
    // 81
    Shrinking is really helpful to show
    the boundaries where errors happen
    Shrinking is built into the check:

    View full-size slide

  53. Part IV:
    How to choose properties

    View full-size slide

  54. ABC
    123
    do X do X
    do Y
    do Y
    "Different paths, same destination"
    Examples:
    - Commutivity
    - Associativity
    - Map
    - Monad & Functor laws

    View full-size slide

  55. "Different paths, same destination"
    Applied to a sort function
    [1;2;3]
    ?
    do ? do ?
    List.sort
    List.sort

    View full-size slide

  56. "Different paths, same destination"
    Applied to a sort function
    [2;3;1]
    [-2;-3;-1] [-3;-2;-1]
    [1;2;3]
    Negate
    List.sort
    List.sort
    Negate
    then reverse

    View full-size slide

  57. "Different paths, same destination"
    Applied to a map function
    Some(2)
    .Map(x => x * 3)
    Some(2 * 3)
    x
    Option (x) Option (f x)
    f x
    Create
    Map f
    f
    Create
    f x = x * 3

    View full-size slide

  58. "There and back again"
    ABC 100101001
    Do X
    Inverse
    Examples:
    - Serialization/Deserialization
    - Addition/Subtraction
    - Write/Read
    - SetProperty/GetProperty

    View full-size slide

  59. "There and back again"
    Applied to a list reverse function
    [1;2;3] [3;2;1]
    reverse
    reverse

    View full-size slide

  60. "Some things never change"
     
    transform
    Examples:
    - Size of a collection
    - Contents of a collection
    - Balanced trees

    View full-size slide

  61. [2;3;1]
    [-2;-3;-1] [-3;-2;-1]
    [1;2;3]
    Negate
    List.sort
    List.sort
    Negate
    then reverse
    The EDFH and List.Sort
    The EDFH can beat this!

    View full-size slide

  62. The EDFH and List.Sort
    [2;3;1]
    [-2;-3;-1] [ ]
    [ ]
    Negate
    List.evilSort
    List.evilSort
    Negate
    then reverse
    EvilSort just returns an empty list!
    This passes the "commutivity" test!

    View full-size slide

  63. "Some things never change"
    [2;3;1]
    [1; 2; 3]; [2; 1; 3]; [2; 3; 1];
    [1; 3; 2]; [3; 1; 2]; [3; 2; 1]
    [1;2;3]
    List.sort
    Must be one of these
    permutations
    Used to ensure the sort function is good

    View full-size slide

  64. "The more things change,
    the more they stay the same"
     
    distinct
    
    distinct
    Idempotence:
    - Sort
    - Filter
    - Event processing
    - Required for distributed designs

    View full-size slide

  65. "Solve a smaller problem first"
         
    - Divide and conquer algorithms (e.g. quicksort)
    - Structural induction (recursive data structures)

    View full-size slide

  66. "Hard to prove, easy to verify"
    - Prime number factorization
    - Too many others to mention!

    View full-size slide

  67. "Hard to prove, easy to verify"
    Applied to a tokenizer
    “a,b,c”
    split
    “a” “b” “c”
    “a,b,c”
    Combine and
    verify
    To verify the tokenizer, just check that the
    concatenated tokens give us back the original string

    View full-size slide

  68. "Hard to prove, easy to verify"
    Applied to a sort
    To verify the sort,
    check that each pair is ordered
    [2;3;1]
    (1<=2) (2<=3)
    [1;2;3]
    List.sort

    View full-size slide

  69. ABC
    ABC 123
    123
    Compare
    System
    under test
    Test Oracle
    "The test oracle"
    - Compare optimized with slow brute-force version
    - Compare parallel with single thread version.

    View full-size slide

  70. Part V:
    Model based testing
    Using the test oracle approach
    for complex implementations

    View full-size slide

  71. Testing a simple database
    Open Incr Close Incr Open Close
    Open Decr Open
    Four operations: Open, Close, Increment, Decrement
    How do we know that our db works?
    Let QuickCheck generate a random list of these actions for each client
    Open Incr
    Client
    A
    Client
    B
    Two clients: Client A and Client B

    View full-size slide

  72. Testing a simple database
    Compare model result with real system!
    Open Incr Close Incr Open Close
    Open Decr Open Open Incr
    Test on real
    system
    Open Incr Close Incr Open Close
    Open Decr Open Open Incr
    Test on very
    simple model
    1 0
    0 0 1
    (just an in-memory
    accumulator)
    Connection closed,
    so no change

    View full-size slide

  73. Real world example
    • Subtle bugs in an Erlang module
    • The steps to reproduce were bizarre
    – open-close-open file then exactly 3 parallel ops
    – no human would ever think to write this test case
    • Shrinker critical in finding minimal sequence
    • War stories from John Hughes at
    https://vimeo.com/68383317

    View full-size slide

  74. Example-based tests vs.
    Property-based tests

    View full-size slide

  75. Example-based tests vs. Property-based tests
    • PBTs are more general
    – One property-based test can replace many example-
    based tests.
    • PBTs can reveal overlooked edge cases
    – Nulls, negative numbers, weird strings, etc.
    • PBTs ensure deep understanding of requirements
    – Property-based tests force you to think! 
    • Example-based tests are still helpful though!
    – Easier to understand for newcomers

    View full-size slide

  76. Summary
    Be lazy! Don't write tests, generate them!
    Use property-based thinking to gain
    deeper insight into the requirements

    View full-size slide

  77. The lazy programmer's guide to
    writing 1000's of tests
    An introduction to property based testing
    Let us know if you
    need help with F#
    Thanks!
    @ScottWlaschin
    fsharpforfunandprofit.com/pbt
    fsharpworks.com Slides and video here
    Contact me

    View full-size slide