Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Advances in Lazy SmallCheck

Advances in Lazy SmallCheck

A presentation on efficient testing of higher-order properties with mixed quantification.

Jason Reich

August 30, 2012
Tweet

Other Decks in Technology

Transcript

  1. Advances in Lazy SmallCheck Jason S. Reich, Matthew Naylor, Colin

    Runciman 30/08/12 – IFL 2012, Oxford, UK Wednesday, 29 August 12
  2. A ‘conjectured’ property prop_ReduceFold :: ([Bool] -> Bool) -> Property

    prop_ReduceFold r = exists $ \f z -> forAll $ \xs -> foldr f z xs == r xs Wednesday, 29 August 12
  3. A ‘conjectured’ property prop_ReduceFold :: ([Bool] -> Bool) -> Property

    prop_ReduceFold r = exists $ \f z -> forAll $ \xs -> foldr f z xs == r xs “All reductions on lists of Boolean values to a single Boolean value can be expressed as a foldr.” Wednesday, 29 August 12
  4. A ‘conjectured’ property prop_ReduceFold :: ([Bool] -> Bool) -> Property

    prop_ReduceFold r = exists $ \f z -> forAll $ \xs -> foldr f z xs == r xs “All reductions on lists of Boolean values to a single Boolean value can be expressed as a foldr.” Functional values Wednesday, 29 August 12
  5. A ‘conjectured’ property prop_ReduceFold :: ([Bool] -> Bool) -> Property

    prop_ReduceFold r = exists $ \f z -> forAll $ \xs -> foldr f z xs == r xs “All reductions on lists of Boolean values to a single Boolean value can be expressed as a foldr.” Existential quantifier Functional values Wednesday, 29 August 12
  6. A ‘conjectured’ property prop_ReduceFold :: ([Bool] -> Bool) -> Property

    prop_ReduceFold r = exists $ \f z -> forAll $ \xs -> foldr f z xs == r xs “All reductions on lists of Boolean values to a single Boolean value can be expressed as a foldr.” Existential quantifier Functional values Nested quantification Wednesday, 29 August 12
  7. Property-based testing QuickCheck SmallCheck Lazy SmallCheck (2008) Strategy Demand-driven Functional

    values Existentials Nested quantification Random Bounded Exhaustive Lazy Bounded Exhaustive ◦ ◦ • • • ◦ ◦ • ◦ • • ◦ Wednesday, 29 August 12
  8. Property-based testing QuickCheck SmallCheck Lazy SmallCheck (2008) Strategy Demand-driven Functional

    values Existentials Nested quantification Random Bounded Exhaustive Lazy Bounded Exhaustive ◦ ◦ • • • ◦ ◦ • ◦ • • ◦ Wednesday, 29 August 12
  9. In SmallCheck... >>> test prop_ReduceFold Depth 0: Completed 4 test(s)

    without failure. ... Depth 2: Failed test no. 3. Test values follow. []-> True [True]-> True [True,True]-> True [True,True,True]-> True [True,True,False]-> True [True,False]-> True Wednesday, 29 August 12
  10. In SmallCheck... >>> test prop_ReduceFold Depth 0: Completed 4 test(s)

    without failure. ... Depth 2: Failed test no. 3. Test values follow. []-> True [True]-> True [True,True]-> True [True,True,True]-> True [True,True,False]-> True [True,False]-> True Wednesday, 29 August 12
  11. In SmallCheck... True [True,False]-> True [True,False,True]-> True [True,False,False]-> True [False]->

    True [False,True]-> True [False,True,True]-> True [False,True,False]-> True [False,False]-> False [False,False,True]-> True [False,False,False]-> True Wednesday, 29 August 12
  12. In SmallCheck... True [True,False]-> True [True,False,True]-> True [True,False,False]-> True [False]->

    True [False,True]-> True [False,True,True]-> True [False,True,False]-> True [False,False]-> False [False,False,True]-> True [False,False,False]-> True r = (/= [False, False]) Wednesday, 29 August 12
  13. LSC? • Lazy SmallCheck (Runciman et al., 2008). • Check

    – Property-based testing library. Wednesday, 29 August 12
  14. LSC? • Lazy SmallCheck (Runciman et al., 2008). • Check

    – Property-based testing library. • Small – Exhaustive search for minimal counterexamples in bounded test-data space. Wednesday, 29 August 12
  15. LSC? • Lazy SmallCheck (Runciman et al., 2008). • Check

    – Property-based testing library. • Small – Exhaustive search for minimal counterexamples in bounded test-data space. • Lazy – Space includes partial values and evaluation order guides search. Wednesday, 29 August 12
  16. >>> depthCheck 7 prop_insertSet Depth 7: Completed 109600 test(s) without

    failure. But 108576 did not meet ==> condition. Benefit of being lazy prop_insertSet :: Char -> [Char] -> Property prop_insertSet x xs = isOrdered xs ==> isOrdered (insert x xs) In SC Wednesday, 29 August 12
  17. >>> depthCheck 7 prop_insertSet Depth 7: Completed 109600 test(s) without

    failure. But 108576 did not meet ==> condition. Benefit of being lazy prop_insertSet :: Char -> [Char] -> Property prop_insertSet x xs = isOrdered xs ==> isOrdered (insert x xs) In SC Wednesday, 29 August 12
  18. Benefit of being lazy prop_insertSet :: Char -> [Char] ->

    Property prop_insertSet x xs = isOrdered xs ==> isOrdered (insert x xs) >>> depthCheck 7 prop_insertSet OK, required 1716 tests at depth 7 In LSC 2008 1.6% of tests performed by SC Wednesday, 29 August 12
  19. Benefit of being lazy prop_insertSet :: Char -> [Char] ->

    Property prop_insertSet x xs = isOrdered xs ==> isOrdered (insert x xs) Lazy antecedent >>> depthCheck 7 prop_insertSet OK, required 1716 tests at depth 7 In LSC 2008 1.6% of tests performed by SC Wednesday, 29 August 12
  20. • xs = (1:0:⊥) falsifies the antecedent. Benefit of being

    lazy prop_insertSet :: Char -> [Char] -> Property prop_insertSet x xs = isOrdered xs ==> isOrdered (insert x xs) Lazy antecedent Wednesday, 29 August 12
  21. • xs = (1:0:⊥) falsifies the antecedent. • Therefore, the

    LSC doesn’t need to test; xs = [1,0] xs = [1,0,2,3] xs = [1,0,5,4] e.t.c. Benefit of being lazy prop_insertSet :: Char -> [Char] -> Property prop_insertSet x xs = isOrdered xs ==> isOrdered (insert x xs) Lazy antecedent Wednesday, 29 August 12
  22. • xs = (1:0:⊥) falsifies the antecedent. • Therefore, the

    LSC doesn’t need to test; xs = [1,0] xs = [1,0,2,3] xs = [1,0,5,4] e.t.c. • Or even any value of x for this class of xs. Benefit of being lazy prop_insertSet :: Char -> [Char] -> Property prop_insertSet x xs = isOrdered xs ==> isOrdered (insert x xs) Lazy antecedent Wednesday, 29 August 12
  23. Property-based testing Lazy SmallCheck (2008) Lazy SmallCheck (2012) Strategy Demand-driven

    Functional values Existentials Nested quantification Lazy Bounded Exhaustive Lazy Bounded Exhaustive • • ◦ • ◦ • ◦ • Wednesday, 29 August 12
  24. Property-based testing Lazy SmallCheck (2008) Lazy SmallCheck (2012) Strategy Demand-driven

    Functional values Existentials Nested quantification Lazy Bounded Exhaustive Lazy Bounded Exhaustive • • ◦ • ◦ • ◦ • + better display of counter- examples Wednesday, 29 August 12
  25. Property-based testing Lazy SmallCheck (2008) Lazy SmallCheck (2012) Strategy Demand-driven

    Functional values Existentials Nested quantification Lazy Bounded Exhaustive Lazy Bounded Exhaustive • • ◦ • ◦ • ◦ • + speedup! + better display of counter- examples Wednesday, 29 August 12
  26. >>> test prop_ReduceFold ... Depth 6: Var 0: { []

    -> False ; _:[] -> False ; _:_:_ -> True } In LSC 2012... prop_ReduceFold :: ([Bool] -> Bool) -> Property prop_ReduceFold r = exists $ \f z -> forAll $ \xs -> foldr f z xs == r xs Wednesday, 29 August 12
  27. >>> test prop_ReduceFold ... Depth 6: Var 0: { []

    -> False ; _:[] -> False ; _:_:_ -> True } In LSC 2012... prop_ReduceFold :: ([Bool] -> Bool) -> Property prop_ReduceFold r = exists $ \f z -> forAll $ \xs -> foldr f z xs == r xs “Tests for multi-item lists.” Wednesday, 29 August 12
  28. >>> test prop_ReduceFold ... Depth 6: Var 0: { []

    -> False ; _:[] -> False ; _:_:_ -> True } In LSC 2012... prop_ReduceFold :: ([Bool] -> Bool) -> Property prop_ReduceFold r = exists $ \f z -> forAll $ \xs -> foldr f z xs == r xs “Tests for multi-item lists.” Wildcard patterns Wednesday, 29 August 12
  29. >>> :{ >>| let prop_BitString p = >>| p [False,

    False, True, False, False, True] >>| && p [False, False, False, False, True, True] >>| ==> p [False, False, False, False, False, True] >>| :} >>> test prop_BitString ... Depth 14: Var 0: { _:_:False:_:False:_ -> False ; _:_:False:_:True:_ -> True ; _:_:True:_ -> True } Functional values I Wednesday, 29 August 12
  30. >>> :{ >>| let prop_BitString p = >>| p [False,

    False, True, False, False, True] >>| && p [False, False, False, False, True, True] >>| ==> p [False, False, False, False, False, True] >>| :} >>> test prop_BitString ... Depth 14: Var 0: { _:_:False:_:False:_ -> False ; _:_:False:_:True:_ -> True ; _:_:True:_ -> True } Functional values I Wednesday, 29 August 12
  31. >>> :{ >>| let prop_BitString p = >>| p [False,

    False, True, False, False, True] >>| && p [False, False, False, False, True, True] >>| ==> p [False, False, False, False, False, True] >>| :} >>> test prop_BitString ... Depth 14: Var 0: { _:_:False:_:False:_ -> False ; _:_:False:_:True:_ -> True ; _:_:True:_ -> True } Functional values I Partial function Wednesday, 29 August 12
  32. >>> :{ >>| let prop_BitString p = >>| p [False,

    False, True, False, False, True] >>| && p [False, False, False, False, True, True] >>| ==> p [False, False, False, False, False, True] >>| :} >>> test prop_BitString ... Depth 14: Var 0: { _:_:False:_:False:_ -> False ; _:_:False:_:True:_ -> True ; _:_:True:_ -> True } Functional values I Partial function Wildcard patterns Wednesday, 29 August 12
  33. >>> :{ >>| let prop_BitString p = >>| p [False,

    False, True, False, False, True] >>| && p [False, False, False, False, True, True] >>| ==> p [False, False, False, False, False, True] >>| :} >>> test prop_BitString ... Depth 14: Var 0: { _:_:False:_:False:_ -> False ; _:_:False:_:True:_ -> True ; _:_:True:_ -> True } Functional values I Partial function Wildcard patterns + = Minimal example Wednesday, 29 August 12
  34. Functional values II • LSC now generates partial functions including

    wildcard patterns. • Tries in disguise! • Wildcards explicit but partiality of functions is a result of partial values. • Users need to implement ‘Argument’ instance for functional value argument types. • ‘deriveArgument’ does this automatically using Template Haskell. Wednesday, 29 August 12
  35. >>> :{ >>| let prop_Foldx1 :: (Bool -> Bool ->

    Bool) -> [Bool] >>| -> Bool >>| prop_Foldx1 f xs = (not.null) xs >>| ==> foldl1 f xs == foldr1 f xs >>| :} >>> test $ prop_Foldx1 $ const not ... [False,False,False] Displaying counterexamples In LSC 2008 Wednesday, 29 August 12
  36. >>> :{ >>| let prop_Foldx1 :: (Bool -> Bool ->

    Bool) -> [Bool] >>| -> Bool >>| prop_Foldx1 f xs = (not.null) xs >>| ==> foldl1 f xs == foldr1 f xs >>| :} >>> test $ prop_Foldx1 $ const not ... [False,False,False] Displaying counterexamples In LSC 2008 Displays the first totally defined counterexample. Wednesday, 29 August 12
  37. >>> :{ >>| let prop_Foldx1 :: (Bool -> Bool ->

    Bool) -> [Bool] >>| -> Bool >>| prop_Foldx1 f xs = (not.null) xs >>| ==> foldl1 f xs == foldr1 f xs >>| :} >>> test $ prop_Foldx1 $ const not ... Var 0: _:_:False:[] Displaying counterexamples In LSC 2012 Wednesday, 29 August 12
  38. >>> :{ >>| let prop_Foldx1 :: (Bool -> Bool ->

    Bool) -> [Bool] >>| -> Bool >>| prop_Foldx1 f xs = (not.null) xs >>| ==> foldl1 f xs == foldr1 f xs >>| :} >>> test $ prop_Foldx1 $ const not ... Var 0: _:_:False:[] Displaying counterexamples Displaying partial values gives more information! Uses “Chasing Bottoms” (Danielsson, 2004) In LSC 2012 Wednesday, 29 August 12
  39. >>> :{ >>| let prop_Skolem :: (Peano -> Peano ->

    Bool) >>| -> Property >>| prop_Skolem r = exists $ \f -> forAll $ \x -> >>| (exists $ \y -> r x y) >>| <=> >>| (r x (f x)) >>| :} >>> :s +s >>> depthCheck 8 prop_skolem LSC2: Passed in 3342802 tests. (60.85 secs, 61941317512 bytes) Quantification I Wednesday, 29 August 12
  40. >>> :{ >>| let prop_Skolem :: (Peano -> Peano ->

    Bool) >>| -> Property >>| prop_Skolem r = exists $ \f -> forAll $ \x -> >>| (exists $ \y -> r x y) >>| <=> >>| (r x (f x)) >>| :} >>> :s +s >>> depthCheck 8 prop_skolem LSC2: Passed in 3342802 tests. (60.85 secs, 61941317512 bytes) Quantification I Existential quantifiers Wednesday, 29 August 12
  41. >>> :{ >>| let prop_Skolem :: (Peano -> Peano ->

    Bool) >>| -> Property >>| prop_Skolem r = exists $ \f -> forAll $ \x -> >>| (exists $ \y -> r x y) >>| <=> >>| (r x (f x)) >>| :} >>> :s +s >>> depthCheck 8 prop_skolem LSC2: Passed in 3342802 tests. (60.85 secs, 61941317512 bytes) Quantification I Existential quantifiers Nested quantification Wednesday, 29 August 12
  42. >>> :{ >>| let prop_Skolem :: (Peano -> Peano ->

    Bool) >>| -> Property >>| prop_Skolem r = exists $ \f -> forAll $ \x -> >>| (exists $ \y -> r x y) >>| <=> >>| (r x (f x)) >>| :} >>> :s +s >>> depthCheck 8 prop_skolem LSC2: Passed in 3342802 tests. (60.85 secs, 61941317512 bytes) Quantification I Existential quantifiers Nested quantification Wednesday, 29 August 12
  43. >>> :{ >>| let prop_Skolem :: (Peano -> Peano ->

    Bool) >>| -> Property >>| prop_Skolem r = exists $ \f -> forAll $ \x -> >>| (exists $ \y -> r x y) >>| <=> >>| (r x (f x)) >>| :} >>> :s +s >>> depthCheck 8 prop_skolem LSC2: Passed in 3342802 tests. (60.85 secs, 61941317512 bytes) Quantification I Existential quantifiers Nested quantification n.b. Don’t even get to depth 3 in SC. Wednesday, 29 August 12
  44. Quantification II • Lazy pruning is beneficial for existentials too.

    • Nested quantification necessary for existentials to be useful. • Adds forAll and exists to Property DSL. • Required a complete rethink of underlying structure and refutation algorithm. Wednesday, 29 August 12
  45. Performance Name Ratio Catch Circuits1 Circuits2 Circuits3 Countdown1 Countdown2 Huffman1

    0.28 1.00 1.04 0.56 0.55 1.01 0.67 Name Ratio Huffman2 ListSet1 Mate RedBlack SumPuz Turner Geo. Mean 0.59 0.80 0.60 0.66 0.97 0.62 0.68 Ratio = LSC2012 execution time LSC2008 execution time Ratio < 1 is improvement. Wednesday, 29 August 12
  46. Related work • Koen Claessen, Shrinking and Showing Functions (Functional

    Pearl), Haskell 2012. • Extends QuickCheck’s functional value capabilities. • Uses tries (different formulation) to provide additional features. • Must wrap functional values in a ‘modifier’. Wednesday, 29 August 12
  47. Further work • Looking at Claessen’s trie formulation. • Could

    use SYB instead of TH for automatic instances. • Difficult to judge the depth of a functional value. Wednesday, 29 August 12
  48. Further work • Looking at Claessen’s trie formulation. • Could

    use SYB instead of TH for automatic instances. • Difficult to judge the depth of a functional value. • Parallel LSC (with JMCT) • Naive so far. Testing on 8 cores. • Scales well for most examples. Wednesday, 29 August 12
  49. Conclusions I • SCs handling of functional values wasn’t entirely

    satisfying. • New formulation for LSC leverages the ‘lazy’ for maximum effect. • Displaying partial counterexample gives more information than a totally defined one. Wednesday, 29 August 12
  50. Conclusions II • Existentials also benefit from laziness. • Making

    things more complicated can strangely make them faster? • Broader range of functionality, looking for interesting applications. Wednesday, 29 August 12