Tweet
Tweet

Slide 1

Slide 1 text

Advances in Lazy SmallCheck Jason S. Reich, Matthew Naylor, Colin Runciman 30/08/12 – IFL 2012, Oxford, UK Wednesday, 29 August 12

Slide 2

Slide 2 text

A ‘conjectured’ property prop_ReduceFold :: ([Bool] -> Bool) -> Property prop_ReduceFold r = exists $ \f z -> forAll $ \xs -> foldr f z xs == r xs Wednesday, 29 August 12

Slide 3

Slide 3 text

A ‘conjectured’ property prop_ReduceFold :: ([Bool] -> Bool) -> Property prop_ReduceFold r = exists $ \f z -> forAll $ \xs -> foldr f z xs == r xs “All reductions on lists of Boolean values to a single Boolean value can be expressed as a foldr.” Wednesday, 29 August 12

Slide 4

Slide 4 text

A ‘conjectured’ property prop_ReduceFold :: ([Bool] -> Bool) -> Property prop_ReduceFold r = exists $ \f z -> forAll $ \xs -> foldr f z xs == r xs “All reductions on lists of Boolean values to a single Boolean value can be expressed as a foldr.” Functional values Wednesday, 29 August 12

Slide 5

Slide 5 text

A ‘conjectured’ property prop_ReduceFold :: ([Bool] -> Bool) -> Property prop_ReduceFold r = exists $ \f z -> forAll $ \xs -> foldr f z xs == r xs “All reductions on lists of Boolean values to a single Boolean value can be expressed as a foldr.” Existential quantifier Functional values Wednesday, 29 August 12

Slide 6

Slide 6 text

A ‘conjectured’ property prop_ReduceFold :: ([Bool] -> Bool) -> Property prop_ReduceFold r = exists $ \f z -> forAll $ \xs -> foldr f z xs == r xs “All reductions on lists of Boolean values to a single Boolean value can be expressed as a foldr.” Existential quantifier Functional values Nested quantification Wednesday, 29 August 12

Slide 7

Slide 7 text

Property-based testing QuickCheck SmallCheck Lazy SmallCheck (2008) Strategy Demand-driven Functional values Existentials Nested quantification Random Bounded Exhaustive Lazy Bounded Exhaustive ○ ○ ● ● ● ○ ○ ● ○ ● ● ○ Wednesday, 29 August 12

Slide 8

Slide 8 text

Property-based testing QuickCheck SmallCheck Lazy SmallCheck (2008) Strategy Demand-driven Functional values Existentials Nested quantification Random Bounded Exhaustive Lazy Bounded Exhaustive ○ ○ ● ● ● ○ ○ ● ○ ● ● ○ Wednesday, 29 August 12

Slide 9

Slide 9 text

In SmallCheck... >>> test prop_ReduceFold Depth 0: Completed 4 test(s) without failure. ... Depth 2: Failed test no. 3. Test values follow. []-> True [True]-> True [True,True]-> True [True,True,True]-> True [True,True,False]-> True [True,False]-> True Wednesday, 29 August 12

Slide 10

Slide 10 text

In SmallCheck... >>> test prop_ReduceFold Depth 0: Completed 4 test(s) without failure. ... Depth 2: Failed test no. 3. Test values follow. []-> True [True]-> True [True,True]-> True [True,True,True]-> True [True,True,False]-> True [True,False]-> True Wednesday, 29 August 12

Slide 11

Slide 11 text

In SmallCheck... True [True,False]-> True [True,False,True]-> True [True,False,False]-> True [False]-> True [False,True]-> True [False,True,True]-> True [False,True,False]-> True [False,False]-> False [False,False,True]-> True [False,False,False]-> True Wednesday, 29 August 12

Slide 12

Slide 12 text

In SmallCheck... True [True,False]-> True [True,False,True]-> True [True,False,False]-> True [False]-> True [False,True]-> True [False,True,True]-> True [False,True,False]-> True [False,False]-> False [False,False,True]-> True [False,False,False]-> True r = (/= [False, False]) Wednesday, 29 August 12

Slide 13

Slide 13 text

Lazy SmallCheck: A refresher Wednesday, 29 August 12

Slide 14

Slide 14 text

LSC? • Lazy SmallCheck (Runciman et al., 2008). Wednesday, 29 August 12

Slide 15

Slide 15 text

LSC? • Lazy SmallCheck (Runciman et al., 2008). • Check – Property-based testing library. Wednesday, 29 August 12

Slide 16

Slide 16 text

LSC? • Lazy SmallCheck (Runciman et al., 2008). • Check – Property-based testing library. • Small – Exhaustive search for minimal counterexamples in bounded test-data space. Wednesday, 29 August 12

Slide 17

Slide 17 text

LSC? • Lazy SmallCheck (Runciman et al., 2008). • Check – Property-based testing library. • Small – Exhaustive search for minimal counterexamples in bounded test-data space. • Lazy – Space includes partial values and evaluation order guides search. Wednesday, 29 August 12

Slide 18

Slide 18 text

>>> depthCheck 7 prop_insertSet Depth 7: Completed 109600 test(s) without failure. But 108576 did not meet ==> condition. Benefit of being lazy prop_insertSet :: Char -> [Char] -> Property prop_insertSet x xs = isOrdered xs ==> isOrdered (insert x xs) In SC Wednesday, 29 August 12

Slide 19

Slide 19 text

>>> depthCheck 7 prop_insertSet Depth 7: Completed 109600 test(s) without failure. But 108576 did not meet ==> condition. Benefit of being lazy prop_insertSet :: Char -> [Char] -> Property prop_insertSet x xs = isOrdered xs ==> isOrdered (insert x xs) In SC Wednesday, 29 August 12

Slide 20

Slide 20 text

Benefit of being lazy prop_insertSet :: Char -> [Char] -> Property prop_insertSet x xs = isOrdered xs ==> isOrdered (insert x xs) >>> depthCheck 7 prop_insertSet OK, required 1716 tests at depth 7 In LSC 2008 1.6% of tests performed by SC Wednesday, 29 August 12

Slide 21

Slide 21 text

Benefit of being lazy prop_insertSet :: Char -> [Char] -> Property prop_insertSet x xs = isOrdered xs ==> isOrdered (insert x xs) Lazy antecedent >>> depthCheck 7 prop_insertSet OK, required 1716 tests at depth 7 In LSC 2008 1.6% of tests performed by SC Wednesday, 29 August 12

Slide 22

Slide 22 text

• xs = (1:0:⊥) falsifies the antecedent. Benefit of being lazy prop_insertSet :: Char -> [Char] -> Property prop_insertSet x xs = isOrdered xs ==> isOrdered (insert x xs) Lazy antecedent Wednesday, 29 August 12

Slide 23

Slide 23 text

• xs = (1:0:⊥) falsifies the antecedent. • Therefore, the LSC doesn’t need to test; xs = [1,0] xs = [1,0,2,3] xs = [1,0,5,4] e.t.c. Benefit of being lazy prop_insertSet :: Char -> [Char] -> Property prop_insertSet x xs = isOrdered xs ==> isOrdered (insert x xs) Lazy antecedent Wednesday, 29 August 12

Slide 24

Slide 24 text

• xs = (1:0:⊥) falsifies the antecedent. • Therefore, the LSC doesn’t need to test; xs = [1,0] xs = [1,0,2,3] xs = [1,0,5,4] e.t.c. • Or even any value of x for this class of xs. Benefit of being lazy prop_insertSet :: Char -> [Char] -> Property prop_insertSet x xs = isOrdered xs ==> isOrdered (insert x xs) Lazy antecedent Wednesday, 29 August 12

Slide 25

Slide 25 text

New features Wednesday, 29 August 12

Slide 26

Slide 26 text

Property-based testing Lazy SmallCheck (2008) Lazy SmallCheck (2012) Strategy Demand-driven Functional values Existentials Nested quantification Lazy Bounded Exhaustive Lazy Bounded Exhaustive ● ● ○ ● ○ ● ○ ● Wednesday, 29 August 12

Slide 27

Slide 27 text

Property-based testing Lazy SmallCheck (2008) Lazy SmallCheck (2012) Strategy Demand-driven Functional values Existentials Nested quantification Lazy Bounded Exhaustive Lazy Bounded Exhaustive ● ● ○ ● ○ ● ○ ● + better display of counter- examples Wednesday, 29 August 12

Slide 28

Slide 28 text

Property-based testing Lazy SmallCheck (2008) Lazy SmallCheck (2012) Strategy Demand-driven Functional values Existentials Nested quantification Lazy Bounded Exhaustive Lazy Bounded Exhaustive ● ● ○ ● ○ ● ○ ● + speedup! + better display of counter- examples Wednesday, 29 August 12

Slide 29

Slide 29 text

>>> test prop_ReduceFold ... Depth 6: Var 0: { [] -> False ; _:[] -> False ; _:_:_ -> True } In LSC 2012... prop_ReduceFold :: ([Bool] -> Bool) -> Property prop_ReduceFold r = exists $ \f z -> forAll $ \xs -> foldr f z xs == r xs Wednesday, 29 August 12

Slide 30

Slide 30 text

>>> test prop_ReduceFold ... Depth 6: Var 0: { [] -> False ; _:[] -> False ; _:_:_ -> True } In LSC 2012... prop_ReduceFold :: ([Bool] -> Bool) -> Property prop_ReduceFold r = exists $ \f z -> forAll $ \xs -> foldr f z xs == r xs “Tests for multi-item lists.” Wednesday, 29 August 12

Slide 31

Slide 31 text

>>> test prop_ReduceFold ... Depth 6: Var 0: { [] -> False ; _:[] -> False ; _:_:_ -> True } In LSC 2012... prop_ReduceFold :: ([Bool] -> Bool) -> Property prop_ReduceFold r = exists $ \f z -> forAll $ \xs -> foldr f z xs == r xs “Tests for multi-item lists.” Wildcard patterns Wednesday, 29 August 12

Slide 32

Slide 32 text

>>> :{ >>| let prop_BitString p = >>| p [False, False, True, False, False, True] >>| && p [False, False, False, False, True, True] >>| ==> p [False, False, False, False, False, True] >>| :} >>> test prop_BitString ... Depth 14: Var 0: { _:_:False:_:False:_ -> False ; _:_:False:_:True:_ -> True ; _:_:True:_ -> True } Functional values I Wednesday, 29 August 12

Slide 33

Slide 33 text

>>> :{ >>| let prop_BitString p = >>| p [False, False, True, False, False, True] >>| && p [False, False, False, False, True, True] >>| ==> p [False, False, False, False, False, True] >>| :} >>> test prop_BitString ... Depth 14: Var 0: { _:_:False:_:False:_ -> False ; _:_:False:_:True:_ -> True ; _:_:True:_ -> True } Functional values I Wednesday, 29 August 12

Slide 34

Slide 34 text

>>> :{ >>| let prop_BitString p = >>| p [False, False, True, False, False, True] >>| && p [False, False, False, False, True, True] >>| ==> p [False, False, False, False, False, True] >>| :} >>> test prop_BitString ... Depth 14: Var 0: { _:_:False:_:False:_ -> False ; _:_:False:_:True:_ -> True ; _:_:True:_ -> True } Functional values I Partial function Wednesday, 29 August 12

Slide 35

Slide 35 text

>>> :{ >>| let prop_BitString p = >>| p [False, False, True, False, False, True] >>| && p [False, False, False, False, True, True] >>| ==> p [False, False, False, False, False, True] >>| :} >>> test prop_BitString ... Depth 14: Var 0: { _:_:False:_:False:_ -> False ; _:_:False:_:True:_ -> True ; _:_:True:_ -> True } Functional values I Partial function Wildcard patterns Wednesday, 29 August 12

Slide 36

Slide 36 text

>>> :{ >>| let prop_BitString p = >>| p [False, False, True, False, False, True] >>| && p [False, False, False, False, True, True] >>| ==> p [False, False, False, False, False, True] >>| :} >>> test prop_BitString ... Depth 14: Var 0: { _:_:False:_:False:_ -> False ; _:_:False:_:True:_ -> True ; _:_:True:_ -> True } Functional values I Partial function Wildcard patterns + = Minimal example Wednesday, 29 August 12

Slide 37

Slide 37 text

Functional values II • LSC now generates partial functions including wildcard patterns. • Tries in disguise! • Wildcards explicit but partiality of functions is a result of partial values. • Users need to implement ‘Argument’ instance for functional value argument types. • ‘deriveArgument’ does this automatically using Template Haskell. Wednesday, 29 August 12

Slide 38

Slide 38 text

>>> :{ >>| let prop_Foldx1 :: (Bool -> Bool -> Bool) -> [Bool] >>| -> Bool >>| prop_Foldx1 f xs = (not.null) xs >>| ==> foldl1 f xs == foldr1 f xs >>| :} >>> test $ prop_Foldx1 $ const not ... [False,False,False] Displaying counterexamples In LSC 2008 Wednesday, 29 August 12

Slide 39

Slide 39 text

>>> :{ >>| let prop_Foldx1 :: (Bool -> Bool -> Bool) -> [Bool] >>| -> Bool >>| prop_Foldx1 f xs = (not.null) xs >>| ==> foldl1 f xs == foldr1 f xs >>| :} >>> test $ prop_Foldx1 $ const not ... [False,False,False] Displaying counterexamples In LSC 2008 Displays the first totally defined counterexample. Wednesday, 29 August 12

Slide 40

Slide 40 text

>>> :{ >>| let prop_Foldx1 :: (Bool -> Bool -> Bool) -> [Bool] >>| -> Bool >>| prop_Foldx1 f xs = (not.null) xs >>| ==> foldl1 f xs == foldr1 f xs >>| :} >>> test $ prop_Foldx1 $ const not ... Var 0: _:_:False:[] Displaying counterexamples In LSC 2012 Wednesday, 29 August 12

Slide 41

Slide 41 text

>>> :{ >>| let prop_Foldx1 :: (Bool -> Bool -> Bool) -> [Bool] >>| -> Bool >>| prop_Foldx1 f xs = (not.null) xs >>| ==> foldl1 f xs == foldr1 f xs >>| :} >>> test $ prop_Foldx1 $ const not ... Var 0: _:_:False:[] Displaying counterexamples Displaying partial values gives more information! Uses “Chasing Bottoms” (Danielsson, 2004) In LSC 2012 Wednesday, 29 August 12

Slide 42

Slide 42 text

>>> :{ >>| let prop_Skolem :: (Peano -> Peano -> Bool) >>| -> Property >>| prop_Skolem r = exists $ \f -> forAll $ \x -> >>| (exists $ \y -> r x y) >>| <=> >>| (r x (f x)) >>| :} >>> :s +s >>> depthCheck 8 prop_skolem LSC2: Passed in 3342802 tests. (60.85 secs, 61941317512 bytes) Quantification I Wednesday, 29 August 12

Slide 43

Slide 43 text

>>> :{ >>| let prop_Skolem :: (Peano -> Peano -> Bool) >>| -> Property >>| prop_Skolem r = exists $ \f -> forAll $ \x -> >>| (exists $ \y -> r x y) >>| <=> >>| (r x (f x)) >>| :} >>> :s +s >>> depthCheck 8 prop_skolem LSC2: Passed in 3342802 tests. (60.85 secs, 61941317512 bytes) Quantification I Existential quantifiers Wednesday, 29 August 12

Slide 44

Slide 44 text

>>> :{ >>| let prop_Skolem :: (Peano -> Peano -> Bool) >>| -> Property >>| prop_Skolem r = exists $ \f -> forAll $ \x -> >>| (exists $ \y -> r x y) >>| <=> >>| (r x (f x)) >>| :} >>> :s +s >>> depthCheck 8 prop_skolem LSC2: Passed in 3342802 tests. (60.85 secs, 61941317512 bytes) Quantification I Existential quantifiers Nested quantification Wednesday, 29 August 12

Slide 45

Slide 45 text

>>> :{ >>| let prop_Skolem :: (Peano -> Peano -> Bool) >>| -> Property >>| prop_Skolem r = exists $ \f -> forAll $ \x -> >>| (exists $ \y -> r x y) >>| <=> >>| (r x (f x)) >>| :} >>> :s +s >>> depthCheck 8 prop_skolem LSC2: Passed in 3342802 tests. (60.85 secs, 61941317512 bytes) Quantification I Existential quantifiers Nested quantification Wednesday, 29 August 12

Slide 46

Slide 46 text

>>> :{ >>| let prop_Skolem :: (Peano -> Peano -> Bool) >>| -> Property >>| prop_Skolem r = exists $ \f -> forAll $ \x -> >>| (exists $ \y -> r x y) >>| <=> >>| (r x (f x)) >>| :} >>> :s +s >>> depthCheck 8 prop_skolem LSC2: Passed in 3342802 tests. (60.85 secs, 61941317512 bytes) Quantification I Existential quantifiers Nested quantification n.b. Don’t even get to depth 3 in SC. Wednesday, 29 August 12

Slide 47

Slide 47 text

Quantification II • Lazy pruning is beneficial for existentials too. • Nested quantification necessary for existentials to be useful. • Adds forAll and exists to Property DSL. • Required a complete rethink of underlying structure and refutation algorithm. Wednesday, 29 August 12

Slide 48

Slide 48 text

Evaluation Wednesday, 29 August 12

Slide 49

Slide 49 text

Performance Name Ratio Catch Circuits1 Circuits2 Circuits3 Countdown1 Countdown2 Huffman1 0.28 1.00 1.04 0.56 0.55 1.01 0.67 Name Ratio Huffman2 ListSet1 Mate RedBlack SumPuz Turner Geo. Mean 0.59 0.80 0.60 0.66 0.97 0.62 0.68 Ratio = LSC2012 execution time LSC2008 execution time Ratio < 1 is improvement. Wednesday, 29 August 12

Slide 50

Slide 50 text

Related work • Koen Claessen, Shrinking and Showing Functions (Functional Pearl), Haskell 2012. • Extends QuickCheck’s functional value capabilities. • Uses tries (different formulation) to provide additional features. • Must wrap functional values in a ‘modifier’. Wednesday, 29 August 12

Slide 51

Slide 51 text

Further work • Looking at Claessen’s trie formulation. • Could use SYB instead of TH for automatic instances. • Difficult to judge the depth of a functional value. Wednesday, 29 August 12

Slide 52

Slide 52 text

Further work • Looking at Claessen’s trie formulation. • Could use SYB instead of TH for automatic instances. • Difficult to judge the depth of a functional value. • Parallel LSC (with JMCT) • Naive so far. Testing on 8 cores. • Scales well for most examples. Wednesday, 29 August 12

Slide 53

Slide 53 text

Conclusions I • SCs handling of functional values wasn’t entirely satisfying. • New formulation for LSC leverages the ‘lazy’ for maximum effect. • Displaying partial counterexample gives more information than a totally defined one. Wednesday, 29 August 12

Slide 54

Slide 54 text

Conclusions II • Existentials also benefit from laziness. • Making things more complicated can strangely make them faster? • Broader range of functionality, looking for interesting applications. Wednesday, 29 August 12