Property-based testing (FrOSCon 9)

Transcript

1. testing with properties Christoph Neuroth ( ) @c089
2. [horribly predictable quote] “Testing shows the presence, not the absence of bugs.” -- Edsger W. Dijkstra
3. Hoare: “ The role of testing, in theory, is to establish the base propositions of an inductive proof.[...] At present, this is mainly theory [but] seems to show a possibility of practical results, though proving correctness is a laborious and expensive process.” Perlis: “ Much of program complexity is spurious and a number of test cases properly studied will exhaust the testing problem. The problem is to isolate the right test cases, not to prove the algorithm, for that follows after the choice of the proper test cases.” Diijkstra: “Testing shows the presence, not the absence of bugs”
4. Testing with examples specify correct output for chosen inputs relatively easy to come up with good communication tool straightforward to implement straightforward to overlook important ones usually only cover a very small portion of your domain
5. yay, all tests passing! / / o b v i o u s l y e x p e c t ( M a t h . a b s ( 0 ) ) . t o . e q u a l ( 0 ) ; / / n e g a t i v e v a l u e s b e c o m e p o s i t i v e e x p e c t ( M a t h . a b s ( - 1 ) ) . t o . e q u a l ( 1 ) ; e x p e c t ( M a t h . a b s ( - 5 ) ) . t o . e q u a l ( 5 ) ; / / w h i l e p o s i t i v e v a l u e s s t a y s p o s i t i v e e x p e c t ( M a t h . a b s ( 1 ) ) . t o . e q u a l ( 1 ) ; / / s h o u l d a l s o w o r k w i t h f l o a t i n g p o i n t n u m b e r s e x p e c t ( M a t h . a b s ( - 1 . 2 3 ) ) . t o . e q u a l ( 1 . 2 3 ) ; e x p e c t ( M a t h . a b s ( 1 . 2 3 ) ) . t o . e q u a l ( 1 . 2 3 ) ; / / o h , a n d j a v a s c r i p t k n o w s a b o u t n e g a t i v e 0 e x p e c t ( M a t h . a b s ( - 0 ) ) . t o . e q u a l ( 0 ) ;
6. Problem -- ECMAScript® Language Specification “The Number type has exactly 18437736874454810627 (that is, 264−253+3) values” -- me “The box above has 90000 pixels”
7. remember when you started learning TDD? M a t h . a b s = f u n c t i o n ( x ) { i f ( x = = = 0 ) r e t u r n 0 ; i f ( x = = = - 1 | | x = = = 1 ) r e t u r n 1 ; i f ( ( x > 1 & & x < 2 ) | | ( x > - 2 & & x < - 1 ) ) r e t u r n 1 . 2 3 ; r e t u r n 5 ; }
8. Testing with properties not to be confused with properties on JavaScript objects specify a property that holds for all / specified inputs instead of finding a proof, test random inputs similar to contracts examples use jsverify, other implementations available
9. Describing a property Humans: “whatever number we pass to abs, it should return a number greater or equal than 0.” Mathematicians: ∀ x ∈ ℝ: abs(x) ≥ 0 Programmers: v a r p = r e q u i r e ( ' j s v e r i f y ' ) ; p . c h e c k ( p . f o r a l l ( p . n u m b e r ( ) , f u n c t i o n ( x ) { v a r a c t u a l = M a t h . a b s ( x ) ; r e t u r n t y p e o f a c t u a l = = = ' n u m b e r ' & & a c t u a l > = 0 ; } ) ) ; O K , p a s s e d 1 0 0 t e s t s
10. Examples / Property / Proof
11. ∀ x ∈ ℝ: abs(x) ≥ x p . c h e c k ( p . f o r a l l ( p . n u m b e r ( ) , f u n c t i o n ( x ) { r e t u r n M a t h . a b s ( x ) > = x ; } ) ) ; E r r o r : F a i l e d a f t e r 3 t e s t s a n d 0 s h r i n k s . r n g S t a t e : 8 c 0 a 4 3 e f 8 5 c 7 6 1 d b 9 2 ; C o u n t e r e x a m p l e : 1 . 5 6 4 4 7 3 2 9 0 9 2 7 7 0 8 1 ;
12. More, um, well, examples
13. Defining the domain using filters v a r s l o w S u m = f u n c t i o n ( n ) { v a r s u m = 0 , i ; f o r ( i = 1 ; i < = n ; i + + ) { s u m + = i ; } r e t u r n s u m ; } , g a u s s S u m = f u n c t i o n ( n ) { r e t u r n ( n / 2 ) * ( n + 1 ) ; } , p o s i t i v e N u m b e r = p . s u c h t h a t ( p . i n t e g e r ( ) , f u n c t i o n ( i ) { r e t u r n i > 0 ; } ) ; p . a s s e r t ( p . f o r a l l ( p o s i t i v e N u m b e r , f u n c t i o n ( i ) { r e t u r n s l o w S u m ( i ) = = = g a u s s S u m ( i ) ; } ) ) ;
14. Defining the domain using custom generators o d d N u m b e r = f u n c t i o n ( ) { r e t u r n { a r b i t r a r y : f u n c t i o n ( r ) { v a r n = p . i n t e g e r ( ) . a r b i t r a r y ( r ) ; r e t u r n 2 * n + 1 ; } , s h r i n k : s h r i n k . n o o p , s h o w : s h o w . d e f } ; } ; d e s c r i b e ( ' o d d ' , f u n c t i o n ( ) { / / v a r o d d = f u n c t i o n ( n ) { r e t u r n n % 2 = = = 1 ; } ; v a r o d d = f u n c t i o n ( n ) { r e t u r n M a t h . a b s ( n ) % 2 = = = 1 ; } ; i t ( ' r e t u r n s t r u e f o r a l l o d d n u m b e r s ' , f u n c t i o n ( ) { p . a s s e r t ( p . f o r a l l ( o d d N u m b e r ( ) , o d d ) ) ; } ) ;
15. idempotence: ∀ x: f(x) === f(f(x)) Reusable properties v a r i s _ i d e m p o t e n t = f u n c t i o n ( g e n e r a t o r , f n ) { r e t u r n p . f o r a l l ( g e n e r a t o r , f u n c t i o n ( x ) { r e t u r n _ . i s E q u a l ( f n ( f n ( s ) ) , f n ( s ) ) ; } ) ; } ; p . a s s e r t ( i s _ i d e m p o t e n t ( p . a r r a y ( ) , _ . c o m p a c t ) ) ; p . a s s e r t ( i s _ i d e m p o t e n t ( p . a r r a y ( ) , _ . u n i q ) ) ;
16. Shrinking Counterexamples p . a s s e r t ( p . f o r a l l ( p . a r r a y ( p . i n t e g e r ( ) ) , f u n c t i o n ( l ) { r e t u r n _ . a l l ( l , f u n c t i o n ( n ) { r e t u r n n ! = 3 ; } ) ; } ) ) ; E r r o r : F a i l e d a f t e r 2 7 t e s t s a n d 4 s h r i n k s . r n g S t a t e : 8 8 3 6 1 e d 7 8 2 a 9 c 0 b 4 5 f ; C o u n t e r e x a m p l e : [ 3 ] ;
17. Common use cases f ( x ) > = y / / a s s e r t i n g t h e f u n c t i o n ' s r a n g e f ( x ) = = = f ( f ( x ) ) ; / / i d e m p o t e n c e a ( x ) = = = b ( x ) ; / / r e g r e s s i o n t e s t f o r r e i m p l e m e n t a t i o n n e w C 1 ( ) . f ( x ) = = = n e w C 2 ( ) . f ( x ) ; / / c l o s e l y r e l a t e d n e w C ( c 1 ) . f ( x ) = = = n e w C ( c 2 ) . f ( x ) ; / / c l o s e l y r e l a t e d m a x ( a , b ) = = = m a x ( b , a ) ) ; / / c o m m u t a t i v i t y z o o m ( z o o m ( i m g , n ) , - n ) = = = i m g ) ; / / i n v e r t i b i l i t y
18. Bonus crazieness What about higher-order functions? “If we are to check properties involving function valued variables, then we must be able to generate arbitrary functions. Rather surprisingly, we are able to do so.” p . a s s e r t ( p . f o r a l l ( p . a r r a y ( ) , p . f n ( ) , p . v a l u e ( ) , f u n c t i o n ( a , f , c ) { r e t u r n _ . m a p ( a , f , c ) . l e n g t h = = = a . l e n g t h ; } ) ) ;
19. Conclusion finding good properties makes you think harder very functional style, but should work on testing OO code not a replacement for TDD using examples but can be used to help finding missed edge cases best for unit testing (because of high number of test cases) also good for verifying assumptions on 3dparty code