Property-Based Testing (XP Days Germany 2014)

Transcript

1. XP Days Germany 2014 Christoph Neuroth [ ] @c089 testing with properties
2. [horribly predictable quote] “Testing shows the presence, not the absence of bugs. ” -- Edsger W. Dijkstra
3. Hoare: “ The role of testing, in theory, is to establish the base propositions of an inductive proof.[...] At present, this is mainly theory [but] seems to show a possibility of practical results, though proving correctness is a laborious and expensive process.” Perlis: “ Much of program complexity is spurious and a number of test cases properly studied will exhaust the testing problem. The problem is to isolate the right test cases, not to prove the algorithm, for that follows after the choice of the proper test cases.” Dijkstra: “Testing shows the presence, not the absence of bugs”
4. Testing with examples specify correct output for chosen inputs relatively easy to come up with good communication tool straightforward to implement straightforward to overlook important ones usually only cover a very small portion of your domain often, programmers just get "expected" from the REPL v a r a c t u a l = f u n c t i o n _ u n d e r _ t e s t ( i n p u t ) ; e x p e c t ( a c t u a l ) . t o . e q u a l ( e x p e c t e d ) ; i f ( f u n c t i o n _ u n d e r _ t e s t ( i n p u t ) ! = = e x p e c t e d ) t h r o w n e w E r r o r ( " p r i m i t i v e u n i t t e s t i n g f r a m e w o r k " ) ;
5. yay, all tests passing! / / 1 : o b v i o u s l y e x p e c t ( M a t h . a b s ( 0 ) ) . t o . e q u a l ( 0 ) ; / / 2 , 3 : n e g a t i v e v a l u e s b e c o m e p o s i t i v e e x p e c t ( M a t h . a b s ( - 1 ) ) . t o . e q u a l ( 1 ) ; e x p e c t ( M a t h . a b s ( - 5 ) ) . t o . e q u a l ( 5 ) ; / / 4 : w h i l e p o s i t i v e v a l u e s s t a y p o s i t i v e e x p e c t ( M a t h . a b s ( 1 ) ) . t o . e q u a l ( 1 ) ; / / 5 , 6 : s h o u l d a l s o w o r k w i t h f l o a t i n g p o i n t n u m b e r s e x p e c t ( M a t h . a b s ( - 1 . 2 3 ) ) . t o . e q u a l ( 1 . 2 3 ) ; e x p e c t ( M a t h . a b s ( 1 . 2 3 ) ) . t o . e q u a l ( 1 . 2 3 ) ; / / 7 : o h , a n d j a v a s c r i p t k n o w s a b o u t n e g a t i v e 0 e x p e c t ( M a t h . a b s ( - 0 ) ) . t o . e q u a l ( 0 ) ;
6. Problem -- ECMAScript® Language Specification “The Number type has exactly 18437736874454810627 (that is, 264−253+3) values” -- me “The box above has 90000 pixels”
7. remember when you started learning TDD? M a t h . a b s = f u n c t i o n ( x ) { i f ( x = = = 0 ) r e t u r n 0 ; i f ( x = = = - 1 | | x = = = 1 ) r e t u r n 1 ; i f ( ( x > 1 & & x < 2 ) | | ( x > - 2 & & x < - 1 ) ) r e t u r n 1 . 2 3 ; r e t u r n 5 ; } ... but even with TDD done right, missing test cases happen.
8. Testing with properties not to be confused with properties on JavaScript objects a form of randomized testing specify a property that holds for all / specified inputs instead of finding a proof, test random inputs similar to contracts introduced in a paper (and Haskell implementation, quickcheck) in 2000 code samples use JavaScript and jsverify, implementations available for most langauges
9. Describing a property Math.abs(x): returns the absolute value of a number "x". Humans: “whatever number we pass to Math.abs(), it should return a number greater or equal than 0.” Mathematicians: ∀ x ∈ ℝ: abs(x) ≥ 0 Programmers: v a r p = r e q u i r e ( ' j s v e r i f y ' ) ; p . c h e c k ( p . f o r a l l ( p . n u m b e r ( ) , f u n c t i o n ( x ) { v a r a c t u a l = M a t h . a b s ( x ) ; r e t u r n t y p e o f a c t u a l = = = ' n u m b e r ' & & a c t u a l > = 0 ; } ) ) ; O K , p a s s e d 1 0 0 t e s t s
10. Examples / Property / Proof
11. ∀ x ∈ ℝ: abs(x) ≥ x p . c h e c k ( p . f o r a l l ( p . n u m b e r ( ) , f u n c t i o n ( x ) { r e t u r n M a t h . a b s ( x ) > = x ; } ) ) ; E r r o r : F a i l e d a f t e r 3 t e s t s a n d 0 s h r i n k s . r n g S t a t e : 8 c 0 a 4 3 e f 8 5 c 7 6 1 d b 9 2 ; C o u n t e r e x a m p l e : 1 . 5 6 4 4 7 3 2 9 0 9 2 7 7 0 8 1 ;
12. More, um, well, examples
13. Defining the domain using filters v a r s l o w S u m = f u n c t i o n ( n ) { v a r s u m = 0 , i ; f o r ( i = 1 ; i < = n ; i + + ) { s u m + = i ; } r e t u r n s u m ; } , g a u s s S u m = f u n c t i o n ( n ) { r e t u r n ( n / 2 ) * ( n + 1 ) ; } , p o s i t i v e N u m b e r = p . s u c h t h a t ( p . i n t e g e r ( ) , f u n c t i o n ( i ) { r e t u r n i > 0 ; } ) ; p . a s s e r t ( p . f o r a l l ( p o s i t i v e N u m b e r , f u n c t i o n ( i ) { r e t u r n s l o w S u m ( i ) = = = g a u s s S u m ( i ) ; } ) ) ;
14. Defining the domain using custom generators o d d N u m b e r = f u n c t i o n ( ) { r e t u r n { a r b i t r a r y : f u n c t i o n ( r ) { v a r n = p . i n t e g e r ( ) . a r b i t r a r y ( r ) ; r e t u r n 2 * n + 1 ; } , s h r i n k : s h r i n k . n o o p , s h o w : s h o w . d e f } ; } ; d e s c r i b e ( ' o d d ' , f u n c t i o n ( ) { / / v a r o d d = f u n c t i o n ( n ) { r e t u r n n % 2 = = = 1 ; } ; v a r o d d = f u n c t i o n ( n ) { r e t u r n M a t h . a b s ( n ) % 2 = = = 1 ; } ; i t ( ' r e t u r n s t r u e f o r a l l o d d n u m b e r s ' , f u n c t i o n ( ) { p . a s s e r t ( p . f o r a l l ( o d d N u m b e r ( ) , o d d ) ) ; } ) ;
15. Reusable properties idempotence: ∀ x: f(x) === f(f(x)) v a r i s _ i d e m p o t e n t = f u n c t i o n ( g e n e r a t o r , f n ) { r e t u r n p . f o r a l l ( g e n e r a t o r , f u n c t i o n ( x ) { r e t u r n _ . i s E q u a l ( f n ( f n ( s ) ) , f n ( s ) ) ; } ) ; } ; p . a s s e r t ( i s _ i d e m p o t e n t ( p . a r r a y ( ) , _ . c o m p a c t ) ) ; p . a s s e r t ( i s _ i d e m p o t e n t ( p . a r r a y ( ) , _ . u n i q ) ) ;
16. Shrinking Counterexamples p . a s s e r t ( p . f o r a l l ( p . a r r a y ( p . i n t e g e r ( ) ) , f u n c t i o n ( l ) { r e t u r n _ . a l l ( l , f u n c t i o n ( n ) { r e t u r n n ! = 3 ; } ) ; } ) ) ; E r r o r : F a i l e d a f t e r 2 7 t e s t s a n d 4 s h r i n k s . r n g S t a t e : 8 8 3 6 1 e d 7 8 2 a 9 c 0 b 4 5 f ; C o u n t e r e x a m p l e : [ 3 ] ;
17. Common use cases f ( x ) > = y / / a s s e r t i n g t h e f u n c t i o n ' s r a n g e f ( x ) = = = f ( f ( x ) ) ; / / i d e m p o t e n c e a ( x ) = = = b ( x ) ; / / r e g r e s s i o n t e s t f o r r e i m p l e m e n t a t i o n n e w C 1 ( ) . f ( x ) = = = n e w C 2 ( ) . f ( x ) ; / / c l o s e l y r e l a t e d n e w C ( c 1 ) . f ( x ) = = = n e w C ( c 2 ) . f ( x ) ; / / c l o s e l y r e l a t e d m a x ( a , b ) = = = m a x ( b , a ) ; / / c o m m u t a t i v i t y z o o m ( z o o m ( i m g , n ) , - n ) = = = i m g ) ; / / i n v e r t i b i l i t y d e c o d e ( e n c o d e ( o r i g i n a l ) ) = = = o r i g i n a l ) ; / / i n v e r t i b i l i t y
18. Bonus crazieness What about higher-order functions? “If we are to check properties involving function valued variables, then we must be able to generate arbitrary functions. Rather surprisingly, we are able to do so.” p . a s s e r t ( p . f o r a l l ( p . a r r a y ( ) , p . f n ( ) , p . v a l u e ( ) , f u n c t i o n ( a , f , c ) { r e t u r n _ . m a p ( a , f , c ) . l e n g t h = = = a . l e n g t h ; } ) ) ;
19. pop quiz!
20. Conclusions not a replacement for TDD using examples (for me) but a great way to find missed edge cases finding good properties makes you think much harder (higher level of abstraction required, no REPL copy paste shortcut) even if no counterexample is found: new learnings, free documentation (as with other kinds of tests) very functional style, but should work on OO code also, having a spec helps (e.g. often used in automotive systems) best for unit testing (high number of test cases needs fast tests)
21. Thank you Original Paper: “QuickCheck: a lightweight tool for random testing of Haskell programs” - K Claessen, J Hughes Implementation used in this talk: github.com/phadej/jsverify Feedback welcome! @c089 / christoph.neuroth@gmail.com