Mutation Testint in PHP with Humbug

Mutation testing is a technique that measures the quality of a test suite. This is done by making small changes (mutations) to our code, which will break our application, and ensuring that there is at least one test that no longer passes.

This talk introduces the concept of mutation testing.We show that by using mutation testing we can find bugs in code which has a 100% code coverage. We will look at a few open source projects and use the padraic/humbug package to analyze their Mutation Score Indicator. Lastly we look at some of the disadvantages of mutation testing, namely performance and false positives.

These slides can also be found at:
http://mutation.markredeman.nl

Transcript

1. MUTATION TESTING IN PHP WITH HUMBUG
2. @MARKREDEMAN STUDENT APPLIED MATHEMATICS FREELANCE (WEB) DEVELOPER
3. Introduction to mutation testing Humbug, a mutation testing framework for PHP. Analyzing code coverage of open source projects Improving your workflow with mutation testing What's this talk about?
4. A tool to analyze stability of a piece of code Similar to code coverage, but better Can find missing tests WHAT IS MUTATION TESTING?
5. A quick example c l a s s C u s t o m e r { p r i v a t e $ o r d e r s = 0 ; p u b l i c f u n c t i o n _ _ c o n s t r u c t ( $ o r d e r s ) { $ t h i s ­ > o r d e r s = $ o r d e r s ; } p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s ­ > o r d e r s > 1 0 ; } } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s ­ > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; / / 1 0 0 % l i n e c o v e r a g e : D $ t h i s ­ > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; } p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s ­ > o r d e r s > = 1 0 ; }
6. c l a s s C u s t o m e r { p r i v a t e $ o r d e r s = 0 ; p u b l i c f u n c t i o n _ _ c o n s t r u c t ( $ o r d e r s ) { $ t h i s ­ > o r d e r s = $ o r d e r s ; } p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s ­ > o r d e r s > 1 0 ; } } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s ­ > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; / / 1 0 0 % l i n e c o v e r a g e : D $ t h i s ­ > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 1 0 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; $ t h i s ­ > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; }
7. Generated mutants are similar to real faults - Andrew, Briand, Labiche, ICSE 2005
8. Some Definitions
9. a piece of code that has been changed (mutated) by a mutator killed if at least 1 test fails escaped if at all test pass equivalent if there does not exist a test case which can distinguish the mutant from the original code uncovered mutant is not covered by a test fatal mutant produces a fatal error timout unit tests exceed allowed timeout Mutation
10. Killed Mutant c l a s s C u s t o m e r { p r i v a t e $ o r d e r s = 0 ; p u b l i c f u n c t i o n _ _ c o n s t r u c t ( $ o r d e r s ) { $ t h i s ­ > o r d e r s = $ o r d e r s ; } p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s ­ > o r d e r s < 1 0 ; / / m u t a t e d " > " t o " < " } } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s ­ > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; $ t h i s ­ > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; }
11. Escaped Mutant c l a s s C u s t o m e r { p r i v a t e $ o r d e r s = 0 ; p u b l i c f u n c t i o n _ _ c o n s t r u c t ( $ o r d e r s ) { $ t h i s ­ > o r d e r s = $ o r d e r s ; } p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s ­ > o r d e r s > = 1 0 ; / / m u t a t e d " > " t o " > = " } } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s ­ > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; $ t h i s ­ > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; }
12. Equivalent Mutant f u n c t i o n s u m _ i s _ z e r o ( a r r a y $ v a l u e s ) { $ s u m = 0 ; f o r e a c h ( $ v a l u e s a s $ v a l u e ) { $ s u m + = $ v a l u e ; } r e t u r n $ s u m = = = 0 ; } f u n c t i o n s u m _ i s _ z e r o ( a r r a y $ v a l u e s ) { $ s u m = 0 ; f o r e a c h ( $ v a l u e s a s $ v a l u e ) { $ s u m ­ = $ v a l u e ; / / m u t a t e d " + " t o " ­ " } r e t u r n $ s u m = = = 0 ; }
13. Increments Mutator Decrements Mutator Invert Negatives Mutator Return Values Mutator Math Mutator Negate Mutator Conditionals Boundary Mutator Remove Conditionals Values Mutator MUTATOR Operator that changes (mutates) a piece of code
14. Original Mutated + ­ ­ + * / / * % * MATH MUTATORS
15. Original Mutated = = ! = ! = = = < = > > = < < > = > < = CONDITIONAL MUTATORS
16. :: Mutator description Negate Conditionals replace condition by ! ( c o n d i t i o n ) Remove Conditionals substitute conditional with t r u e or f a l s e Increments crement numeric values by 1 Decrements crement numeric values by 1 Invert Negatives multiply numeric values by ­ 1 Return Values remove r e t u r n statements
17. Metrics Mutation Score Indicator (MSI): percentage of mutants covered & killed by tests Mutation Code Coverage: percentage of mutants covered by tests Covered Code MSI: percentage of killed mutants that were coverd by tests $ v a n q u i s h e d T o t a l = $ k i l l e d C o u n t + $ t i m e o u t C o u n t + $ e r r o r C o u n t ; $ m e a s u r a b l e T o t a l = $ t o t a l C o u n t ­ $ u n c o v e r e d C o u n t ; / / = $ v a n q u i s h e d T o t a l + $ e s c a p e d C o u n t $ m s i = r o u n d ( 1 0 0 * ( $ v a n q u i s h e d T o t a l / $ t o t a l C o u n t ) ) ; $ c o v e r e d R a t e = r o u n d ( 1 0 0 * ( $ m e a s u r a b l e T o t a l / $ t o t a l C o u n t ) ) ; $ c c _ m s i = r o u n d ( 1 0 0 * ( $ v a n q u i s h e d T o t a l / $ m e a s u r a b l e T o t a l ) ) ;
18. Metrics Example Tests: 361 Line Coverage: 64.86% Mutation Score Indicator (MSI): 47% Mutation Code Coverage: 67% Covered Code MSI 70% 653 Mutants were generated 284 mutants were killed 218 mutants were not covered by tests 131 covered mutants were not detected 17 fatal errors were encountered 3 time outs were encountered 47% of all mutations were detected versus 65% line coverage.
19. HUMBUG A Mutation Testing framework for PHP Measures the real effectiveness of your test suites and assist in their improvement. It eats Code Coverage for breakfast
20. Git: g i t c l o n e h t t p s : / / g i t h u b . c o m / p a d r a i c / h u m b u g . g i t c d h u m b u g / p a t h / t o / c o m p o s e r . p h a r i n s t a l l b i n / h u m b u g Phar: w g e t h t t p s : / / p a d r a i c . g i t h u b . i o / h u m b u g / d o w n l o a d s / h u m b u g . p h a r w g e t h t t p s : / / p a d r a i c . g i t h u b . i o / h u m b u g / d o w n l o a d s / h u m b u g . p h a r . p u b k e y c h m o d + x h u m b u g . p h a r Composer: c o m p o s e r g l o b a l r e q u i r e ' h u m b u g / h u m b u g = ~ 1 . 0 @ d e v ' # A n d a d d ~ / . c o m p o s e r / v e n d o r / b i n t o y o u r p a t h t o m a k e i t e a s i l y e x e c u t a b l e e x p o r t P A T H = ~ / . c o m p o s e r / v e n d o r / b i n : $ P A T H Installation
21. Configuration $ > g i t c l o n e g i t @ g i t h u b . c o m : y o u r n a m e / y o u r p r o j e c t . g i t # C h e c k i f a l l t e s t s a r e g r e e n $ > p h p u n i t $ > h u m b u g c o n f i g u r e _ _ _ | | | | _ _ _ _ _ | | _ _ _ _ _ _ _ | _ _ | | | | ' \ | ' _ \ | | / _ ` | | _ | | _ | \ _ , _ | _ | _ | _ | _ . _ _ / \ _ , _ \ _ _ , | | _ _ _ / H u m b u g v e r s i o n 1 . 0 ­ d e v H u m b u g c o n f i g u r a t i o n t o o l . I t w i l l g u i d e y o u t h r o u g h H u m b u g c o n f i g u r a t i o n i n f e w s e c o n d s . W h e n c h o o s i n g d i r e c t o r i e s , y o u m a y e n t e r e a c h d i r e c t o r y a n d p r e s s r e t u r n . T o e x i t d i r e c t o r y s e l e c t i o n , p l e a s e l e a v e t h e n e x t a n s w e r b l a n k a n d p r e s s r e t u r n . W h a t s o u r c e d i r e c t o r i e s d o y o u w a n t t o i n c l u d e ? : s r c W h a t s o u r c e d i r e c t o r i e s d o y o u w a n t t o i n c l u d e ? : A n y d i r e c t o r i e s t o e x c l u d e f r o m w i t h i n y o u r s o u r c e d i r e c t o r i e s ? : S i n g l e t e s t s u i t e t i m e o u t i n s e c o n d s [ 1 0 ] : W h e r e d o y o u w a n t t o s t o r e t h e t e x t l o g ? [ h u m b u g l o g . t x t ] : W h e r e d o y o u w a n t t o s t o r e t h e j s o n l o g ( i f y o u n e e d i t ) ? : G e n e r a t e " h u m b u g . j s o n . d i s t " ? [ Y ] : C o n f i g u r a t i o n f i l e " h u m b u g . j s o n . d i s t " w a s c r e a t e d .
22. Running Humbug $ > h u m b u g _ _ _ | | | | _ _ _ _ _ | | _ _ _ _ _ _ _ | _ _ | | | | ' \ | ' _ \ | | / _ ` | | _ | | _ | \ _ , _ | _ | _ | _ | _ . _ _ / \ _ , _ \ _ _ , | | _ _ _ / H u m b u g v e r s i o n 1 . 0 ­ d e v H u m b u g r u n n i n g t e s t s u i t e t o g e n e r a t e l o g s a n d c o d e c o v e r a g e d a t a . . . 9 7 [ = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = ] 2 s e c s H u m b u g h a s c o m p l e t e d t h e i n i t i a l t e s t r u n s u c c e s s f u l l y . T e s t s : 9 7 L i n e C o v e r a g e : 9 8 . 2 7 % H u m b u g i s a n a l y s i n g s o u r c e f i l e s . . . M u t a t i o n T e s t i n g i s c o m m e n c i n g o n 1 9 f i l e s . . . ( . : k i l l e d , M : e s c a p e d , S : u n c o v e r e d , E : f a t a l e r r o r , T : t i m e d o u t ) E . . . . . . M . M . . . . . . . M M T T . . . . . M M S . . . . . . . . . . . . . . . . . . . . . . . . . . M . M . . | 6 0 ( 9 / 1 9 ) M . . . . . . . . . . . . . . . . . . M . . . . E . . . . . . . M M M M S 9 7 m u t a t i o n s w e r e g e n e r a t e d : 7 7 m u t a n t s w e r e k i l l e d 2 m u t a n t s w e r e n o t c o v e r e d b y t e s t s 1 4 c o v e r e d m u t a n t s w e r e n o t d e t e c t e d 2 f a t a l e r r o r s w e r e e n c o u n t e r e d 2 t i m e o u t s w e r e e n c o u n t e r e d M e t r i c s : M u t a t i o n S c o r e I n d i c a t o r ( M S I ) : 8 4 % M u t a t i o n C o d e C o v e r a g e : 9 8 % C o v e r e d C o d e M S I : 8 5 % R e m e m b e r t h a t s o m e m u t a n t s w i l l i n e v i t a b l y b e h a r m l e s s ( i . e . f a l s e p o s i t i v e s ) . T i m e : 3 1 . 3 9 s e c o n d s M e m o r y : 8 . 7 5 M B H u m b u g r e s u l t s a r e b e i n g l o g g e d a s T E X T t o : h u m b u g l o g . t x t
23. Analyzing Humbug humbug.log.txt 2 ) \ H u m b u g \ M u t a t o r \ C o n d i t i o n a l B o u n d a r y \ G r e a t e r T h a n D i f f o n \ C a r b o n \ C a r b o n I n t e r v a l : : _ _ c o n s t r u c t ( ) i n / C a r b o n / s r c / C a r b o n / C a r b o n I n t e r v a l . p h p : ­ ­ ­ O r i g i n a l + + + N e w @ @ @ @ $ s p e c D a y s + = $ w e e k s > 0 ? $ w e e k s * C a r b o n : : D A Y S _ P E R _ W E E K : 0 ; ­ $ s p e c D a y s + = $ d a y s > 0 ? $ d a y s : 0 ; + $ s p e c D a y s + = $ d a y s > = 0 ? $ d a y s : 0 ; $ s p e c . = ( $ s p e c D a y s > 0 ) ? $ s p e c D a y s . s t a t i c : : P E R I O D _ D A Y S : ' ' ; i f ( $ h o u r s > 0 | | $ m i n u t e s > 0 | | $ s e c o n d s > 0 ) { $ s p e c . = s t a t i c : : P E R I O D _ T I M E _ P R E F I X ; $ s p e c . = $ h o u r s > 0 ? $ h o u r s . s t a t i c : : P E R I O D _ H O U R S : ' ' ;
24. humbug.log.json " s u m m a r y " : { " t o t a l " : 2 6 , " k i l l s " : 2 3 , " e s c a p e s " : 1 , " e r r o r s " : 0 , " t i m e o u t s " : 0 , " n o t e s t s " : 2 , " c o v e r e d _ s c o r e " : 9 6 , " c o m b i n e d _ s c o r e " : 8 8 , " m u t a t i o n _ c o v e r a g e " : 9 2 } , " e s c a p e d " : [ { " f i l e " : " s r c \ / B r o a d w a y D e m o \ / B a s k e t \ / B a s k e t . p h p " , " m u t a t o r " : " \ \ H u m b u g \ \ M u t a t o r \ \ C o n d i t i o n a l B o u n d a r y \ \ G r e a t e r T h a n " , " c l a s s " : " \ \ B r o a d w a y D e m o \ \ B a s k e t \ \ B a s k e t " , " m e t h o d " : " p r o d u c t I s I n B a s k e t " , " l i n e " : 1 0 1 , " d i f f " : " ­ ­ ­ O r i g i n a l \ n + + + N e w \ n @ @ @ @ \ n { \ n ­ r e t u r n i s s e t ( $ t h i s ­ > p r o d u c t C o u n t B y I d [ " t e s t s " : [ " B r o a d w a y D e m o \ \ B a s k e t \ \ A d d P r o d u c t T o B a s k e t T e s t : : i t _ a d d s _ a _ p r o d u c t _ t o _ a _ b a s k e t " , " B r o a d w a y D e m o \ \ B a s k e t \ \ A d d P r o d u c t T o B a s k e t T e s t : : m u l t i p l e _ p r o d u c t s _ c a n _ b e _ a d d e d _ t o _ a _ b a s k e t " " B r o a d w a y D e m o \ \ B a s k e t \ \ A d d P r o d u c t T o B a s k e t T e s t : : a _ p r o d u c t _ c a n _ b e _ a d d e d _ t o _ a _ b a s k e t _ m u l t i p l e _ t i m e s " " B r o a d w a y D e m o \ \ B a s k e t \ \ C h e c k o u t T e s t : : i t _ c h e c k s _ o u t _ a _ b a s k e t " , " B r o a d w a y D e m o \ \ B a s k e t \ \ C h e c k o u t T e s t : : i t _ c a n n o t _ c h e c k o u t _ a _ b a s k e t _ t h a t _ h a s _ b e e n _ e m p t i e d " " B r o a d w a y D e m o \ \ B a s k e t \ \ C h e c k o u t T e s t : : n o t h i n g _ h a p p e n s _ w h e n _ c h e c k i n g _ o u t _ a _ b a s k e t _ f o r _ a _ s e c o n d _ t i m e " " B r o a d w a y D e m o \ \ B a s k e t \ \ R e m o v e P r o d u c t F r o m B a s k e t T e s t : : i t _ r e m o v e s _ a _ p r o d u c t _ t h a t _ w a s _ a d d e d " " B r o a d w a y D e m o \ \ B a s k e t \ \ R e m o v e P r o d u c t F r o m B a s k e t T e s t : : i t _ d o e s _ n o t h i n g _ w h e n _ r e m o v i n g _ a _ p r o d u c t _ t h a t _ i s _ n o t _ i n _ a _ b a s k e t " " B r o a d w a y D e m o \ \ B a s k e t \ \ R e m o v e P r o d u c t F r o m B a s k e t T e s t : : i t _ o n l y _ r e m o v e s _ o n e _ i n s t a n c e _ o f _ a _ p r o d u c t " ] , " s t d e r r " : " " , " s t d o u t " : " T A P v e r s i o n 1 3 " } ] ,
25. Options Timeout: h u m b u g ­ ­ t i m e o u t = 1 0 Restricting files: h u m b u g ­ ­ f i l e = P r i m e F a c t o r . p h p h u m b u g ­ ­ f i l e = * D r i v e r . p h p Incremental analysis: h u m b u g ­ ­ i n c r e m e n t a l Can off significant performance boosts by caching previous results in / h o m e / p a d r a i c / . h u m b u g . .
26. Mutators Binary Arithmetic Boolean Substitution Conditional Boundaries Negated Conditionals Increments Return Values Literal Numbers If Statements
27. HUMBUG TEST RESULTS Package LC MSI MCC CCM Execution time carbon/carbon 61% 95% 100% 95% 2.25m symfony/event- dispatcher 85% 54% 69% 78% 20s hylianshield/validator 100% 75% 89% 85% 1.35m mathiasverraes/money 96% 92% 100% 92% 16.9s thephpleague/fractal 98% 84% 98% 85% 31s broadway/broadway- demo 96% 88% 92% 86% 5s broadway/broadway 57% 49% 56% 87% 46s Line Coverage (LC), Mutation Score Indicator (MSI), Mutation Code Coverage (MCC), Covered Code MSI (CCM) Timout option set to 2 seconds.
28. TDD workflow with mutation testing
29. Read the book
30. Other tools Ruby: Mutant Java: Pitest C#: Ninjaturtles Python: MutPy Matlab: MatMute
31. Concluding remarks Mutation testing will improve the quality of your tests Is becoming more mainstream over the last years Write small (fast) tests
32. I have not failed. I've just found 10,000 ways that won't work - Thomas Edison
33. Find these slides at mutation.markredeman.nl