MUTATION TESTING IN PHP WITH HUMBUG 

@MARKREDEMAN STUDENT APPLIED MATHEMATICS FREELANCE (WEB) DEVELOPER 

Introduction to mutation testing Humbug, a mutation testing framework for PHP. Analyzing code coverage of open source projects Improving your workflow with mutation testing What's this talk about? 

A tool to analyze stability of a piece of code Similar to code coverage, but better Can find missing tests WHAT IS MUTATION TESTING? 

A quick example c l a s s C u s t o m e r { p r i v a t e $ o r d e r s = 0 ; p u b l i c f u n c t i o n _ _ c o n s t r u c t ( $ o r d e r s ) { $ t h i s ­ > o r d e r s = $ o r d e r s ; } p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s ­ > o r d e r s > 1 0 ; } } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s ­ > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; / / 1 0 0 % l i n e c o v e r a g e : D $ t h i s ­ > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; } p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s ­ > o r d e r s > = 1 0 ; } 

c l a s s C u s t o m e r { p r i v a t e $ o r d e r s = 0 ; p u b l i c f u n c t i o n _ _ c o n s t r u c t ( $ o r d e r s ) { $ t h i s ­ > o r d e r s = $ o r d e r s ; } p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s ­ > o r d e r s > 1 0 ; } } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s ­ > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; / / 1 0 0 % l i n e c o v e r a g e : D $ t h i s ­ > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 1 0 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; $ t h i s ­ > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; } 

Generated mutants are similar to real faults - Andrew, Briand, Labiche, ICSE 2005 

Some Definitions 

a piece of code that has been changed (mutated) by a mutator killed if at least 1 test fails escaped if at all test pass equivalent if there does not exist a test case which can distinguish the mutant from the original code uncovered mutant is not covered by a test fatal mutant produces a fatal error timout unit tests exceed allowed timeout Mutation 

Killed Mutant c l a s s C u s t o m e r { p r i v a t e $ o r d e r s = 0 ; p u b l i c f u n c t i o n _ _ c o n s t r u c t ( $ o r d e r s ) { $ t h i s ­ > o r d e r s = $ o r d e r s ; } p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s ­ > o r d e r s < 1 0 ; / / m u t a t e d " > " t o " < " } } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s ­ > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; $ t h i s ­ > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; } 

Escaped Mutant c l a s s C u s t o m e r { p r i v a t e $ o r d e r s = 0 ; p u b l i c f u n c t i o n _ _ c o n s t r u c t ( $ o r d e r s ) { $ t h i s ­ > o r d e r s = $ o r d e r s ; } p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s ­ > o r d e r s > = 1 0 ; / / m u t a t e d " > " t o " > = " } } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s ­ > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; $ t h i s ­ > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; } 

Equivalent Mutant f u n c t i o n s u m _ i s _ z e r o ( a r r a y $ v a l u e s ) { $ s u m = 0 ; f o r e a c h ( $ v a l u e s a s $ v a l u e ) { $ s u m + = $ v a l u e ; } r e t u r n $ s u m = = = 0 ; } f u n c t i o n s u m _ i s _ z e r o ( a r r a y $ v a l u e s ) { $ s u m = 0 ; f o r e a c h ( $ v a l u e s a s $ v a l u e ) { $ s u m ­ = $ v a l u e ; / / m u t a t e d " + " t o " ­ " } r e t u r n $ s u m = = = 0 ; } 

Increments Mutator Decrements Mutator Invert Negatives Mutator Return Values Mutator Math Mutator Negate Mutator Conditionals Boundary Mutator Remove Conditionals Values Mutator MUTATOR Operator that changes (mutates) a piece of code 

Original Mutated + ­ ­ + * / / * % * MATH MUTATORS 

Original Mutated = = ! = ! = = = < = > > = < < > = > < = CONDITIONAL MUTATORS 

:: Mutator description Negate Conditionals replace condition by ! ( c o n d i t i o n ) Remove Conditionals substitute conditional with t r u e or f a l s e Increments crement numeric values by 1 Decrements crement numeric values by 1 Invert Negatives multiply numeric values by ­ 1 Return Values remove r e t u r n statements 

Metrics Mutation Score Indicator (MSI): percentage of mutants covered & killed by tests Mutation Code Coverage: percentage of mutants covered by tests Covered Code MSI: percentage of killed mutants that were coverd by tests $ v a n q u i s h e d T o t a l = $ k i l l e d C o u n t + $ t i m e o u t C o u n t + $ e r r o r C o u n t ; $ m e a s u r a b l e T o t a l = $ t o t a l C o u n t ­ $ u n c o v e r e d C o u n t ; / / = $ v a n q u i s h e d T o t a l + $ e s c a p e d C o u n t $ m s i = r o u n d ( 1 0 0 * ( $ v a n q u i s h e d T o t a l / $ t o t a l C o u n t ) ) ; $ c o v e r e d R a t e = r o u n d ( 1 0 0 * ( $ m e a s u r a b l e T o t a l / $ t o t a l C o u n t ) ) ; $ c c _ m s i = r o u n d ( 1 0 0 * ( $ v a n q u i s h e d T o t a l / $ m e a s u r a b l e T o t a l ) ) ; 

Metrics Example Tests: 361 Line Coverage: 64.86% Mutation Score Indicator (MSI): 47% Mutation Code Coverage: 67% Covered Code MSI 70% 653 Mutants were generated 284 mutants were killed 218 mutants were not covered by tests 131 covered mutants were not detected 17 fatal errors were encountered 3 time outs were encountered 47% of all mutations were detected versus 65% line coverage. 

HUMBUG A Mutation Testing framework for PHP Measures the real effectiveness of your test suites and assist in their improvement. It eats Code Coverage for breakfast 

Git: g i t c l o n e h t t p s : / / g i t h u b . c o m / p a d r a i c / h u m b u g . g i t c d h u m b u g / p a t h / t o / c o m p o s e r . p h a r i n s t a l l b i n / h u m b u g Phar: w g e t h t t p s : / / p a d r a i c . g i t h u b . i o / h u m b u g / d o w n l o a d s / h u m b u g . p h a r w g e t h t t p s : / / p a d r a i c . g i t h u b . i o / h u m b u g / d o w n l o a d s / h u m b u g . p h a r . p u b k e y c h m o d + x h u m b u g . p h a r Composer: c o m p o s e r g l o b a l r e q u i r e ' h u m b u g / h u m b u g = ~ 1 . 0 @ d e v ' # A n d a d d ~ / . c o m p o s e r / v e n d o r / b i n t o y o u r p a t h t o m a k e i t e a s i l y e x e c u t a b l e e x p o r t P A T H = ~ / . c o m p o s e r / v e n d o r / b i n : $ P A T H Installation 

Configuration $ > g i t c l o n e g i t @ g i t h u b . c o m : y o u r n a m e / y o u r p r o j e c t . g i t # C h e c k i f a l l t e s t s a r e g r e e n $ > p h p u n i t $ > h u m b u g c o n f i g u r e _ _ _ | | | | _ _ _ _ _ | | _ _ _ _ _ _ _ | _ _ | | | | ' \ | ' _ \ | | / _ ` | | _ | | _ | \ _ , _ | _ | _ | _ | _ . _ _ / \ _ , _ \ _ _ , | | _ _ _ / H u m b u g v e r s i o n 1 . 0 ­ d e v H u m b u g c o n f i g u r a t i o n t o o l . I t w i l l g u i d e y o u t h r o u g h H u m b u g c o n f i g u r a t i o n i n f e w s e c o n d s . W h e n c h o o s i n g d i r e c t o r i e s , y o u m a y e n t e r e a c h d i r e c t o r y a n d p r e s s r e t u r n . T o e x i t d i r e c t o r y s e l e c t i o n , p l e a s e l e a v e t h e n e x t a n s w e r b l a n k a n d p r e s s r e t u r n . W h a t s o u r c e d i r e c t o r i e s d o y o u w a n t t o i n c l u d e ? : s r c W h a t s o u r c e d i r e c t o r i e s d o y o u w a n t t o i n c l u d e ? : A n y d i r e c t o r i e s t o e x c l u d e f r o m w i t h i n y o u r s o u r c e d i r e c t o r i e s ? : S i n g l e t e s t s u i t e t i m e o u t i n s e c o n d s [ 1 0 ] : W h e r e d o y o u w a n t t o s t o r e t h e t e x t l o g ? [ h u m b u g l o g . t x t ] : W h e r e d o y o u w a n t t o s t o r e t h e j s o n l o g ( i f y o u n e e d i t ) ? : G e n e r a t e " h u m b u g . j s o n . d i s t " ? [ Y ] : C o n f i g u r a t i o n f i l e " h u m b u g . j s o n . d i s t " w a s c r e a t e d . 

Running Humbug $ > h u m b u g _ _ _ | | | | _ _ _ _ _ | | _ _ _ _ _ _ _ | _ _ | | | | ' \ | ' _ \ | | / _ ` | | _ | | _ | \ _ , _ | _ | _ | _ | _ . _ _ / \ _ , _ \ _ _ , | | _ _ _ / H u m b u g v e r s i o n 1 . 0 ­ d e v H u m b u g r u n n i n g t e s t s u i t e t o g e n e r a t e l o g s a n d c o d e c o v e r a g e d a t a . . . 9 7 [ = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = ] 2 s e c s H u m b u g h a s c o m p l e t e d t h e i n i t i a l t e s t r u n s u c c e s s f u l l y . T e s t s : 9 7 L i n e C o v e r a g e : 9 8 . 2 7 % H u m b u g i s a n a l y s i n g s o u r c e f i l e s . . . M u t a t i o n T e s t i n g i s c o m m e n c i n g o n 1 9 f i l e s . . . ( . : k i l l e d , M : e s c a p e d , S : u n c o v e r e d , E : f a t a l e r r o r , T : t i m e d o u t ) E . . . . . . M . M . . . . . . . M M T T . . . . . M M S . . . . . . . . . . . . . . . . . . . . . . . . . . M . M . . | 6 0 ( 9 / 1 9 ) M . . . . . . . . . . . . . . . . . . M . . . . E . . . . . . . M M M M S 9 7 m u t a t i o n s w e r e g e n e r a t e d : 7 7 m u t a n t s w e r e k i l l e d 2 m u t a n t s w e r e n o t c o v e r e d b y t e s t s 1 4 c o v e r e d m u t a n t s w e r e n o t d e t e c t e d 2 f a t a l e r r o r s w e r e e n c o u n t e r e d 2 t i m e o u t s w e r e e n c o u n t e r e d M e t r i c s : M u t a t i o n S c o r e I n d i c a t o r ( M S I ) : 8 4 % M u t a t i o n C o d e C o v e r a g e : 9 8 % C o v e r e d C o d e M S I : 8 5 % R e m e m b e r t h a t s o m e m u t a n t s w i l l i n e v i t a b l y b e h a r m l e s s ( i . e . f a l s e p o s i t i v e s ) . T i m e : 3 1 . 3 9 s e c o n d s M e m o r y : 8 . 7 5 M B H u m b u g r e s u l t s a r e b e i n g l o g g e d a s T E X T t o : h u m b u g l o g . t x t 

Analyzing Humbug humbug.log.txt 2 ) \ H u m b u g \ M u t a t o r \ C o n d i t i o n a l B o u n d a r y \ G r e a t e r T h a n D i f f o n \ C a r b o n \ C a r b o n I n t e r v a l : : _ _ c o n s t r u c t ( ) i n / C a r b o n / s r c / C a r b o n / C a r b o n I n t e r v a l . p h p : ­ ­ ­ O r i g i n a l + + + N e w @ @ @ @ $ s p e c D a y s + = $ w e e k s > 0 ? $ w e e k s * C a r b o n : : D A Y S _ P E R _ W E E K : 0 ; ­ $ s p e c D a y s + = $ d a y s > 0 ? $ d a y s : 0 ; + $ s p e c D a y s + = $ d a y s > = 0 ? $ d a y s : 0 ; $ s p e c . = ( $ s p e c D a y s > 0 ) ? $ s p e c D a y s . s t a t i c : : P E R I O D _ D A Y S : ' ' ; i f ( $ h o u r s > 0 | | $ m i n u t e s > 0 | | $ s e c o n d s > 0 ) { $ s p e c . = s t a t i c : : P E R I O D _ T I M E _ P R E F I X ; $ s p e c . = $ h o u r s > 0 ? $ h o u r s . s t a t i c : : P E R I O D _ H O U R S : ' ' ; 

humbug.log.json " s u m m a r y " : { " t o t a l " : 2 6 , " k i l l s " : 2 3 , " e s c a p e s " : 1 , " e r r o r s " : 0 , " t i m e o u t s " : 0 , " n o t e s t s " : 2 , " c o v e r e d _ s c o r e " : 9 6 , " c o m b i n e d _ s c o r e " : 8 8 , " m u t a t i o n _ c o v e r a g e " : 9 2 } , " e s c a p e d " : [ { " f i l e " : " s r c \ / B r o a d w a y D e m o \ / B a s k e t \ / B a s k e t . p h p " , " m u t a t o r " : " \ \ H u m b u g \ \ M u t a t o r \ \ C o n d i t i o n a l B o u n d a r y \ \ G r e a t e r T h a n " , " c l a s s " : " \ \ B r o a d w a y D e m o \ \ B a s k e t \ \ B a s k e t " , " m e t h o d " : " p r o d u c t I s I n B a s k e t " , " l i n e " : 1 0 1 , " d i f f " : " ­ ­ ­ O r i g i n a l \ n + + + N e w \ n @ @ @ @ \ n { \ n ­ r e t u r n i s s e t ( $ t h i s ­ > p r o d u c t C o u n t B y I d [ " t e s t s " : [ " B r o a d w a y D e m o \ \ B a s k e t \ \ A d d P r o d u c t T o B a s k e t T e s t : : i t _ a d d s _ a _ p r o d u c t _ t o _ a _ b a s k e t " , " B r o a d w a y D e m o \ \ B a s k e t \ \ A d d P r o d u c t T o B a s k e t T e s t : : m u l t i p l e _ p r o d u c t s _ c a n _ b e _ a d d e d _ t o _ a _ b a s k e t " " B r o a d w a y D e m o \ \ B a s k e t \ \ A d d P r o d u c t T o B a s k e t T e s t : : a _ p r o d u c t _ c a n _ b e _ a d d e d _ t o _ a _ b a s k e t _ m u l t i p l e _ t i m e s " " B r o a d w a y D e m o \ \ B a s k e t \ \ C h e c k o u t T e s t : : i t _ c h e c k s _ o u t _ a _ b a s k e t " , " B r o a d w a y D e m o \ \ B a s k e t \ \ C h e c k o u t T e s t : : i t _ c a n n o t _ c h e c k o u t _ a _ b a s k e t _ t h a t _ h a s _ b e e n _ e m p t i e d " " B r o a d w a y D e m o \ \ B a s k e t \ \ C h e c k o u t T e s t : : n o t h i n g _ h a p p e n s _ w h e n _ c h e c k i n g _ o u t _ a _ b a s k e t _ f o r _ a _ s e c o n d _ t i m e " " B r o a d w a y D e m o \ \ B a s k e t \ \ R e m o v e P r o d u c t F r o m B a s k e t T e s t : : i t _ r e m o v e s _ a _ p r o d u c t _ t h a t _ w a s _ a d d e d " " B r o a d w a y D e m o \ \ B a s k e t \ \ R e m o v e P r o d u c t F r o m B a s k e t T e s t : : i t _ d o e s _ n o t h i n g _ w h e n _ r e m o v i n g _ a _ p r o d u c t _ t h a t _ i s _ n o t _ i n _ a _ b a s k e t " " B r o a d w a y D e m o \ \ B a s k e t \ \ R e m o v e P r o d u c t F r o m B a s k e t T e s t : : i t _ o n l y _ r e m o v e s _ o n e _ i n s t a n c e _ o f _ a _ p r o d u c t " ] , " s t d e r r " : " " , " s t d o u t " : " T A P v e r s i o n 1 3 " } ] , 

Options Timeout: h u m b u g ­ ­ t i m e o u t = 1 0 Restricting files: h u m b u g ­ ­ f i l e = P r i m e F a c t o r . p h p h u m b u g ­ ­ f i l e = * D r i v e r . p h p Incremental analysis: h u m b u g ­ ­ i n c r e m e n t a l Can off significant performance boosts by caching previous results in / h o m e / p a d r a i c / . h u m b u g . . 

Mutators Binary Arithmetic Boolean Substitution Conditional Boundaries Negated Conditionals Increments Return Values Literal Numbers If Statements 

HUMBUG TEST RESULTS Package LC MSI MCC CCM Execution time carbon/carbon 61% 95% 100% 95% 2.25m symfony/event- dispatcher 85% 54% 69% 78% 20s hylianshield/validator 100% 75% 89% 85% 1.35m mathiasverraes/money 96% 92% 100% 92% 16.9s thephpleague/fractal 98% 84% 98% 85% 31s broadway/broadway- demo 96% 88% 92% 86% 5s broadway/broadway 57% 49% 56% 87% 46s Line Coverage (LC), Mutation Score Indicator (MSI), Mutation Code Coverage (MCC), Covered Code MSI (CCM) Timout option set to 2 seconds. 

TDD workflow with mutation testing 

Read the book 

Other tools Ruby: Mutant Java: Pitest C#: Ninjaturtles Python: MutPy Matlab: MatMute 

Concluding remarks Mutation testing will improve the quality of your tests Is becoming more mainstream over the last years Write small (fast) tests 

I have not failed. I've just found 10,000 ways that won't work - Thomas Edison 

Find these slides at mutation.markredeman.nl