MUTATION TESTING IN PHP WITH HUMBUG 

MARKREDEMAN STUDENT APPLIED MATHEMATICS FREELANCE (WEB) DEVELOPER 

Testing Makes us more comfortable with changing code 

Code Coverage Gives us an indication how well our software is tested * not yet supported by P H P _ C o d e C o v e r a g e Line coverage Function and Method coverage Class and trait coverage Opcode coverage * Branch coverage * 

Code coverage alone is not enough 

100% cod coverag p u b l i c f u n c t i o n i s N o t A S e r v e r E r r o r ( ) { r e t u r n $ t h i s - > s t a t u s C o d e ( ) < 5 0 0 | | $ t h i s - > s t a t u s C o d e ( ) > = 6 0 0 ; } p u b l i c f u n c t i o n t e s t _ a _ s u c c e s s f u l _ r e s p o n s e _ i s _ n o t _ a _ s e r v e r _ e r r o r ( ) { $ r e s p o n s e = n e w R e s p o n s e ( 2 0 0 ) ; $ t h i s - > a s s e r t T r u e ( $ r e s p o n s e - > i s N o t A S e r v e r E r r o r ( ) ) ; } But the second condition is never called! 

Still 100% line coverage p u b l i c f u n c t i o n h a n d l e ( C o m m a n d $ c o m m a n d ) { $ t h i s - > l o g g e r - > h a n d l i n g C o m m a n d ( $ c o m m a n d ) ; $ t h i s - > h a n d l e r - > h a n d l e ( $ c o m m a n d ) ; $ t h i s - > l o g g e r - > h a n d e d C o m m a n d ( $ c o m m a n d ) ; } $ i n t e r n a l H a n d l e r = $ t h i s - > p r o p h e s i z e ( C o m m a n d H a n d l e r : : c l a s s ) ; $ l o g g e r = $ t h i s - > p r o p h e s i z e ( C o m m a n d L o g g e r : : c l a s s ) ; $ h a n d l e r = n e w C o m m a n d L o g g e r ( $ i n t e r n a l H a n d l e r - > r e v e a l ( ) , $ l o g g e r - > r e v e a l ( ) ) ; $ h a n d l e r - > h a n d l e ( $ c o m m a n d ) ; / / W e f o r g o t t o c h e c k t h e i n t e r n a l h a n d l e r ! $ l o g g e r - > h a n d l i n g C o m m a n d ( $ c o m m a n d ) - > s h o u l d H a v e B e e n C a l l e d ( ) ; $ l o g g e r - > h a n d l e d C o m m a n d ( $ c o m m a n d ) - > s h o u l d H a v e B e e n C a l l e d ( ) ; 

Using mutation testing, we can solve these problems 

WHAT IS MUTATION TESTING? A tool to provide insight in stability of your code Noti es about uncovered code or missing specs Warns you about redundant and dead code 

Original customer class p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s - > o r d e r s > 1 0 ; } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s - > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) - > i s G o l d C u s t o m e r ( ) ) ; $ t h i s - > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) - > i s G o l d C u s t o m e r ( ) ) ; } 

Mutated customer class The new code is called the mutant. p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { - r e t u r n $ t h i s - > o r d e r s > 1 0 ; + r e t u r n $ t h i s - > o r d e r s > = 1 0 ; } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s - > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) - > i s G o l d C u s t o m e r ( ) ) ; $ t h i s - > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) - > i s G o l d C u s t o m e r ( ) ) ; } Still passes all assertions! (The mutant is alive / has escaped) 

Killing the mutant p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { - r e t u r n $ t h i s - > o r d e r s > 1 0 ; + r e t u r n $ t h i s - > o r d e r s > = 1 0 ; } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { - $ t h i s - > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) - > i s G o l d C u s t o m e r ( ) ) ; + $ t h i s - > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 1 0 ) ) - > i s G o l d C u s t o m e r ( ) ) ; $ t h i s - > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) - > i s G o l d C u s t o m e r ( ) ) ; } The new test has killed the mutant 

Mutation Testing 1. Run the test suite, if something fails: stop and x the code 2. For each le, nd and create all possible mutations 3. For each mutant, run the test suite on the mutated code 4. Analyze the results 

(Humbug also uses the concept of timeout, fatal error and uncovered mutants) Mutation Code that has been mutated by a mutation operator A mutation can either be killed, or it can escape A mutation is said to be killed if at least 1 test fails A mutation is said to have escaped if the test suite passes 

Uncovered Mutant p u b l i c f u n c t i o n l o c a t i o n ( ) { i f ( i s s e t ( $ t h i s - > l o c a t i o n ) ) { - r e t u r n $ t h i s - > l o c a t i o n ; + $ t h i s - > l o c a t i o n ; r e t u r n ; } r e t u r n L o c a t i o n : : u n s p e c i f i e d ( ) ; } p u b l i c f u n c t i o n t e s t _ a _ l o c a t i o n _ c a n _ b e _ u n s p e c i f i e d ( ) { $ a c t i v i t y = A c t i v i t y : : p l a n ( . . . ) ; $ t h i s - > a s s e r t E q u a l s ( $ a c t i v i t y - > l o c a t i o n ( ) , L o c a t i o n : : u n s p e c i f i e d ( ) ) ; } 

Equivalent Mutant p r i v a t e f u n c t i o n s u m I s Z e r o ( a r r a y $ v a l u e s ) { $ s u m = 0 ; f o r e a c h ( $ v a l u e s a s $ v a l u e ) { - $ s u m + = $ v a l u e ; + $ s u m - = $ v a l u e ; } r e t u r n $ s u m = = = 0 ; } Not a problem: refactor and use the std library p r i v a t e f u n c t i o n s u m I s Z e r o ( a r r a y $ v a l u e s ) { r e t u r n a r r a y _ s u m ( $ v a l u e s ) = = = 0 ; } 

But how do we generate mutants? 

Come in di erent avors: Binary Arithmetic MUTATION OPERATOR Operator that changes (mutates) a piece of code Boolean Substitution Conditional Boundaries Return values And more... 

Original Mutated + - - + * / / * % * p u b l i c f u n c t i o n c a l c u l a t e M S I ( i n t $ t o t a l K i l l e d = 0 , i n t $ t o t a l M u t a t i o n s = 0 ) : f l o a t { - r e t u r n 1 0 0 * $ t o t a l K i l l e d / $ t o t a l M u t a t i o n s ; + r e t u r n 1 0 0 * $ t o t a l K i l l e d * $ t o t a l M u t a t i o n s ; } BINARY ARITHMETIC OPERATOR 

Come in di erent avors: Boolean Substitution MUTATION OPERATOR Operator that changes (mutates) a piece of code Binary Arithmetic Conditional Boundaries Return values And more... 

Original Mutated t r u e f a l s e f a l s e t r u e & & | | | | & & a n d o r o r a n d ! BOOLEAN SUBSTITUTION OPERATOR 

Come in di erent avors: Conditional Boundaries MUTATION OPERATOR Operator that changes (mutates) a piece of code Binary Arithmetic Boolean Substitution Return values And more... 

Original Mutated > > = < < = > = > < = < We've already seen this mutaiton in one of the previous examples CONDITIONAL BOUNDARY OPERATOR 

Come in di erent avors: Return values MUTATION OPERATOR Operator that changes (mutates) a piece of code Binary Arithmetic Boolean Substitution Conditional Boundaries And more... 

Original Mutated r e t u r n t r u e ; r e t u r n f a l s e ; r e t u r n 1 . 0 ; r e t u r n - ( + 1 ) ; r e t u r n $ t h i s ; r e t u r n n u l l ; r e t u r n ( s t m t ) ; ( s t m t ) ; r e t u r n n u l l ; r e t u r n 1 . 0 ; r e t u r n 0 . 0 ; Very useful due to php's dynamic types RETURN VALUES OPERATOR 

Come in di erent avors: And more... MUTATION OPERATOR Operator that changes (mutates) a piece of code Binary Arithmetic Boolean Substitution Conditional Boundaries Return values 

Negated Conditionals negates conditional, i.e. ! = = becomes = = = Increments interchanges + + and - - Literal Numbers changes literal int and oat values (useful for checking boundaries) 

Semantic Mutations Consider a "DateTime" mutator - $ d a t e = n e w D a t e T i m e ( $ _ G E T [ ' s i n c e ' ] ) ; + $ d a t e = D a t e T i m e : : c r e a t e F r o m F o r m a t ( D a t e T i m e : : I S O 8 6 0 1 , $ _ G E T [ ' s i n c e ' ] ) ; blog.blockscore.com/how-to-write-better-code-using-mutation-testing 

Semantic Mutations Or the " nd" mutator - $ u s e r = U s e r : : f i n d ( $ i d ) ; + $ u s e r = U s e r : : f i n d O r F a i l ( $ i d ) ; Take away: mutations do not have to be bad, sometimes you should replace the original source by the mutation 

Recap 1. Run the test suite, if something fails: stop and x the code 2. For each le, nd and create all possible mutations 3. For each mutant, run the test suite on the mutated code 4. Analyze the results 

Metrics Mutation Score Indicator (MSI): percentage of mutants covered & killed by tests Mutation Code Coverage: percentage of mutants covered by tests Covered Code MSI: percentage of killed mutants that were coverd by tests 

Metrics Example Tests: 361 Line Coverage: 64.86% 653 Mutants were generated 284 mutants were killed 218 mutants were not covered by tests 131 covered mutants were not detected 17 fatal errors were encountered 3 time outs were encountered 

Mutatio Scor Indicator msi = = = 0.47. Out of 653 mutants, 284 mutants were killed. total mutants killed mutants 653 284 

Mutatio Cod Coverag mcc = = = 0.67. Out of 653 mutants, 218 mutants were not covered. total mutants covered mutants 653 653−218 

Covere Cod MSI cc msi = = = 0.70. Out of 653 mutants, 435 mutants were covered of which 284 were killed. covered mutants killed mutants 435 284 

A summary A project with 361 tests and 65% code coverage 653 mutants were generated of which only 47% were killed and only 67% were covered. Mutation Score Indicator: 47% Mutation Code Coverage: 67% Covered Code MSI: 70% 

Implementing a mutation testing tool How do we implement an operator? How can we make it faster? 

A mutation operator can be applied by changing The lesystem The Abstract Syntax Tree (AST) The bytecode / opcode Perhaps PHP 7 can change this Implementing a mutation operator Humbug implements its mutations by changing les of the lesystem 

c l a s s M u l t i p l i c a t i o n e x t e n d s M u t a t o r A b s t r a c t { / * * * R e p l a c e ( * ) w i t h ( / ) * / p u b l i c s t a t i c f u n c t i o n g e t M u t a t i o n ( a r r a y & $ t o k e n s , $ i n d e x ) { $ t o k e n s [ $ i n d e x ] = ' / ' ; } p u b l i c s t a t i c f u n c t i o n m u t a t e s ( a r r a y & $ t o k e n s , $ i n d e x ) { $ t = $ t o k e n s [ $ i n d e x ] ; i f ( ! i s _ a r r a y ( $ t ) & & $ t = = ' * ' ) { r e t u r n t r u e ; } r e t u r n f a l s e ; } } 

Test Selection per mutation Running all tests for each mutation is ine cient Before running tests, gather coverage data from phpunit Next only run the tests that cover the mutated code Stop testing a mutation as soon as at least 1 test fails Sort the test on their execution time 

HUMBUG A Mutation Testing framework for PHP Measures the real e ectiveness of your test suites and assist in their improvement. It eats Code Coverage for breakfast 

Installation Globally using Git Globally using phar les Globally using composer Locally per project using composer 

Installing locally using composer c o m p o s e r r e q u i r e ' h u m b u g / h u m b u g = ~ 1 . 0 @ d e v ' v e n d o r / b i n / h u m b u g 

Con guration Create a con guration le using, h u m b u g c o n f i g u r e giving you a cli interface for con guring: What directories to include as mutation targets What directories to exclude Default timeout How to store the results 

Comand line options Timeout: h u m b u g - - t i m e o u t = 1 0 Restricting les: h u m b u g - - f i l e = P r i m e F a c t o r . p h p h u m b u g - - f i l e = * D r i v e r . p h p 

Comand line options Incremental analysis: h u m b u g - - i n c r e m e n t a l Cache results locally. Experimental: can have issues when using inheritance.. 

Analyzing results Results can be stored in text format or json format 

Results in text format Contains a list of escaped mutants with their di s Also contains output send to stderr in case something went 2 ) \ H u m b u g \ M u t a t o r \ C o n d i t i o n a l B o u n d a r y \ G r e a t e r T h a n D i f f o n \ C a r b o n \ C a r b o n I n t e r v a l : : _ _ c o n s t r u c t ( ) i n / C a r b o n / s r c / C a r b o n / C a r b o n I n t e r v a l . p h p : - - - O r i g i n a l + + + N e w @ @ @ @ $ s p e c D a y s + = $ w e e k s > 0 ? $ w e e k s * C a r b o n : : D A Y S _ P E R _ W E E K : 0 ; - $ s p e c D a y s + = $ d a y s > 0 ? $ d a y s : 0 ; + $ s p e c D a y s + = $ d a y s > = 0 ? $ d a y s : 0 ; $ s p e c . = ( $ s p e c D a y s > 0 ) ? $ s p e c D a y s . s t a t i c : : P E R I O D _ D A Y S : ' ' ; i f ( $ h o u r s > 0 | | $ m i n u t e s > 0 | | $ s e c o n d s > 0 ) { $ s p e c . = s t a t i c : : P E R I O D _ T I M E _ P R E F I X ; $ s p e c . = $ h o u r s > 0 ? $ h o u r s . s t a t i c : : P E R I O D _ H O U R S : ' ' ; 

Results in json format Can potentially be used by CI services " s u m m a r y " : { " t o t a l " : 2 6 , " k i l l s " : 2 3 , " e s c a p e s " : 1 , " e r r o r s " : 0 , " t i m e o u t s " : 0 , " n o t e s t s " : 2 , " c o v e r e d _ s c o r e " : 9 6 , " c o m b i n e d _ s c o r e " : 8 8 , " m u t a t i o n _ c o v e r a g e " : 9 2 } , " e s c a p e d " : [ { " f i l e " : " s r c \ / B r o a d w a y D e m o \ / B a s k e t \ / B a s k e t . p h p " , " m u t a t o r " : " \ \ H u m b u g \ \ M u t a t o r \ \ C o n d i t i o n a l B o u n d a r y \ \ G r e a t e r T h a n " , " c l a s s " : " \ \ B r o a d w a y D e m o \ \ B a s k e t \ \ B a s k e t " , " m e t h o d " : " p r o d u c t I s I n B a s k e t " , " l i n e " : 1 0 1 , " d i f f " : " - - - O r i g i n a l \ n + + + N e w \ n @ @ @ @ \ n { \ n - r e t u r n i s s e t ( $ t h i s - > p r o d u c t C o u n t B y I d [ " t e s t s " : [ " B r o a d w a y D e m o \ \ B a s k e t \ \ A d d P r o d u c t T o B a s k e t T e s t : : i t _ a d d s _ a _ p r o d u c t _ t o _ a _ b a s k e t " , Contains all covered mutants: escaped, errored, timeout, killed 

Time for a demo 

How and when to use it 

No content

Use it while code reviewing 

Mutation Testing Better Code By Making Bugs 

Ruby: Mutant (https://github.com/mbj/mutant) Java: Pitest (http://pitest.org/) C#: Ninjaturtles (http://ninjaturtles.codeplex.com/) Python: Cosmic Ray (https://github.com/sixty-north/cosmic- ray) Matlab: MatMute (http://matmute.sourceforge.net/) Tools for other languages 

Concluding remarks Mutation testing will improve the quality of your tests Is becoming more mainstream over the last years Write small (fast) tests 

Maintainer of Mutant, the ruby mutation testing tool Gave me very useful feedback and insights Thank you Markus Schrip 

I have not failed. I've just found 10,000 ways that won't work - Thomas Edison 

Thanks for listening Find these slides at mutation.markredeman.nl