Mutation testing in PHP with Humbug

Exported pdf slides of my talk at dpc16. You can find an up to date slide deck with animations etc at http://mutation.markredeman.nl

---

Mutation testing is a technique that measures the quality of a test suite and helps you write more robust code. This is done by making small changes (mutations) to code under the assumption that each mutation introduces a bug. By automating this process we can find bugs in our code that can't be detected by traditional code coverage tools.

This talk introduces the concept of mutation testing. First we show that using code coverage as a metric of quality of a test suite is flawed. Next we show that by using mutation testing we can find bugs in code which has 100% code coverage. An overview is given on how a mutation testing tool works and some of its disadvantages are discussed. Most of these disadvantages however, can be solved by writing well designed software. Lastly we will take a look at Humbug, a mutation testing tool for PHP and we conclude with a few examples of how a mutation testing tool can help you improve your software and how it can fit in your current workflow.

1. MUTATION TESTING IN PHP WITH HUMBUG
2. MARKREDEMAN STUDENT APPLIED MATHEMATICS FREELANCE (WEB) DEVELOPER
3. Testing Makes us more comfortable with changing code
4. Code Coverage Gives us an indication how well our software is tested * not yet supported by P H P _ C o d e C o v e r a g e Line coverage Function and Method coverage Class and trait coverage Opcode coverage * Branch coverage *
5. Code coverage alone is not enough
6. 100% cod coverag p u b l i c f u n c t i o n i s N o t A S e r v e r E r r o r ( ) { r e t u r n $ t h i s - > s t a t u s C o d e ( ) < 5 0 0 | | $ t h i s - > s t a t u s C o d e ( ) > = 6 0 0 ; } p u b l i c f u n c t i o n t e s t _ a _ s u c c e s s f u l _ r e s p o n s e _ i s _ n o t _ a _ s e r v e r _ e r r o r ( ) { $ r e s p o n s e = n e w R e s p o n s e ( 2 0 0 ) ; $ t h i s - > a s s e r t T r u e ( $ r e s p o n s e - > i s N o t A S e r v e r E r r o r ( ) ) ; } But the second condition is never called!
7. Still 100% line coverage p u b l i c f u n c t i o n h a n d l e ( C o m m a n d $ c o m m a n d ) { $ t h i s - > l o g g e r - > h a n d l i n g C o m m a n d ( $ c o m m a n d ) ; $ t h i s - > h a n d l e r - > h a n d l e ( $ c o m m a n d ) ; $ t h i s - > l o g g e r - > h a n d e d C o m m a n d ( $ c o m m a n d ) ; } $ i n t e r n a l H a n d l e r = $ t h i s - > p r o p h e s i z e ( C o m m a n d H a n d l e r : : c l a s s ) ; $ l o g g e r = $ t h i s - > p r o p h e s i z e ( C o m m a n d L o g g e r : : c l a s s ) ; $ h a n d l e r = n e w C o m m a n d L o g g e r ( $ i n t e r n a l H a n d l e r - > r e v e a l ( ) , $ l o g g e r - > r e v e a l ( ) ) ; $ h a n d l e r - > h a n d l e ( $ c o m m a n d ) ; / / W e f o r g o t t o c h e c k t h e i n t e r n a l h a n d l e r ! $ l o g g e r - > h a n d l i n g C o m m a n d ( $ c o m m a n d ) - > s h o u l d H a v e B e e n C a l l e d ( ) ; $ l o g g e r - > h a n d l e d C o m m a n d ( $ c o m m a n d ) - > s h o u l d H a v e B e e n C a l l e d ( ) ;
8. Using mutation testing, we can solve these problems
9. WHAT IS MUTATION TESTING? A tool to provide insight in stability of your code Noti es about uncovered code or missing specs Warns you about redundant and dead code
10. Original customer class p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s - > o r d e r s > 1 0 ; } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s - > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) - > i s G o l d C u s t o m e r ( ) ) ; $ t h i s - > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) - > i s G o l d C u s t o m e r ( ) ) ; }
11. Mutated customer class The new code is called the mutant. p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { - r e t u r n $ t h i s - > o r d e r s > 1 0 ; + r e t u r n $ t h i s - > o r d e r s > = 1 0 ; } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s - > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) - > i s G o l d C u s t o m e r ( ) ) ; $ t h i s - > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) - > i s G o l d C u s t o m e r ( ) ) ; } Still passes all assertions! (The mutant is alive / has escaped)
12. Killing the mutant p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { - r e t u r n $ t h i s - > o r d e r s > 1 0 ; + r e t u r n $ t h i s - > o r d e r s > = 1 0 ; } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { - $ t h i s - > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) - > i s G o l d C u s t o m e r ( ) ) ; + $ t h i s - > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 1 0 ) ) - > i s G o l d C u s t o m e r ( ) ) ; $ t h i s - > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) - > i s G o l d C u s t o m e r ( ) ) ; } The new test has killed the mutant
13. Mutation Testing 1. Run the test suite, if something fails: stop and x the code 2. For each le, nd and create all possible mutations 3. For each mutant, run the test suite on the mutated code 4. Analyze the results
14. (Humbug also uses the concept of timeout, fatal error and uncovered mutants) Mutation Code that has been mutated by a mutation operator A mutation can either be killed, or it can escape A mutation is said to be killed if at least 1 test fails A mutation is said to have escaped if the test suite passes
15. Uncovered Mutant p u b l i c f u n c t i o n l o c a t i o n ( ) { i f ( i s s e t ( $ t h i s - > l o c a t i o n ) ) { - r e t u r n $ t h i s - > l o c a t i o n ; + $ t h i s - > l o c a t i o n ; r e t u r n ; } r e t u r n L o c a t i o n : : u n s p e c i f i e d ( ) ; } p u b l i c f u n c t i o n t e s t _ a _ l o c a t i o n _ c a n _ b e _ u n s p e c i f i e d ( ) { $ a c t i v i t y = A c t i v i t y : : p l a n ( . . . ) ; $ t h i s - > a s s e r t E q u a l s ( $ a c t i v i t y - > l o c a t i o n ( ) , L o c a t i o n : : u n s p e c i f i e d ( ) ) ; }
16. Equivalent Mutant p r i v a t e f u n c t i o n s u m I s Z e r o ( a r r a y $ v a l u e s ) { $ s u m = 0 ; f o r e a c h ( $ v a l u e s a s $ v a l u e ) { - $ s u m + = $ v a l u e ; + $ s u m - = $ v a l u e ; } r e t u r n $ s u m = = = 0 ; } Not a problem: refactor and use the std library p r i v a t e f u n c t i o n s u m I s Z e r o ( a r r a y $ v a l u e s ) { r e t u r n a r r a y _ s u m ( $ v a l u e s ) = = = 0 ; }
17. But how do we generate mutants?
18. Come in di erent avors: Binary Arithmetic MUTATION OPERATOR Operator that changes (mutates) a piece of code Boolean Substitution Conditional Boundaries Return values And more...
19. Original Mutated + - - + * / / * % * p u b l i c f u n c t i o n c a l c u l a t e M S I ( i n t $ t o t a l K i l l e d = 0 , i n t $ t o t a l M u t a t i o n s = 0 ) : f l o a t { - r e t u r n 1 0 0 * $ t o t a l K i l l e d / $ t o t a l M u t a t i o n s ; + r e t u r n 1 0 0 * $ t o t a l K i l l e d * $ t o t a l M u t a t i o n s ; } BINARY ARITHMETIC OPERATOR
20. Come in di erent avors: Boolean Substitution MUTATION OPERATOR Operator that changes (mutates) a piece of code Binary Arithmetic Conditional Boundaries Return values And more...
21. Original Mutated t r u e f a l s e f a l s e t r u e & & | | | | & & a n d o r o r a n d ! BOOLEAN SUBSTITUTION OPERATOR
22. Come in di erent avors: Conditional Boundaries MUTATION OPERATOR Operator that changes (mutates) a piece of code Binary Arithmetic Boolean Substitution Return values And more...
23. Original Mutated > > = < < = > = > < = < We've already seen this mutaiton in one of the previous examples CONDITIONAL BOUNDARY OPERATOR
24. Come in di erent avors: Return values MUTATION OPERATOR Operator that changes (mutates) a piece of code Binary Arithmetic Boolean Substitution Conditional Boundaries And more...
25. Original Mutated r e t u r n t r u e ; r e t u r n f a l s e ; r e t u r n 1 . 0 ; r e t u r n - ( + 1 ) ; r e t u r n $ t h i s ; r e t u r n n u l l ; r e t u r n ( s t m t ) ; ( s t m t ) ; r e t u r n n u l l ; r e t u r n 1 . 0 ; r e t u r n 0 . 0 ; Very useful due to php's dynamic types RETURN VALUES OPERATOR
26. Come in di erent avors: And more... MUTATION OPERATOR Operator that changes (mutates) a piece of code Binary Arithmetic Boolean Substitution Conditional Boundaries Return values
27. Negated Conditionals negates conditional, i.e. ! = = becomes = = = Increments interchanges + + and - - Literal Numbers changes literal int and oat values (useful for checking boundaries)
28. Semantic Mutations Consider a "DateTime" mutator - $ d a t e = n e w D a t e T i m e ( $ _ G E T [ ' s i n c e ' ] ) ; + $ d a t e = D a t e T i m e : : c r e a t e F r o m F o r m a t ( D a t e T i m e : : I S O 8 6 0 1 , $ _ G E T [ ' s i n c e ' ] ) ; blog.blockscore.com/how-to-write-better-code-using-mutation-testing
29. Semantic Mutations Or the " nd" mutator - $ u s e r = U s e r : : f i n d ( $ i d ) ; + $ u s e r = U s e r : : f i n d O r F a i l ( $ i d ) ; Take away: mutations do not have to be bad, sometimes you should replace the original source by the mutation
30. Recap 1. Run the test suite, if something fails: stop and x the code 2. For each le, nd and create all possible mutations 3. For each mutant, run the test suite on the mutated code 4. Analyze the results
31. Metrics Mutation Score Indicator (MSI): percentage of mutants covered & killed by tests Mutation Code Coverage: percentage of mutants covered by tests Covered Code MSI: percentage of killed mutants that were coverd by tests
32. Metrics Example Tests: 361 Line Coverage: 64.86% 653 Mutants were generated 284 mutants were killed 218 mutants were not covered by tests 131 covered mutants were not detected 17 fatal errors were encountered 3 time outs were encountered
33. Mutatio Scor Indicator msi = = = 0.47. Out of 653 mutants, 284 mutants were killed. total mutants killed mutants 653 284
34. Mutatio Cod Coverag mcc = = = 0.67. Out of 653 mutants, 218 mutants were not covered. total mutants covered mutants 653 653−218
35. Covere Cod MSI cc msi = = = 0.70. Out of 653 mutants, 435 mutants were covered of which 284 were killed. covered mutants killed mutants 435 284
36. A summary A project with 361 tests and 65% code coverage 653 mutants were generated of which only 47% were killed and only 67% were covered. Mutation Score Indicator: 47% Mutation Code Coverage: 67% Covered Code MSI: 70%
37. Implementing a mutation testing tool How do we implement an operator? How can we make it faster?
38. A mutation operator can be applied by changing The lesystem The Abstract Syntax Tree (AST) The bytecode / opcode Perhaps PHP 7 can change this Implementing a mutation operator Humbug implements its mutations by changing les of the lesystem
39. c l a s s M u l t i p l i c a t i o n e x t e n d s M u t a t o r A b s t r a c t { / * * * R e p l a c e ( * ) w i t h ( / ) * / p u b l i c s t a t i c f u n c t i o n g e t M u t a t i o n ( a r r a y & $ t o k e n s , $ i n d e x ) { $ t o k e n s [ $ i n d e x ] = ' / ' ; } p u b l i c s t a t i c f u n c t i o n m u t a t e s ( a r r a y & $ t o k e n s , $ i n d e x ) { $ t = $ t o k e n s [ $ i n d e x ] ; i f ( ! i s _ a r r a y ( $ t ) & & $ t = = ' * ' ) { r e t u r n t r u e ; } r e t u r n f a l s e ; } }
40. Test Selection per mutation Running all tests for each mutation is ine cient Before running tests, gather coverage data from phpunit Next only run the tests that cover the mutated code Stop testing a mutation as soon as at least 1 test fails Sort the test on their execution time
41. HUMBUG A Mutation Testing framework for PHP Measures the real e ectiveness of your test suites and assist in their improvement. It eats Code Coverage for breakfast
42. Installation Globally using Git Globally using phar les Globally using composer Locally per project using composer
43. Installing locally using composer c o m p o s e r r e q u i r e ' h u m b u g / h u m b u g = ~ 1 . 0 @ d e v ' v e n d o r / b i n / h u m b u g
44. Con guration Create a con guration le using, h u m b u g c o n f i g u r e giving you a cli interface for con guring: What directories to include as mutation targets What directories to exclude Default timeout How to store the results
45. Comand line options Timeout: h u m b u g - - t i m e o u t = 1 0 Restricting les: h u m b u g - - f i l e = P r i m e F a c t o r . p h p h u m b u g - - f i l e = * D r i v e r . p h p
46. Comand line options Incremental analysis: h u m b u g - - i n c r e m e n t a l Cache results locally. Experimental: can have issues when using inheritance..
47. Analyzing results Results can be stored in text format or json format
48. Results in text format Contains a list of escaped mutants with their di s Also contains output send to stderr in case something went 2 ) \ H u m b u g \ M u t a t o r \ C o n d i t i o n a l B o u n d a r y \ G r e a t e r T h a n D i f f o n \ C a r b o n \ C a r b o n I n t e r v a l : : _ _ c o n s t r u c t ( ) i n / C a r b o n / s r c / C a r b o n / C a r b o n I n t e r v a l . p h p : - - - O r i g i n a l + + + N e w @ @ @ @ $ s p e c D a y s + = $ w e e k s > 0 ? $ w e e k s * C a r b o n : : D A Y S _ P E R _ W E E K : 0 ; - $ s p e c D a y s + = $ d a y s > 0 ? $ d a y s : 0 ; + $ s p e c D a y s + = $ d a y s > = 0 ? $ d a y s : 0 ; $ s p e c . = ( $ s p e c D a y s > 0 ) ? $ s p e c D a y s . s t a t i c : : P E R I O D _ D A Y S : ' ' ; i f ( $ h o u r s > 0 | | $ m i n u t e s > 0 | | $ s e c o n d s > 0 ) { $ s p e c . = s t a t i c : : P E R I O D _ T I M E _ P R E F I X ; $ s p e c . = $ h o u r s > 0 ? $ h o u r s . s t a t i c : : P E R I O D _ H O U R S : ' ' ;
49. Results in json format Can potentially be used by CI services " s u m m a r y " : { " t o t a l " : 2 6 , " k i l l s " : 2 3 , " e s c a p e s " : 1 , " e r r o r s " : 0 , " t i m e o u t s " : 0 , " n o t e s t s " : 2 , " c o v e r e d _ s c o r e " : 9 6 , " c o m b i n e d _ s c o r e " : 8 8 , " m u t a t i o n _ c o v e r a g e " : 9 2 } , " e s c a p e d " : [ { " f i l e " : " s r c \ / B r o a d w a y D e m o \ / B a s k e t \ / B a s k e t . p h p " , " m u t a t o r " : " \ \ H u m b u g \ \ M u t a t o r \ \ C o n d i t i o n a l B o u n d a r y \ \ G r e a t e r T h a n " , " c l a s s " : " \ \ B r o a d w a y D e m o \ \ B a s k e t \ \ B a s k e t " , " m e t h o d " : " p r o d u c t I s I n B a s k e t " , " l i n e " : 1 0 1 , " d i f f " : " - - - O r i g i n a l \ n + + + N e w \ n @ @ @ @ \ n { \ n - r e t u r n i s s e t ( $ t h i s - > p r o d u c t C o u n t B y I d [ " t e s t s " : [ " B r o a d w a y D e m o \ \ B a s k e t \ \ A d d P r o d u c t T o B a s k e t T e s t : : i t _ a d d s _ a _ p r o d u c t _ t o _ a _ b a s k e t " , Contains all covered mutants: escaped, errored, timeout, killed
50. Time for a demo
51. How and when to use it
53. Use it while code reviewing
54. Mutation Testing Better Code By Making Bugs
55. Ruby: Mutant (https://github.com/mbj/mutant) Java: Pitest (http://pitest.org/) C#: Ninjaturtles (http://ninjaturtles.codeplex.com/) Python: Cosmic Ray (https://github.com/sixty-north/cosmic- ray) Matlab: MatMute (http://matmute.sourceforge.net/) Tools for other languages
56. Concluding remarks Mutation testing will improve the quality of your tests Is becoming more mainstream over the last years Write small (fast) tests
57. Maintainer of Mutant, the ruby mutation testing tool Gave me very useful feedback and insights Thank you Markus Schrip
58. I have not failed. I've just found 10,000 ways that won't work - Thomas Edison
59. Thanks for listening Find these slides at mutation.markredeman.nl