Mutation testing in PHP with Humbug

Mutation testing in PHP with Humbug

Exported pdf slides of my talk at dpc16. You can find an up to date slide deck with animations etc at http://mutation.markredeman.nl

---

Mutation testing is a technique that measures the quality of a test suite and helps you write more robust code. This is done by making small changes (mutations) to code under the assumption that each mutation introduces a bug. By automating this process we can find bugs in our code that can't be detected by traditional code coverage tools.

This talk introduces the concept of mutation testing. First we show that using code coverage as a metric of quality of a test suite is flawed. Next we show that by using mutation testing we can find bugs in code which has 100% code coverage. An overview is given on how a mutation testing tool works and some of its disadvantages are discussed. Most of these disadvantages however, can be solved by writing well designed software. Lastly we will take a look at Humbug, a mutation testing tool for PHP and we conclude with a few examples of how a mutation testing tool can help you improve your software and how it can fit in your current workflow.

Fdddf387d081e976aab7bda0b64d535f?s=128

Mark Redeman

June 25, 2016
Tweet

Transcript

  1. MUTATION TESTING IN PHP WITH HUMBUG

  2. MARKREDEMAN STUDENT APPLIED MATHEMATICS FREELANCE (WEB) DEVELOPER

  3. Testing Makes us more comfortable with changing code

  4. Code Coverage Gives us an indication how well our software

    is tested * not yet supported by P H P _ C o d e C o v e r a g e Line coverage Function and Method coverage Class and trait coverage Opcode coverage * Branch coverage *
  5. Code coverage alone is not enough

  6. 100% cod coverag p u b l i c f

    u n c t i o n i s N o t A S e r v e r E r r o r ( ) { r e t u r n $ t h i s - > s t a t u s C o d e ( ) < 5 0 0 | | $ t h i s - > s t a t u s C o d e ( ) > = 6 0 0 ; } p u b l i c f u n c t i o n t e s t _ a _ s u c c e s s f u l _ r e s p o n s e _ i s _ n o t _ a _ s e r v e r _ e r r o r ( ) { $ r e s p o n s e = n e w R e s p o n s e ( 2 0 0 ) ; $ t h i s - > a s s e r t T r u e ( $ r e s p o n s e - > i s N o t A S e r v e r E r r o r ( ) ) ; } But the second condition is never called!
  7. Still 100% line coverage p u b l i c

    f u n c t i o n h a n d l e ( C o m m a n d $ c o m m a n d ) { $ t h i s - > l o g g e r - > h a n d l i n g C o m m a n d ( $ c o m m a n d ) ; $ t h i s - > h a n d l e r - > h a n d l e ( $ c o m m a n d ) ; $ t h i s - > l o g g e r - > h a n d e d C o m m a n d ( $ c o m m a n d ) ; } $ i n t e r n a l H a n d l e r = $ t h i s - > p r o p h e s i z e ( C o m m a n d H a n d l e r : : c l a s s ) ; $ l o g g e r = $ t h i s - > p r o p h e s i z e ( C o m m a n d L o g g e r : : c l a s s ) ; $ h a n d l e r = n e w C o m m a n d L o g g e r ( $ i n t e r n a l H a n d l e r - > r e v e a l ( ) , $ l o g g e r - > r e v e a l ( ) ) ; $ h a n d l e r - > h a n d l e ( $ c o m m a n d ) ; / / W e f o r g o t t o c h e c k t h e i n t e r n a l h a n d l e r ! $ l o g g e r - > h a n d l i n g C o m m a n d ( $ c o m m a n d ) - > s h o u l d H a v e B e e n C a l l e d ( ) ; $ l o g g e r - > h a n d l e d C o m m a n d ( $ c o m m a n d ) - > s h o u l d H a v e B e e n C a l l e d ( ) ;
  8. Using mutation testing, we can solve these problems

  9. WHAT IS MUTATION TESTING? A tool to provide insight in

    stability of your code Noti es about uncovered code or missing specs Warns you about redundant and dead code
  10. Original customer class p u b l i c f

    u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s - > o r d e r s > 1 0 ; } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s - > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) - > i s G o l d C u s t o m e r ( ) ) ; $ t h i s - > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) - > i s G o l d C u s t o m e r ( ) ) ; }
  11. Mutated customer class The new code is called the mutant.

    p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { - r e t u r n $ t h i s - > o r d e r s > 1 0 ; + r e t u r n $ t h i s - > o r d e r s > = 1 0 ; } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s - > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) - > i s G o l d C u s t o m e r ( ) ) ; $ t h i s - > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) - > i s G o l d C u s t o m e r ( ) ) ; } Still passes all assertions! (The mutant is alive / has escaped)
  12. Killing the mutant p u b l i c f

    u n c t i o n i s G o l d C u s t o m e r ( ) { - r e t u r n $ t h i s - > o r d e r s > 1 0 ; + r e t u r n $ t h i s - > o r d e r s > = 1 0 ; } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { - $ t h i s - > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) - > i s G o l d C u s t o m e r ( ) ) ; + $ t h i s - > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 1 0 ) ) - > i s G o l d C u s t o m e r ( ) ) ; $ t h i s - > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) - > i s G o l d C u s t o m e r ( ) ) ; } The new test has killed the mutant
  13. Mutation Testing 1. Run the test suite, if something fails:

    stop and x the code 2. For each le, nd and create all possible mutations 3. For each mutant, run the test suite on the mutated code 4. Analyze the results
  14. (Humbug also uses the concept of timeout, fatal error and

    uncovered mutants) Mutation Code that has been mutated by a mutation operator A mutation can either be killed, or it can escape A mutation is said to be killed if at least 1 test fails A mutation is said to have escaped if the test suite passes
  15. Uncovered Mutant p u b l i c f u

    n c t i o n l o c a t i o n ( ) { i f ( i s s e t ( $ t h i s - > l o c a t i o n ) ) { - r e t u r n $ t h i s - > l o c a t i o n ; + $ t h i s - > l o c a t i o n ; r e t u r n ; } r e t u r n L o c a t i o n : : u n s p e c i f i e d ( ) ; } p u b l i c f u n c t i o n t e s t _ a _ l o c a t i o n _ c a n _ b e _ u n s p e c i f i e d ( ) { $ a c t i v i t y = A c t i v i t y : : p l a n ( . . . ) ; $ t h i s - > a s s e r t E q u a l s ( $ a c t i v i t y - > l o c a t i o n ( ) , L o c a t i o n : : u n s p e c i f i e d ( ) ) ; }
  16. Equivalent Mutant p r i v a t e f

    u n c t i o n s u m I s Z e r o ( a r r a y $ v a l u e s ) { $ s u m = 0 ; f o r e a c h ( $ v a l u e s a s $ v a l u e ) { - $ s u m + = $ v a l u e ; + $ s u m - = $ v a l u e ; } r e t u r n $ s u m = = = 0 ; } Not a problem: refactor and use the std library p r i v a t e f u n c t i o n s u m I s Z e r o ( a r r a y $ v a l u e s ) { r e t u r n a r r a y _ s u m ( $ v a l u e s ) = = = 0 ; }
  17. But how do we generate mutants?

  18. Come in di erent avors: Binary Arithmetic MUTATION OPERATOR Operator

    that changes (mutates) a piece of code Boolean Substitution Conditional Boundaries Return values And more...
  19. Original Mutated + - - + * / / *

    % * p u b l i c f u n c t i o n c a l c u l a t e M S I ( i n t $ t o t a l K i l l e d = 0 , i n t $ t o t a l M u t a t i o n s = 0 ) : f l o a t { - r e t u r n 1 0 0 * $ t o t a l K i l l e d / $ t o t a l M u t a t i o n s ; + r e t u r n 1 0 0 * $ t o t a l K i l l e d * $ t o t a l M u t a t i o n s ; } BINARY ARITHMETIC OPERATOR
  20. Come in di erent avors: Boolean Substitution MUTATION OPERATOR Operator

    that changes (mutates) a piece of code Binary Arithmetic Conditional Boundaries Return values And more...
  21. Original Mutated t r u e f a l s

    e f a l s e t r u e & & | | | | & & a n d o r o r a n d ! BOOLEAN SUBSTITUTION OPERATOR
  22. Come in di erent avors: Conditional Boundaries MUTATION OPERATOR Operator

    that changes (mutates) a piece of code Binary Arithmetic Boolean Substitution Return values And more...
  23. Original Mutated > > = < < = > =

    > < = < We've already seen this mutaiton in one of the previous examples CONDITIONAL BOUNDARY OPERATOR
  24. Come in di erent avors: Return values MUTATION OPERATOR Operator

    that changes (mutates) a piece of code Binary Arithmetic Boolean Substitution Conditional Boundaries And more...
  25. Original Mutated r e t u r n t r

    u e ; r e t u r n f a l s e ; r e t u r n 1 . 0 ; r e t u r n - ( + 1 ) ; r e t u r n $ t h i s ; r e t u r n n u l l ; r e t u r n ( s t m t ) ; ( s t m t ) ; r e t u r n n u l l ; r e t u r n 1 . 0 ; r e t u r n 0 . 0 ; Very useful due to php's dynamic types RETURN VALUES OPERATOR
  26. Come in di erent avors: And more... MUTATION OPERATOR Operator

    that changes (mutates) a piece of code Binary Arithmetic Boolean Substitution Conditional Boundaries Return values
  27. Negated Conditionals negates conditional, i.e. ! = = becomes =

    = = Increments interchanges + + and - - Literal Numbers changes literal int and oat values (useful for checking boundaries)
  28. Semantic Mutations Consider a "DateTime" mutator - $ d a

    t e = n e w D a t e T i m e ( $ _ G E T [ ' s i n c e ' ] ) ; + $ d a t e = D a t e T i m e : : c r e a t e F r o m F o r m a t ( D a t e T i m e : : I S O 8 6 0 1 , $ _ G E T [ ' s i n c e ' ] ) ; blog.blockscore.com/how-to-write-better-code-using-mutation-testing
  29. Semantic Mutations Or the " nd" mutator - $ u

    s e r = U s e r : : f i n d ( $ i d ) ; + $ u s e r = U s e r : : f i n d O r F a i l ( $ i d ) ; Take away: mutations do not have to be bad, sometimes you should replace the original source by the mutation
  30. Recap 1. Run the test suite, if something fails: stop

    and x the code 2. For each le, nd and create all possible mutations 3. For each mutant, run the test suite on the mutated code 4. Analyze the results
  31. Metrics Mutation Score Indicator (MSI): percentage of mutants covered &

    killed by tests Mutation Code Coverage: percentage of mutants covered by tests Covered Code MSI: percentage of killed mutants that were coverd by tests
  32. Metrics Example Tests: 361 Line Coverage: 64.86% 653 Mutants were

    generated 284 mutants were killed 218 mutants were not covered by tests 131 covered mutants were not detected 17 fatal errors were encountered 3 time outs were encountered
  33. Mutatio Scor Indicator msi = = = 0.47. Out of

    653 mutants, 284 mutants were killed. total mutants killed mutants 653 284
  34. Mutatio Cod Coverag mcc = = = 0.67. Out of

    653 mutants, 218 mutants were not covered. total mutants covered mutants 653 653−218
  35. Covere Cod MSI cc msi = = = 0.70. Out

    of 653 mutants, 435 mutants were covered of which 284 were killed. covered mutants killed mutants 435 284
  36. A summary A project with 361 tests and 65% code

    coverage 653 mutants were generated of which only 47% were killed and only 67% were covered. Mutation Score Indicator: 47% Mutation Code Coverage: 67% Covered Code MSI: 70%
  37. Implementing a mutation testing tool How do we implement an

    operator? How can we make it faster?
  38. A mutation operator can be applied by changing The lesystem

    The Abstract Syntax Tree (AST) The bytecode / opcode Perhaps PHP 7 can change this Implementing a mutation operator Humbug implements its mutations by changing les of the lesystem
  39. c l a s s M u l t i

    p l i c a t i o n e x t e n d s M u t a t o r A b s t r a c t { / * * * R e p l a c e ( * ) w i t h ( / ) * / p u b l i c s t a t i c f u n c t i o n g e t M u t a t i o n ( a r r a y & $ t o k e n s , $ i n d e x ) { $ t o k e n s [ $ i n d e x ] = ' / ' ; } p u b l i c s t a t i c f u n c t i o n m u t a t e s ( a r r a y & $ t o k e n s , $ i n d e x ) { $ t = $ t o k e n s [ $ i n d e x ] ; i f ( ! i s _ a r r a y ( $ t ) & & $ t = = ' * ' ) { r e t u r n t r u e ; } r e t u r n f a l s e ; } }
  40. Test Selection per mutation Running all tests for each mutation

    is ine cient Before running tests, gather coverage data from phpunit Next only run the tests that cover the mutated code Stop testing a mutation as soon as at least 1 test fails Sort the test on their execution time
  41. HUMBUG A Mutation Testing framework for PHP Measures the real

    e ectiveness of your test suites and assist in their improvement. It eats Code Coverage for breakfast
  42. Installation Globally using Git Globally using phar les Globally using

    composer Locally per project using composer
  43. Installing locally using composer c o m p o s

    e r r e q u i r e ' h u m b u g / h u m b u g = ~ 1 . 0 @ d e v ' v e n d o r / b i n / h u m b u g
  44. Con guration Create a con guration le using, h u

    m b u g c o n f i g u r e giving you a cli interface for con guring: What directories to include as mutation targets What directories to exclude Default timeout How to store the results
  45. Comand line options Timeout: h u m b u g

    - - t i m e o u t = 1 0 Restricting les: h u m b u g - - f i l e = P r i m e F a c t o r . p h p h u m b u g - - f i l e = * D r i v e r . p h p
  46. Comand line options Incremental analysis: h u m b u

    g - - i n c r e m e n t a l Cache results locally. Experimental: can have issues when using inheritance..
  47. Analyzing results Results can be stored in text format or

    json format
  48. Results in text format Contains a list of escaped mutants

    with their di s Also contains output send to stderr in case something went 2 ) \ H u m b u g \ M u t a t o r \ C o n d i t i o n a l B o u n d a r y \ G r e a t e r T h a n D i f f o n \ C a r b o n \ C a r b o n I n t e r v a l : : _ _ c o n s t r u c t ( ) i n / C a r b o n / s r c / C a r b o n / C a r b o n I n t e r v a l . p h p : - - - O r i g i n a l + + + N e w @ @ @ @ $ s p e c D a y s + = $ w e e k s > 0 ? $ w e e k s * C a r b o n : : D A Y S _ P E R _ W E E K : 0 ; - $ s p e c D a y s + = $ d a y s > 0 ? $ d a y s : 0 ; + $ s p e c D a y s + = $ d a y s > = 0 ? $ d a y s : 0 ; $ s p e c . = ( $ s p e c D a y s > 0 ) ? $ s p e c D a y s . s t a t i c : : P E R I O D _ D A Y S : ' ' ; i f ( $ h o u r s > 0 | | $ m i n u t e s > 0 | | $ s e c o n d s > 0 ) { $ s p e c . = s t a t i c : : P E R I O D _ T I M E _ P R E F I X ; $ s p e c . = $ h o u r s > 0 ? $ h o u r s . s t a t i c : : P E R I O D _ H O U R S : ' ' ;
  49. Results in json format Can potentially be used by CI

    services " s u m m a r y " : { " t o t a l " : 2 6 , " k i l l s " : 2 3 , " e s c a p e s " : 1 , " e r r o r s " : 0 , " t i m e o u t s " : 0 , " n o t e s t s " : 2 , " c o v e r e d _ s c o r e " : 9 6 , " c o m b i n e d _ s c o r e " : 8 8 , " m u t a t i o n _ c o v e r a g e " : 9 2 } , " e s c a p e d " : [ { " f i l e " : " s r c \ / B r o a d w a y D e m o \ / B a s k e t \ / B a s k e t . p h p " , " m u t a t o r " : " \ \ H u m b u g \ \ M u t a t o r \ \ C o n d i t i o n a l B o u n d a r y \ \ G r e a t e r T h a n " , " c l a s s " : " \ \ B r o a d w a y D e m o \ \ B a s k e t \ \ B a s k e t " , " m e t h o d " : " p r o d u c t I s I n B a s k e t " , " l i n e " : 1 0 1 , " d i f f " : " - - - O r i g i n a l \ n + + + N e w \ n @ @ @ @ \ n { \ n - r e t u r n i s s e t ( $ t h i s - > p r o d u c t C o u n t B y I d [ " t e s t s " : [ " B r o a d w a y D e m o \ \ B a s k e t \ \ A d d P r o d u c t T o B a s k e t T e s t : : i t _ a d d s _ a _ p r o d u c t _ t o _ a _ b a s k e t " , Contains all covered mutants: escaped, errored, timeout, killed
  50. Time for a demo

  51. How and when to use it

  52. None
  53. Use it while code reviewing

  54. Mutation Testing Better Code By Making Bugs

  55. Ruby: Mutant (https://github.com/mbj/mutant) Java: Pitest (http://pitest.org/) C#: Ninjaturtles (http://ninjaturtles.codeplex.com/) Python:

    Cosmic Ray (https://github.com/sixty-north/cosmic- ray) Matlab: MatMute (http://matmute.sourceforge.net/) Tools for other languages
  56. Concluding remarks Mutation testing will improve the quality of your

    tests Is becoming more mainstream over the last years Write small (fast) tests
  57. Maintainer of Mutant, the ruby mutation testing tool Gave me

    very useful feedback and insights Thank you Markus Schrip
  58. I have not failed. I've just found 10,000 ways that

    won't work - Thomas Edison
  59. Thanks for listening Find these slides at mutation.markredeman.nl