Mutation Testint in PHP with Humbug

Mutation Testint in PHP with Humbug

Mutation testing is a technique that measures the quality of a test suite. This is done by making small changes (mutations) to our code, which will break our application, and ensuring that there is at least one test that no longer passes.

This talk introduces the concept of mutation testing.We show that by using mutation testing we can find bugs in code which has a 100% code coverage. We will look at a few open source projects and use the padraic/humbug package to analyze their Mutation Score Indicator. Lastly we look at some of the disadvantages of mutation testing, namely performance and false positives.

These slides can also be found at:
http://mutation.markredeman.nl

Fdddf387d081e976aab7bda0b64d535f?s=128

Mark Redeman

October 01, 2015
Tweet

Transcript

  1. MUTATION TESTING IN PHP WITH HUMBUG

  2. @MARKREDEMAN STUDENT APPLIED MATHEMATICS FREELANCE (WEB) DEVELOPER

  3. Introduction to mutation testing Humbug, a mutation testing framework for

    PHP. Analyzing code coverage of open source projects Improving your workflow with mutation testing What's this talk about?
  4. A tool to analyze stability of a piece of code

    Similar to code coverage, but better Can find missing tests WHAT IS MUTATION TESTING?
  5. A quick example c l a s s C u

    s t o m e r { p r i v a t e $ o r d e r s = 0 ; p u b l i c f u n c t i o n _ _ c o n s t r u c t ( $ o r d e r s ) { $ t h i s ­ > o r d e r s = $ o r d e r s ; } p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s ­ > o r d e r s > 1 0 ; } } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s ­ > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; / / 1 0 0 % l i n e c o v e r a g e : D $ t h i s ­ > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; } p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s ­ > o r d e r s > = 1 0 ; }
  6. c l a s s C u s t o

    m e r { p r i v a t e $ o r d e r s = 0 ; p u b l i c f u n c t i o n _ _ c o n s t r u c t ( $ o r d e r s ) { $ t h i s ­ > o r d e r s = $ o r d e r s ; } p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s ­ > o r d e r s > 1 0 ; } } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s ­ > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; / / 1 0 0 % l i n e c o v e r a g e : D $ t h i s ­ > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 1 0 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; $ t h i s ­ > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; }
  7. Generated mutants are similar to real faults - Andrew, Briand,

    Labiche, ICSE 2005
  8. Some Definitions

  9. a piece of code that has been changed (mutated) by

    a mutator killed if at least 1 test fails escaped if at all test pass equivalent if there does not exist a test case which can distinguish the mutant from the original code uncovered mutant is not covered by a test fatal mutant produces a fatal error timout unit tests exceed allowed timeout Mutation
  10. Killed Mutant c l a s s C u s

    t o m e r { p r i v a t e $ o r d e r s = 0 ; p u b l i c f u n c t i o n _ _ c o n s t r u c t ( $ o r d e r s ) { $ t h i s ­ > o r d e r s = $ o r d e r s ; } p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s ­ > o r d e r s < 1 0 ; / / m u t a t e d " > " t o " < " } } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s ­ > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; $ t h i s ­ > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; }
  11. Escaped Mutant c l a s s C u s

    t o m e r { p r i v a t e $ o r d e r s = 0 ; p u b l i c f u n c t i o n _ _ c o n s t r u c t ( $ o r d e r s ) { $ t h i s ­ > o r d e r s = $ o r d e r s ; } p u b l i c f u n c t i o n i s G o l d C u s t o m e r ( ) { r e t u r n $ t h i s ­ > o r d e r s > = 1 0 ; / / m u t a t e d " > " t o " > = " } } f u n c t i o n t e s t I s G o l d C u s t o m e r ( ) { $ t h i s ­ > a s s e r t F a l s e ( ( n e w C u s t o m e r ( 0 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; $ t h i s ­ > a s s e r t T r u e ( ( n e w C u s t o m e r ( 1 1 ) ) ­ > i s G o l d C u s t o m e r ( ) ) ; }
  12. Equivalent Mutant f u n c t i o n

    s u m _ i s _ z e r o ( a r r a y $ v a l u e s ) { $ s u m = 0 ; f o r e a c h ( $ v a l u e s a s $ v a l u e ) { $ s u m + = $ v a l u e ; } r e t u r n $ s u m = = = 0 ; } f u n c t i o n s u m _ i s _ z e r o ( a r r a y $ v a l u e s ) { $ s u m = 0 ; f o r e a c h ( $ v a l u e s a s $ v a l u e ) { $ s u m ­ = $ v a l u e ; / / m u t a t e d " + " t o " ­ " } r e t u r n $ s u m = = = 0 ; }
  13. Increments Mutator Decrements Mutator Invert Negatives Mutator Return Values Mutator

    Math Mutator Negate Mutator Conditionals Boundary Mutator Remove Conditionals Values Mutator MUTATOR Operator that changes (mutates) a piece of code
  14. Original Mutated + ­ ­ + * / / *

    % * MATH MUTATORS
  15. Original Mutated = = ! = ! = = =

    < = > > = < < > = > < = CONDITIONAL MUTATORS
  16. :: Mutator description Negate Conditionals replace condition by ! (

    c o n d i t i o n ) Remove Conditionals substitute conditional with t r u e or f a l s e Increments crement numeric values by 1 Decrements crement numeric values by 1 Invert Negatives multiply numeric values by ­ 1 Return Values remove r e t u r n statements
  17. Metrics Mutation Score Indicator (MSI): percentage of mutants covered &

    killed by tests Mutation Code Coverage: percentage of mutants covered by tests Covered Code MSI: percentage of killed mutants that were coverd by tests $ v a n q u i s h e d T o t a l = $ k i l l e d C o u n t + $ t i m e o u t C o u n t + $ e r r o r C o u n t ; $ m e a s u r a b l e T o t a l = $ t o t a l C o u n t ­ $ u n c o v e r e d C o u n t ; / / = $ v a n q u i s h e d T o t a l + $ e s c a p e d C o u n t $ m s i = r o u n d ( 1 0 0 * ( $ v a n q u i s h e d T o t a l / $ t o t a l C o u n t ) ) ; $ c o v e r e d R a t e = r o u n d ( 1 0 0 * ( $ m e a s u r a b l e T o t a l / $ t o t a l C o u n t ) ) ; $ c c _ m s i = r o u n d ( 1 0 0 * ( $ v a n q u i s h e d T o t a l / $ m e a s u r a b l e T o t a l ) ) ;
  18. Metrics Example Tests: 361 Line Coverage: 64.86% Mutation Score Indicator

    (MSI): 47% Mutation Code Coverage: 67% Covered Code MSI 70% 653 Mutants were generated 284 mutants were killed 218 mutants were not covered by tests 131 covered mutants were not detected 17 fatal errors were encountered 3 time outs were encountered 47% of all mutations were detected versus 65% line coverage.
  19. HUMBUG A Mutation Testing framework for PHP Measures the real

    effectiveness of your test suites and assist in their improvement. It eats Code Coverage for breakfast
  20. Git: g i t c l o n e h

    t t p s : / / g i t h u b . c o m / p a d r a i c / h u m b u g . g i t c d h u m b u g / p a t h / t o / c o m p o s e r . p h a r i n s t a l l b i n / h u m b u g Phar: w g e t h t t p s : / / p a d r a i c . g i t h u b . i o / h u m b u g / d o w n l o a d s / h u m b u g . p h a r w g e t h t t p s : / / p a d r a i c . g i t h u b . i o / h u m b u g / d o w n l o a d s / h u m b u g . p h a r . p u b k e y c h m o d + x h u m b u g . p h a r Composer: c o m p o s e r g l o b a l r e q u i r e ' h u m b u g / h u m b u g = ~ 1 . 0 @ d e v ' # A n d a d d ~ / . c o m p o s e r / v e n d o r / b i n t o y o u r p a t h t o m a k e i t e a s i l y e x e c u t a b l e e x p o r t P A T H = ~ / . c o m p o s e r / v e n d o r / b i n : $ P A T H Installation
  21. Configuration $ > g i t c l o n

    e g i t @ g i t h u b . c o m : y o u r n a m e / y o u r p r o j e c t . g i t # C h e c k i f a l l t e s t s a r e g r e e n $ > p h p u n i t $ > h u m b u g c o n f i g u r e _ _ _ | | | | _ _ _ _ _ | | _ _ _ _ _ _ _ | _ _ | | | | ' \ | ' _ \ | | / _ ` | | _ | | _ | \ _ , _ | _ | _ | _ | _ . _ _ / \ _ , _ \ _ _ , | | _ _ _ / H u m b u g v e r s i o n 1 . 0 ­ d e v H u m b u g c o n f i g u r a t i o n t o o l . I t w i l l g u i d e y o u t h r o u g h H u m b u g c o n f i g u r a t i o n i n f e w s e c o n d s . W h e n c h o o s i n g d i r e c t o r i e s , y o u m a y e n t e r e a c h d i r e c t o r y a n d p r e s s r e t u r n . T o e x i t d i r e c t o r y s e l e c t i o n , p l e a s e l e a v e t h e n e x t a n s w e r b l a n k a n d p r e s s r e t u r n . W h a t s o u r c e d i r e c t o r i e s d o y o u w a n t t o i n c l u d e ? : s r c W h a t s o u r c e d i r e c t o r i e s d o y o u w a n t t o i n c l u d e ? : A n y d i r e c t o r i e s t o e x c l u d e f r o m w i t h i n y o u r s o u r c e d i r e c t o r i e s ? : S i n g l e t e s t s u i t e t i m e o u t i n s e c o n d s [ 1 0 ] : W h e r e d o y o u w a n t t o s t o r e t h e t e x t l o g ? [ h u m b u g l o g . t x t ] : W h e r e d o y o u w a n t t o s t o r e t h e j s o n l o g ( i f y o u n e e d i t ) ? : G e n e r a t e " h u m b u g . j s o n . d i s t " ? [ Y ] : C o n f i g u r a t i o n f i l e " h u m b u g . j s o n . d i s t " w a s c r e a t e d .
  22. Running Humbug $ > h u m b u g

    _ _ _ | | | | _ _ _ _ _ | | _ _ _ _ _ _ _ | _ _ | | | | ' \ | ' _ \ | | / _ ` | | _ | | _ | \ _ , _ | _ | _ | _ | _ . _ _ / \ _ , _ \ _ _ , | | _ _ _ / H u m b u g v e r s i o n 1 . 0 ­ d e v H u m b u g r u n n i n g t e s t s u i t e t o g e n e r a t e l o g s a n d c o d e c o v e r a g e d a t a . . . 9 7 [ = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = ] 2 s e c s H u m b u g h a s c o m p l e t e d t h e i n i t i a l t e s t r u n s u c c e s s f u l l y . T e s t s : 9 7 L i n e C o v e r a g e : 9 8 . 2 7 % H u m b u g i s a n a l y s i n g s o u r c e f i l e s . . . M u t a t i o n T e s t i n g i s c o m m e n c i n g o n 1 9 f i l e s . . . ( . : k i l l e d , M : e s c a p e d , S : u n c o v e r e d , E : f a t a l e r r o r , T : t i m e d o u t ) E . . . . . . M . M . . . . . . . M M T T . . . . . M M S . . . . . . . . . . . . . . . . . . . . . . . . . . M . M . . | 6 0 ( 9 / 1 9 ) M . . . . . . . . . . . . . . . . . . M . . . . E . . . . . . . M M M M S 9 7 m u t a t i o n s w e r e g e n e r a t e d : 7 7 m u t a n t s w e r e k i l l e d 2 m u t a n t s w e r e n o t c o v e r e d b y t e s t s 1 4 c o v e r e d m u t a n t s w e r e n o t d e t e c t e d 2 f a t a l e r r o r s w e r e e n c o u n t e r e d 2 t i m e o u t s w e r e e n c o u n t e r e d M e t r i c s : M u t a t i o n S c o r e I n d i c a t o r ( M S I ) : 8 4 % M u t a t i o n C o d e C o v e r a g e : 9 8 % C o v e r e d C o d e M S I : 8 5 % R e m e m b e r t h a t s o m e m u t a n t s w i l l i n e v i t a b l y b e h a r m l e s s ( i . e . f a l s e p o s i t i v e s ) . T i m e : 3 1 . 3 9 s e c o n d s M e m o r y : 8 . 7 5 M B H u m b u g r e s u l t s a r e b e i n g l o g g e d a s T E X T t o : h u m b u g l o g . t x t
  23. Analyzing Humbug humbug.log.txt 2 ) \ H u m b

    u g \ M u t a t o r \ C o n d i t i o n a l B o u n d a r y \ G r e a t e r T h a n D i f f o n \ C a r b o n \ C a r b o n I n t e r v a l : : _ _ c o n s t r u c t ( ) i n / C a r b o n / s r c / C a r b o n / C a r b o n I n t e r v a l . p h p : ­ ­ ­ O r i g i n a l + + + N e w @ @ @ @ $ s p e c D a y s + = $ w e e k s > 0 ? $ w e e k s * C a r b o n : : D A Y S _ P E R _ W E E K : 0 ; ­ $ s p e c D a y s + = $ d a y s > 0 ? $ d a y s : 0 ; + $ s p e c D a y s + = $ d a y s > = 0 ? $ d a y s : 0 ; $ s p e c . = ( $ s p e c D a y s > 0 ) ? $ s p e c D a y s . s t a t i c : : P E R I O D _ D A Y S : ' ' ; i f ( $ h o u r s > 0 | | $ m i n u t e s > 0 | | $ s e c o n d s > 0 ) { $ s p e c . = s t a t i c : : P E R I O D _ T I M E _ P R E F I X ; $ s p e c . = $ h o u r s > 0 ? $ h o u r s . s t a t i c : : P E R I O D _ H O U R S : ' ' ;
  24. humbug.log.json " s u m m a r y "

    : { " t o t a l " : 2 6 , " k i l l s " : 2 3 , " e s c a p e s " : 1 , " e r r o r s " : 0 , " t i m e o u t s " : 0 , " n o t e s t s " : 2 , " c o v e r e d _ s c o r e " : 9 6 , " c o m b i n e d _ s c o r e " : 8 8 , " m u t a t i o n _ c o v e r a g e " : 9 2 } , " e s c a p e d " : [ { " f i l e " : " s r c \ / B r o a d w a y D e m o \ / B a s k e t \ / B a s k e t . p h p " , " m u t a t o r " : " \ \ H u m b u g \ \ M u t a t o r \ \ C o n d i t i o n a l B o u n d a r y \ \ G r e a t e r T h a n " , " c l a s s " : " \ \ B r o a d w a y D e m o \ \ B a s k e t \ \ B a s k e t " , " m e t h o d " : " p r o d u c t I s I n B a s k e t " , " l i n e " : 1 0 1 , " d i f f " : " ­ ­ ­ O r i g i n a l \ n + + + N e w \ n @ @ @ @ \ n { \ n ­ r e t u r n i s s e t ( $ t h i s ­ > p r o d u c t C o u n t B y I d [ " t e s t s " : [ " B r o a d w a y D e m o \ \ B a s k e t \ \ A d d P r o d u c t T o B a s k e t T e s t : : i t _ a d d s _ a _ p r o d u c t _ t o _ a _ b a s k e t " , " B r o a d w a y D e m o \ \ B a s k e t \ \ A d d P r o d u c t T o B a s k e t T e s t : : m u l t i p l e _ p r o d u c t s _ c a n _ b e _ a d d e d _ t o _ a _ b a s k e t " " B r o a d w a y D e m o \ \ B a s k e t \ \ A d d P r o d u c t T o B a s k e t T e s t : : a _ p r o d u c t _ c a n _ b e _ a d d e d _ t o _ a _ b a s k e t _ m u l t i p l e _ t i m e s " " B r o a d w a y D e m o \ \ B a s k e t \ \ C h e c k o u t T e s t : : i t _ c h e c k s _ o u t _ a _ b a s k e t " , " B r o a d w a y D e m o \ \ B a s k e t \ \ C h e c k o u t T e s t : : i t _ c a n n o t _ c h e c k o u t _ a _ b a s k e t _ t h a t _ h a s _ b e e n _ e m p t i e d " " B r o a d w a y D e m o \ \ B a s k e t \ \ C h e c k o u t T e s t : : n o t h i n g _ h a p p e n s _ w h e n _ c h e c k i n g _ o u t _ a _ b a s k e t _ f o r _ a _ s e c o n d _ t i m e " " B r o a d w a y D e m o \ \ B a s k e t \ \ R e m o v e P r o d u c t F r o m B a s k e t T e s t : : i t _ r e m o v e s _ a _ p r o d u c t _ t h a t _ w a s _ a d d e d " " B r o a d w a y D e m o \ \ B a s k e t \ \ R e m o v e P r o d u c t F r o m B a s k e t T e s t : : i t _ d o e s _ n o t h i n g _ w h e n _ r e m o v i n g _ a _ p r o d u c t _ t h a t _ i s _ n o t _ i n _ a _ b a s k e t " " B r o a d w a y D e m o \ \ B a s k e t \ \ R e m o v e P r o d u c t F r o m B a s k e t T e s t : : i t _ o n l y _ r e m o v e s _ o n e _ i n s t a n c e _ o f _ a _ p r o d u c t " ] , " s t d e r r " : " " , " s t d o u t " : " T A P v e r s i o n 1 3 " } ] ,
  25. Options Timeout: h u m b u g ­ ­

    t i m e o u t = 1 0 Restricting files: h u m b u g ­ ­ f i l e = P r i m e F a c t o r . p h p h u m b u g ­ ­ f i l e = * D r i v e r . p h p Incremental analysis: h u m b u g ­ ­ i n c r e m e n t a l Can off significant performance boosts by caching previous results in / h o m e / p a d r a i c / . h u m b u g . .
  26. Mutators Binary Arithmetic Boolean Substitution Conditional Boundaries Negated Conditionals Increments

    Return Values Literal Numbers If Statements
  27. HUMBUG TEST RESULTS Package LC MSI MCC CCM Execution time

    carbon/carbon 61% 95% 100% 95% 2.25m symfony/event- dispatcher 85% 54% 69% 78% 20s hylianshield/validator 100% 75% 89% 85% 1.35m mathiasverraes/money 96% 92% 100% 92% 16.9s thephpleague/fractal 98% 84% 98% 85% 31s broadway/broadway- demo 96% 88% 92% 86% 5s broadway/broadway 57% 49% 56% 87% 46s Line Coverage (LC), Mutation Score Indicator (MSI), Mutation Code Coverage (MCC), Covered Code MSI (CCM) Timout option set to 2 seconds.
  28. TDD workflow with mutation testing

  29. Read the book

  30. Other tools Ruby: Mutant Java: Pitest C#: Ninjaturtles Python: MutPy

    Matlab: MatMute
  31. Concluding remarks Mutation testing will improve the quality of your

    tests Is becoming more mainstream over the last years Write small (fast) tests
  32. I have not failed. I've just found 10,000 ways that

    won't work - Thomas Edison
  33. Find these slides at mutation.markredeman.nl