Can You Trust Your Tests?

Talk on coverage and mutation based testing

Transcript

1. Can you trust your tests? 2015 Tadas Ščerbinskas & Vaidas Pilkauskas
2. Tadas Ščerbinskas @tadassce Vaidas Pilkauskas @liucijus
3. Agenda Test code quality Mutation testing Mutant - mutation testing for Ruby
4. Prod vs. Test code quality Code has bugs. Tests are code. Tests have bugs.
5. Prod vs. Test code quality Code has bugs. Tests are code. Tests have bugs.
7. Test Quality Readable Focused Concise Well named
8. "Program testing can be used to show the presence of bugs, but never to show their absence!" - Edsger W. Dijkstra
9. Code coverage
10. Types of code coverage Line Branch Condition Path Data
11. Coverage tool for Ruby simplecov g e m i n s t a l l s i m p l e c o v r e q u i r e ' s i m p l e c o v ' S i m p l e C o v . s t a r t
12. Line d e f f o o ( a r g = t r u e ) a r g ? " a " : " b " e n d e x p e c t ( f o o ) . t o e q " a " 100% line coverage. No test for "b".
13. Branch d e f f o o ( a r g ) a r g ? " a " : " b " e n d e x p e c t ( f o o ( t r u e ) ) . t o e q " a " e x p e c t ( f o o ( f a l s e ) ) . t o e q " b " 100% branch coverage
14. Demo
15. Can you trust 100% coverage? Code coverage can only show what is not tested 100% code coverage for interpreted languages is kind of like full compilation
16. SUCCESS: 26/26 (100%) Tests passed
17. Mutation testing
18. Mutation testing Changes your program code and expects your tests to fail.
19. Terminology Applying a mutation to some code creates a mutant. If test passes - mutant has survived. If test fails - mutant is killed.
20. Failing is the new passing
21. Tests’ effectiveness is measured by number of killed mutants by your test suite.
22. It’s like hiring a white-hat hacker to try to break into your server and making sure you detect it.
23. What if mutant survives Simplify your code Add additional tests TDD - minimal amount of code to pass the test
24. Hunting tips # A v o i d l i t e r a l s p o s t s [ 0 ] = > p o s t s . f i r s t # D o n ’ t o v e r u s e s y n t a c t i c c o n s t r u c t s : : U s e r = > U s e r # D o n ’ t p a s s l i t e r a l d e f a u l t s n u m . t o _ s ( 1 0 ) = > n u m . t o _ s
25. Challenges High computation cost - slow Equivalent mutants - false negatives Infinite loops
26. Equivalent mutations # O r i g i n a l i = 0 w h i l e i ! = 1 0 d o _ s o m e t h i n g i + = 1 e n d # M u t a n t i = 0 w h i l e i < 1 0 d o _ s o m e t h i n g i + = 1 e n d
27. Infinite Runtime # O r i g i n a l w h i l e e x p r e s s i o n d o _ s o m e t h i n g e n d # M u t a t i o n w h i l e t r u e d o _ s o m e t h i n g e n d
28. Use timeouts c o n f i g . a r o u n d ( : e a c h ) d o | e x a m p l e | T i m e o u t . t i m e o u t ( 2 ) d o e x a m p l e . r u n e n d e n d
29. Mutant Active project Good reporting Only works with rspec
30. How Can’t just do S t r i n g # g s u b
31. How Mutant uses a pure Ruby parser and an unparser to do its magic. AST: p P a r s e r : : C u r r e n t R u b y . p a r s e ( " 2 + 2 " ) # ( s e n d # ( i n t 2 ) : + # ( i n t 2 ) )
32. Mutations Literal / primitive and compound Statement deletion Conditional Binary connective replacement Argument deletion / rename / swap Unary operator exchange Bitwise
33. Test-Selection “Longest rspec example group descriptions’ prefix match”
34. Test-Selection $ m u t a n t - - i n c l u d e l i b - - r e q u i r e b a r - - u s e r s p e c F o o : : B a r # b a 1. Foo::Bar#baz 2. Foo::Bar 3. Foo
35. Demo
36. Disadvantages Slow - cannot be part of TDD rhythm May be very noisy
37. Summary Normal code coverage highlights code that is definitely not tested Mutation testing highlights code that definitely is tested
38. Thanks! Q & A
40. Pictures s05 - s08 - s15 - s16 - s21 - http://crazy-monkeeey.deviantart.com/art/Who-watches-the-Watchmen- 344523349 http://www.chimpsanctuarynw.org/blog/wp-content/uploads/2009/03/negra- covered-in-pink-blanket-front-room_web_mg_8161.jpg https://twitter.com/bloerwald/status/448415935926255618 http://www.smosh.com/smosh-pit/photos/12-more-bizarre-mutant-animals https://tbgsecurity.com/the-history-of-hacking-timeline-of-hacking- techniques-infographic/