Mutation Testing - Quality Talks, May 2019

Mutation testing has been evolving into a real candidate to become the de facto metric for assessing the quality of a test suite, defying the throne that has been occupied by code coverage until now.

The concept of mutation testing is quite simple: mutations are introduced on the codebase and the test suite is run against the mutated code. If your test suite is strong, then it should catch the mutation, by having at least one test failing.

In this talk I plan to show how coverage by itself doesn't guarantee quality on the test suite, and how mutation testing can effectively find bugs on your code and tests.

Transcript

1. 1 An intro to Mutation Testing - or why coverage sucks Pedro Rijo Quality Talks May 23rd, 2019
2. Who? 2
3. 3 @pedrorijo91 https://pedrorijo.com/ Background
4. 4 2009 - 2014 2014 - 2015 2015 - 2017 Background Since 2017
5. 5 • We catch the bad guys on e-commerce fraud • State of the art AI and Machine Learning About Feedzai • Real time processing • Used by biggest banks, payment processors, and retailers across the globe
6. 6 Feedzai Customers
7. 7 About Feedzai
8. Ensuring high quality 8
9. • Bugs may cause • (Continuous) automated testing is fundamental 9 High Quality Code
10. 10 About Feedzai
11. • All our code has Unit Tests (BE and FE) • Many Integration tests (selenium) • Many System Tests (docker) • Failure Model (http://bit.ly/feedzai-failure-model) 11 High Quality Code
12. 12 High Quality Code
13. Code Coverage 13
14. 14 Code Coverage is a measurement of the percentage of code lines executed during the test suite.
15. 15 • Line coverage • Statement coverage • Branch coverage • Method coverage Code Coverage • Data coverage • Path coverage • Modified condition coverage • etc https://en.wikipedia.org/wiki/Code_coverage#Coverage_criteria
16. 16 Code Coverage
17. 17 100% CC but what if X = 10? Code Coverage
18. 18 100% CC but no asserts Code Coverage
19. What is Mutation Testing (MT)? 19
20. 20 • MT is a way to check the strength of our tests • CC is not reliable as we have seen • 1971 by Richard Lipton • Basic idea: introduce small bugs and check if test suite finds those bugs ◦ Netflix Chaos Monkey anyone? Mutation Testing
21. 21 public boolean isOldEnough(int age) { return age >= 18; } public boolean isOldEnough(int age) { return age > 18; } Mutation Testing
22. 22 • MT simulates easy bugs - competent programmer hypothesis According to this hypothesis, programmers write programs that are almost perfect. The competent programmer hypothesis says that program faults are syntactically small and can be corrected with a few keystrokes. • MT should be included in the development lifecycle (should be run before commit/push/PR) Mutation Testing
23. Basic MT concepts 23
24. 24 • Mutation Operators/mutators • Mutants • Killed mutation • Survived mutation • Equivalent Mutations Mutation Testing - Basic Concepts
25. 25 Mutation Testing - Equivalent Mutations
26. 26 Mutation Testing - Equivalent Mutations
27. 27 • Mutations in dead/useless code • Mutations that affect only performance • Mutations that can’t be triggered due to logic elsewhere in the program • Mutations that alter only internal state Mutation Testing - Equivalent Mutations
28. 28 Mutation Testing - Examples
29. 29 Mutation Testing - Examples
30. 30 Mutation Testing - Examples
31. 31 Mutation Testing - Examples
32. PIT - Java Mutation Testing Framework 32
33. 33 PIT - Java Mutation Testing Framework
34. 34 • Easy to use • Maven support: http://pitest.org/quickstart/maven/ • Very efficient • multi module support: https://github.com/STAMP-project/pitmp-maven-plugin • Generates mutants and runs test suite automatically PIT - Java Mutation Testing Framework
35. 35 PIT - Java Mutation Testing Framework
36. 36 PIT - Java Mutation Testing Framework $ mvn org.pitest:pitest-maven:mutationCoverage $ mvn eu.stamp-project:pitmp-maven-plugin:run
37. 37 • Report under <project>/target/pit-reports/YYYYMMDDHHmm PIT - Java Mutation Testing Framework
39. 39 • MT score threshold to fail builds • skip methods (toString, equals, hashcode) • skip full classes • skip test classes PIT - Java Mutation Testing Framework
40. Adding MT to Feedzai codebase 40
41. 41 • PoC using smaller repos ◦ OpenML ▪ https://github.com/feedzai/feedzai-openml/pull/33 ▪ https://github.com/feedzai/feedzai-openml/pull/34 • internal libs Using MT at Feedzai
42. 42 Using MT at Feedzai
43. 43 • Redundant logic in code • Missing test cases ◦ boundary values ◦ null checks ◦ etc Using MT at Feedzai
44. 44 Using MT at Feedzai
45. 45 Using MT at Feedzai
46. 46 Using MT at Feedzai
47. 47 Using MT at Feedzai
48. 48 Can we add MT everywhere? Using MT at Feedzai
49. 49 • Common complaints with MT ◦ Equivalent mutations ◦ MT is slow ▪ compile code and run tests ▪ PIT is very efficient (bytecode manipulation and coverage info) ▪ incremental analysis • Doesn’t work with mocks ◦ PIT supports JMock, EasyMock, Mockito, PowerMock and JMockit Using MT at Feedzai
50. 50 • Replace full method logic by “nullable” statement • Much less mutants • good preliminary analysis • https://github.com/STAMP-project/pitest-descartes Extreme Mutation
51. 51 Extreme Mutation
52. 52 Conclusion • Keeping code base bug-free is hard • We need to know our tests strength • Code coverage has flaws • Mutation Testing is a better metric • MT is computational heavy... • ...but seems the right path :)
53. 53 • https://pedrorijo.com/blog/intro-mutation/ • https://github.com/mbj/mutant for Ruby • http://stryker-mutator.io/ for Javascript, Scala, and C# • https://github.com/sugakandrey/scalamu for Scala Resources
54. 54 WE ARE HIRING ! http://bit.ly/feedzai-tech-blog https://careers.feedzai.com/
55. 55 @pedrorijo91 https://pedrorijo.com/ Questions?