Mutation Testing - Pixels Camp 2019

Mutation testing has been evolving into a real candidate to become the de facto metric for assessing the quality of a test suite, defying the throne that has been occupied by code coverage until now.

The concept of mutation testing is quite simple: mutations are introduced on the codebase and the test suite is run against the mutated code. If your test suite is strong, then it should catch the mutation, by having at least one test failing.

In this talk I plan to show how coverage by itself doesn't guarantee quality on the test suite, and how mutation testing can effectively find bugs on your code and tests.

Transcript

1. 1 An intro to Mutation Testing - or why coverage sucks Pedro Rijo Pixels Camp March 22nd, 2019
2. Self-introduction 1 Motivation 2 What is mutation testing 4 PIT - Java mutation testing framework 5 AGENDA 2 MT at Feedzai 6 Code Coverage 3 Final Remarks 7
3. Who? 3
4. 4 @pedrorijo91 https://pedrorijo.com/ Background
5. 5 2009 - 2014 2014 - 2015 2015 - 2017 Background Since 2017
6. 6 • We catch the bad guys on e-commerce fraud • State of the art AI and Machine Learning About Feedzai • Real time processing • Used by biggest banks, payment processors, and retailers across the globe
7. 7 Feedzai Customers
8. 8 About Feedzai
9. 9 About Feedzai
10. 10 About Feedzai
11. Ensuring high quality 11
12. • Bugs may cause • (Continuous) automated testing is fundamental 12 High Quality Code
13. • All our code has Unit Tests (BE and FE) • Many Integration tests (selenium) • Many System Tests (docker) • Failure Model (http://bit.ly/feedzai-failure-model) 13 High Quality Code
14. • Most run before merging code • Few longer builds are run periodically 14 High Quality Code
15. 15 High Quality Code
16. 16 • How do I safely refactor my tests? • How do I know if I can trust a test suite I inherited? • How do I ensure my team is writing effective tests? • How do I know I have enough tests to refactor my code safely? High Quality Code
17. 17 Not even TDD can answer these questions High Quality Code
18. Code Coverage 18
19. 19 Code Coverage is a measurement of the percentage of code lines executed during the test suite.
20. 20 • Line coverage • Statement coverage • Branch coverage • Method coverage Code Coverage • Data coverage • Path coverage • Modified condition coverage • etc https://en.wikipedia.org/wiki/Code_coverage#Coverage_criteria
21. 21 Code Coverage
22. 22 100% CC but what if X = 10? Code Coverage
23. 23 100% CC but no asserts Code Coverage
24. What is Mutation Testing (MT)? 24
25. 25 Mutation Testing
26. 26 • MT is a way to check the strength of our tests • CC is not reliable as we have seen • 1971 by Richard Lipton • Basic idea: introduce small bugs and check if test suite finds those bugs ◦ Netflix Chaos Monkey anyone? Mutation Testing
27. 27 public boolean isOldEnough(int age) { return age >= 18; } public boolean isOldEnough(int age) { return age > 18; } Mutation Testing
28. 28 • MT simulates easy bugs - competent programmer hypothesis According to this hypothesis, programmers write programs that are almost perfect. The competent programmer hypothesis says that program faults are syntactically small and can be corrected with a few keystrokes. • MT should be included in the development lifecycle (should be run before commit/push/PR) Mutation Testing
29. Basic MT concepts 29
30. 30 • Mutation Operators/mutators • Mutants • Killed mutation • Survived mutation • Equivalent Mutations Mutation Testing - Basic Concepts
31. 31 Mutation Testing - Equivalent Mutations
32. 32 Mutation Testing - Equivalent Mutations
33. 33 • Mutations in dead/useless code • Mutations that affect only performance • Mutations that can’t be triggered due to logic elsewhere in the program • Mutations that alter only internal state Mutation Testing - Equivalent Mutations
34. 34 Mutation Testing - Examples
35. 35 Mutation Testing - Examples
36. 36 Mutation Testing - Examples
37. 37 Mutation Testing - Examples
38. PIT - Java Mutation Testing Framework 38
39. 39 PIT - Java Mutation Testing Framework
40. 40 • Easy to use • Maven support: http://pitest.org/quickstart/maven/ • Very efficient • multi module support: https://github.com/STAMP-project/pitmp-maven-plugin PIT - Java Mutation Testing Framework
41. 41 PIT - Java Mutation Testing Framework
42. 42 PIT - Java Mutation Testing Framework $ mvn org.pitest:pitest-maven:mutationCoverage $ mvn eu.stamp-project:pitmp-maven-plugin:run
43. 43 • Report under <project>/target/pit-reports/YYYYMMDDHHmm PIT - Java Mutation Testing Framework
45. 45 • MT score threshold to fail builds • skip methods (toString, equals, hashcode) • skip full classes • skip test classes PIT - Java Mutation Testing Framework
46. Adding MT to Feedzai codebase 46
47. 47 • PoC using smaller repos ◦ OpenML ▪ https://github.com/feedzai/feedzai-openml/pull/33 ▪ https://github.com/feedzai/feedzai-openml/pull/34 • internal libs Using MT at Feedzai
48. 48 Using MT at Feedzai
49. 49 • Redundant logic in code • Missing test cases ◦ boundary values ◦ null checks ◦ etc Using MT at Feedzai
50. 50 Using MT at Feedzai
51. 51 Using MT at Feedzai
52. 52 Using MT at Feedzai
53. 53 Using MT at Feedzai
54. 54 Can we add MT everywhere? Using MT at Feedzai
55. 55 • Common complaints with MT ◦ Equivalent mutations ◦ MT is slow ▪ compile code and run tests ▪ PIT is very efficient (bytecode manipulation and coverage info) • Doesn’t work with mocks ◦ PIT supports JMock, EasyMock, Mockito, PowerMock and JMockit Using MT at Feedzai
56. 56 • Replace full method logic by “nullable” statement • Much less mutants • good preliminary analysis • https://github.com/STAMP-project/pitest-descartes Extreme Mutation
57. 57 Extreme Mutation
58. 58 Conclusion • Keeping code base bug-free is hard • We need to know our tests strength • Code coverage has flaws • Mutation Testing is a better metric • MT is computational heavy... • ...but seems the right path :)
59. 59 • https://pedrorijo.com/blog/intro-mutation/ • https://github.com/mbj/mutant for Ruby • http://stryker-mutator.io/ for Javascript, Scala, and C# • https://github.com/sugakandrey/scalamu for Scala Resources
60. 60 @pedrorijo91 https://pedrorijo.com/ Questions?