Slide 1
Slide 1 text
1
An intro to Mutation
Testing - or why
coverage sucks
Pedro Rijo
Pixels Camp
March 22nd, 2019
Slide 2
Slide 2 text
Self-introduction
1
Motivation
2
What is mutation testing
4
PIT - Java mutation testing framework
5
AGENDA
2
MT at Feedzai
6
Code Coverage
3
Final Remarks
7
Slide 3
Slide 3 text
Who?
3
Slide 4
Slide 4 text
4
@pedrorijo91
https://pedrorijo.com/
Background
Slide 5
Slide 5 text
5
2009 - 2014
2014 - 2015
2015 - 2017
Background
Since 2017
Slide 6
Slide 6 text
6
● We catch the bad guys on
e-commerce fraud
● State of the art AI and Machine
Learning
About Feedzai
● Real time processing
● Used by biggest banks, payment
processors, and retailers across the
globe
Slide 7
Slide 7 text
7
Feedzai Customers
Slide 8
Slide 8 text
8
About Feedzai
Slide 9
Slide 9 text
9
About Feedzai
Slide 10
Slide 10 text
10
About Feedzai
Slide 11
Slide 11 text
Ensuring high quality
11
Slide 12
Slide 12 text
● Bugs may cause
● (Continuous) automated testing is fundamental
12
High Quality Code
Slide 13
Slide 13 text
● All our code has Unit Tests (BE and FE)
● Many Integration tests (selenium)
● Many System Tests (docker)
● Failure Model (http://bit.ly/feedzai-failure-model)
13
High Quality Code
Slide 14
Slide 14 text
● Most run before merging code
● Few longer builds are run periodically
14
High Quality Code
Slide 15
Slide 15 text
15
High Quality Code
Slide 16
Slide 16 text
16
● How do I safely refactor my tests?
● How do I know if I can trust a test suite I inherited?
● How do I ensure my team is writing effective tests?
● How do I know I have enough tests to refactor my code safely?
High Quality Code
Slide 17
Slide 17 text
17
Not even TDD can answer these questions
High Quality Code
Slide 18
Slide 18 text
Code Coverage
18
Slide 19
Slide 19 text
19
Code Coverage is a measurement of the percentage of code lines
executed during the test suite.
Slide 20
Slide 20 text
20
● Line coverage
● Statement coverage
● Branch coverage
● Method coverage
Code Coverage
● Data coverage
● Path coverage
● Modified condition coverage
● etc
https://en.wikipedia.org/wiki/Code_coverage#Coverage_criteria
Slide 21
Slide 21 text
21
Code Coverage
Slide 22
Slide 22 text
22
100% CC but
what if X = 10?
Code Coverage
Slide 23
Slide 23 text
23
100% CC but
no asserts
Code Coverage
Slide 24
Slide 24 text
What is Mutation Testing (MT)?
24
Slide 25
Slide 25 text
25
Mutation Testing
Slide 26
Slide 26 text
26
● MT is a way to check the strength of our tests
● CC is not reliable as we have seen
● 1971 by Richard Lipton
● Basic idea: introduce small bugs and check if test suite finds those bugs
○ Netflix Chaos Monkey anyone?
Mutation Testing
Slide 27
Slide 27 text
27
public boolean isOldEnough(int age) {
return age >= 18;
}
public boolean isOldEnough(int age) {
return age > 18;
}
Mutation Testing
Slide 28
Slide 28 text
28
● MT simulates easy bugs - competent programmer hypothesis
According to this hypothesis, programmers write programs that are almost perfect. The competent
programmer hypothesis says that program faults are syntactically small and can be corrected with a few
keystrokes.
● MT should be included in the development lifecycle (should be run before
commit/push/PR)
Mutation Testing
Slide 29
Slide 29 text
Basic MT concepts
29
Slide 30
Slide 30 text
30
● Mutation Operators/mutators
● Mutants
● Killed mutation
● Survived mutation
● Equivalent Mutations
Mutation Testing - Basic Concepts
Slide 31
Slide 31 text
31
Mutation Testing - Equivalent Mutations
Slide 32
Slide 32 text
32
Mutation Testing - Equivalent Mutations
Slide 33
Slide 33 text
33
● Mutations in dead/useless code
● Mutations that affect only performance
● Mutations that can’t be triggered due to logic elsewhere in the program
● Mutations that alter only internal state
Mutation Testing - Equivalent Mutations
Slide 34
Slide 34 text
34
Mutation Testing - Examples
Slide 35
Slide 35 text
35
Mutation Testing - Examples
Slide 36
Slide 36 text
36
Mutation Testing - Examples
Slide 37
Slide 37 text
37
Mutation Testing - Examples
Slide 38
Slide 38 text
PIT - Java Mutation Testing Framework
38
Slide 39
Slide 39 text
39
PIT - Java Mutation Testing Framework
Slide 40
Slide 40 text
40
● Easy to use
● Maven support: http://pitest.org/quickstart/maven/
● Very efficient
● multi module support: https://github.com/STAMP-project/pitmp-maven-plugin
PIT - Java Mutation Testing Framework
Slide 41
Slide 41 text
41
PIT - Java Mutation Testing Framework
Slide 42
Slide 42 text
42
PIT - Java Mutation Testing Framework
$ mvn org.pitest:pitest-maven:mutationCoverage
$ mvn eu.stamp-project:pitmp-maven-plugin:run
Slide 43
Slide 43 text
43
● Report under /target/pit-reports/YYYYMMDDHHmm
PIT - Java Mutation Testing Framework
Slide 44
Slide 44 text
44
Slide 45
Slide 45 text
45
● MT score threshold to fail builds
● skip methods (toString, equals, hashcode)
● skip full classes
● skip test classes
PIT - Java Mutation Testing Framework
Slide 46
Slide 46 text
Adding MT to Feedzai codebase
46
Slide 47
Slide 47 text
47
● PoC using smaller repos
○ OpenML
■ https://github.com/feedzai/feedzai-openml/pull/33
■ https://github.com/feedzai/feedzai-openml/pull/34
● internal libs
Using MT at Feedzai
Slide 48
Slide 48 text
48
Using MT at Feedzai
Slide 49
Slide 49 text
49
● Redundant logic in code
● Missing test cases
○ boundary values
○ null checks
○ etc
Using MT at Feedzai
Slide 50
Slide 50 text
50
Using MT at Feedzai
Slide 51
Slide 51 text
51
Using MT at Feedzai
Slide 52
Slide 52 text
52
Using MT at Feedzai
Slide 53
Slide 53 text
53
Using MT at Feedzai
Slide 54
Slide 54 text
54
Can we add MT
everywhere?
Using MT at Feedzai
Slide 55
Slide 55 text
55
● Common complaints with MT
○ Equivalent mutations
○ MT is slow
■ compile code and run tests
■ PIT is very efficient (bytecode manipulation and coverage info)
● Doesn’t work with mocks
○ PIT supports JMock, EasyMock, Mockito, PowerMock and JMockit
Using MT at Feedzai
Slide 56
Slide 56 text
56
● Replace full method logic by “nullable” statement
● Much less mutants
● good preliminary analysis
● https://github.com/STAMP-project/pitest-descartes
Extreme Mutation
Slide 57
Slide 57 text
57
Extreme Mutation
Slide 58
Slide 58 text
58
Conclusion
● Keeping code base bug-free is hard
● We need to know our tests strength
● Code coverage has flaws
● Mutation Testing is a better metric
● MT is computational heavy...
● ...but seems the right path :)
Slide 59
Slide 59 text
59
● https://pedrorijo.com/blog/intro-mutation/
● https://github.com/mbj/mutant for Ruby
● http://stryker-mutator.io/ for Javascript, Scala, and C#
● https://github.com/sugakandrey/scalamu for Scala
Resources
Slide 60
Slide 60 text
60
@pedrorijo91 https://pedrorijo.com/
Questions?