WHO DOES
• UNIT TESTING
• TEST-DRIVEN DEVELOPMENT
• CONTINUOUS INTEGRATION
• MEASURE CODE COVERAGE
• MUTATION TESTING
4
Slide 5
Slide 5 text
PROBLEMS
• How do I safely refactor my tests?
• How do I know I can trust a test
suite?
• How do I ensure my team is writing
effective tests
• How do I know if I’ve retrofitted
enough tests to safely refactor a piece
of legacy code?
5
Slide 6
Slide 6 text
JUST ONE QUESTION
HOW DO I ASSESS THE QUALITY
OF THE TEST SUITE?
6
Slide 7
Slide 7 text
COMMON ANSWERS
• Don’t worry, it’ll be fine
• I’m a ninja rockstar, I know my tests are good
• I do TDD, I know my tests are good
• What about the tests you didn’t write?
• How do you test drive changes to tests?
• Code review
• Inconsistent + Labour intensive
• Code coverage
7
Slide 8
Slide 8 text
CODE COVERAGE MEASURE
DOES NOT TELL YOU WHICH
PART HAS BEEN TESTED
8
Slide 9
Slide 9 text
WHAT CODE COVERAGE DOES TELL YOU
9
Slide 10
Slide 10 text
EXECUTED ≠ TESTED
Executed
Tested
10
Slide 11
Slide 11 text
CODE COVERAGE TELLS YOU ONLY
WHAT HAS NOT BEEN TESTED
11
Slide 12
Slide 12 text
WHERE THOSE LINES TESTED?
12
Slide 13
Slide 13 text
13
Slide 14
Slide 14 text
14
Slide 15
Slide 15 text
15
Slide 16
Slide 16 text
OUR TESTS STILL PASS.
OUR TEST SUITE IS DEFICIENT
16
Slide 17
Slide 17 text
A TEST CASE IS MISSING
17
Slide 18
Slide 18 text
HOW TO DETECT IF A TEST SUITE IS
DEFICIENT?
18
Slide 19
Slide 19 text
INTRODUCE A BUG
19
Slide 20
Slide 20 text
MUTATION TESTING IN PICTURES
20
Slide 21
Slide 21 text
CREATE A MUTANT
SOURCE CODE
MUTATION OPERATOR
MUTATION
PROCESS
MUTANT
21
Slide 22
Slide 22 text
EXAMPLES
• A + B —> A - B
• A && B —> A || B
• Literal value replacement: 0 -> 1,
true -> false…
• Statement deletion: return, break,
continue…
• …
22
Slide 23
Slide 23 text
TRY TO KILL THE MUTANT
MUTANT
TESTS - KILLERS
TEST SUITE
23
Slide 24
Slide 24 text
IF A MUTANT DOES NOT CAUSE THE
TESTS TO FAIL, IT SURVIVED
24
Slide 25
Slide 25 text
IF A MUTANT DOES CAUSE THE
TESTS TO FAIL, IT WAS KILLED
25
Slide 26
Slide 26 text
MUTATION SCORE
Nbr of mutant killed
Nbr of mutant escaped
Mutation score =
26
Slide 27
Slide 27 text
CODE COVERAGE HIGHLIGHTS CODE
THAT IS DEFINITELY NOT TESTED
MUTANT SCORE HIGHLIGHT CODE
THAT IS DEFINITELY TESTED
27
WHY DOES IT WORK?
“Complex faults are coupled to simple
faults in such a way that a test data set
that detects all simple faults in a program
will detect most complex faults”
Demonstrated in 1995 by K. Wah, “Fault coupling in finite bijective functions”
37
Slide 38
Slide 38 text
IT IS NOT NEW… - HISTORY
• Begins in 1971, R. Lipton, “Fault
Diagnosis of Computer Programs”
• Generally accepted in 1978, R.
Lipton and al, “Hints on test data
selection: Help for the practicing
programmer”
38
Slide 39
Slide 39 text
WHY IS IT NOT WIDELY USED?
Maturity Problem: Because testing is
not widely used yet
(Although it is increasing)
39
Slide 40
Slide 40 text
WHY IS IT NOT WIDELY USED?
Integration Problem: Inability to
successfully integrated it into software
development process
(TDD plays a key role now)
40
Slide 41
Slide 41 text
WHY IS IT NOT WIDELY USED?
Technical Problem: It is a brute force
technique!
41
Slide 42
Slide 42 text
BRUTE FORCE TECHNIQUE
N: Number of tests
M: Number of mutants
NxM
42
Slide 43
Slide 43 text
OPTIMISATION STRATEGIES
• 2,155 tests in 6.82 seconds
(0.00316s/tests)
• 957 mutants
• 2,062,335 tests
• ~1h49min
With Humbug
43
Slide 44
Slide 44 text
44
Slide 45
Slide 45 text
OPTIMISATION STRATEGIES
• Reduce equivalent mutant detection
techniques (10 to 40% of mutants are
equivalent)
• Take coverage to mutate only covered
code
• Rely on coverage to know which tests
to run and which to run first
• Increment analysis
45