Mutation Testing

For Java unit tests you can rely upon

Transcript

1. Mutation testing Java Unit tests you can rely upon
2. Mutation testing "I will inject bugs ('mutants') into your code, behind your back, and check your tests fail." If they still pass, your poor-quality tests are giving you a false sense of security
3. Recap: unit testing Test units ('bits') of your code in isolation, in a controlled way.
4. Doing it wrong 1. Got to increase the code coverage %
5. Doing it wrong 1. Got to increase the code coverage % It ran, but what did it do? How do you know?
6. Doing it wrong 1. Got to increase the code coverage % It ran, but what did it do? How do you know? 2. Got to make the tests pass!
7. Doing it wrong 1. Got to increase the code coverage % It ran, but what did it do? How do you know? 2. Got to make the tests pass! Each test that fails is one less bug raised at 5: 28 on Friday afternoon. Tests that can't fail are a waste of your time.
8. Recap: assertions Does the code actually do what it says it does? List x = empty assertThat( x, isEmpty) addOneEntryTo(x) assertThat( x, notEmpty) assertThat( x.size, is(1))
9. Assertions done wrong Too often we get tests like this: Report x = createReport() assertThat( x, notNull) assertThat( x.numRows, atLeast(1)) ... long isbn = getIsbn() assertThat( isbn, greaterThan(0)) Minimal protection: why waste your time?
10. Mutation testing II Don't get confused: does not find bugs. Nor about simulating malicious inputs, security, etc. • If bugs are crimes... • And your tests are the Police... • Mutants are fake crimes that let you see if the Police are doing their job.
11. Example test 1 report = getReport() // no assertion at all Mutation tool tries Mutant: Report getReport() { return new Report(); } ==> Report getReport() { new Report(); return null; } Test still passes = rubbish
12. Example test 2 report = getReport() assertThat( report.numRows, greaterThan(0)) Mutation tool tries Mutant: int numRows() { if specialTest return 2 else 1 } ==> int numRows() { if !specialTest return 2 else 1 } Test still passes = rubbish
13. How to improve I Add good, accurate assertions ('enforce the contract') Which bits of the system don't you want to break at 5:28 on Friday afternoon?
14. How to improve II If you can't, or don't know how to, this is the fundamental problem. Come back when you: • Understand the system you're testing better • Fix your test data / test envt to give yourself more control (If the developer wrote untestable code, there's only so much you can do...)
15. Doing it better // load test data s = getSpecialReport() assertThat( s.numRows, is(2)) n = getNormalReport() assertThat( n.numRows, is(1)) Mutation tool tries Mutant: int numRows() { if specialTest return 2 else 1 } ==> int numRows() { if !specialTest return 2 else 1 } Test fails, mutant killed = good
16. Time is money All testing will be done, and all bugs will be fixed. Question is: where, when, and on whose terms? Unit- and mutation-test when it makes sense, when it helps you, not to hit arbitrary targets.
17. Using PIT & Sonar PIT pitest.org - a good Java implementation. > mvn org.pitest:pitest-maven:mutationCoverage Generates colourful reports by class/package. Has a Sonar plugin for "survived mutant" violations.
18. Next steps 1. Try it out on your own Maven project and see how you get on.