Can You Trust Your Tests (Agile Tour 2015 Kaunas)

Nobody argues these days that unit tests are useful and provide valuable feedback about your code. But who watches the watchmen? Let's talk about test code quality, code coverage and introduce mutation based testing techniques.

Transcript

1. Can You Trust Your Tests? 2015 Vaidas Pilkauskas & Tadas Ščerbinskas
2. Van Halen
3. Band Tour Rider
4. Van Halen’s 1982 Tour Rider
6. Agenda 1. Test quality & code coverage 2. Mutation testing in theory 3. Mutation testing in practice
7. Prod vs. Test code quality Code has bugs. Tests are code. Tests have bugs.
9. Test quality Readable Focused Concise Well named
10. “Program testing can be used to show the presence of bugs, but never to show their absence!” - Edsger W. Dijkstra
11. Code Coverage
12. Types of Code Coverage Lines Branches Instructions Cyclomatic Complexity Methods & more
14. Lines string foo() { return "a" + "b" } assertThat(a.foo(), is("ab"))
15. Lines string foo(boolean arg) { return arg ? "a" : "b" } assertThat(a.foo(true), is("a"))
16. Branches string foo(boolean arg) { return arg ? "a" : "b" } assertThat(a.foo(true), is("a")) assertThat(a.foo(false), is("b"))
17. SUCCESS: 26/26 (100%) Tests passed
18. Can you trust 100% coverage? Code coverage can only show what is not tested. For interpreted languages 100% code coverage is kind of like full compilation.
19. Code Coverage can be gamed On purpose or by accident
20. Mutation testing
21. Mutation testing Changes your program code and expects your tests to fail.
22. What exactly is a mutation? def isFoo(a) { return a == foo } def isFoo(a) { return a != foo } def isFoo(a) { return true } def isFoo(a) { return null } > > >
23. Terminology Applying a mutation to some code creates a mutant. If test passes - mutant has survived. If test fails - mutant is killed.
24. Failing is the new passing
25. array = [a, b, c] max(array) == ???
26. // test max([0]) == 0 ✘
27. // test max([0]) == 0 ✔ // implementation max(a) { return 0 }
28. // test max([0]) == 0 ✔ max([1]) == 1 ✘ // implementation max(a) { return 0 }
29. // test max([0]) == 0 ✔ max([1]) == 1 ✔ // implementation max(a) { return a.first }
30. // test max([0]) == 0 ✔ max([1]) == 1 ✔ max([0, 2]) == 2 ✘ // implementation max(a) { return a.first }
31. // test max([0]) == 0 ✔ max([1]) == 1 ✔ max([0, 2]) == 2 ✔ // implementation max(a) { m = a.first for (e in a) if (e > m) m = e return m }
32. Coverage
33. Mutation // test max([0]) == 0 ✔ max([1]) == 1 ✔ max([0, 2]) == 2 ✔ // implementation max(a) { m = a.first for (e in a) if (e > m) m = e return m }
34. Mutation // test max([0]) == 0 ✔ max([1]) == 1 ✔ max([0, 2]) == 2 ✔ // implementation max(a) { m = a.first for (e in a) if (true) m = e return m }
35. // test max([0]) == 0 ✔ max([1]) == 1 ✔ max([0, 2]) == 2 ✔ // implementation max(a) { return a.last }
36. // test max([0]) == 0 ✔ max([1]) == 1 ✔ max([0, 2]) == 2 ✔ max([2, 1]) == 2 ✘ // implementation max(a) { return a.last }
37. // implementation max(a) { m = a.first for (e in a) if (e > m) m = e return m } // test max([0]) == 0 ✔ max([1]) == 1 ✔ max([0, 2]) == 2 ✔ max([2, 1]) == 2 ✔
38. Baby steps matter
39. Tests’ effectiveness is measured by number of killed mutants by your test suite.
40. It’s like hiring a white-hat hacker to try to break into your server and making sure you detect it.
41. What if mutant survives • Simplify your code • Add additional tests • TDD - minimal amount of code to pass the test
42. Challenges 1. High computation cost - slow 2. Equivalent mutants - false negatives 3. Infinite loops
43. Equivalent mutations // Original int i = 0; while (i != 10) { doSomething(); i += 1; } // Mutant int i = 0; while (i < 10) { doSomething(); i += 1; }
44. Infinite Runtime // Original while (expression) doSomething(); // Mutant while (true) doSomething();
45. Disadvantages • Can slow down your TDD rhythm • May be very noisy
46. Let’s say we have codebase with: • 300 classes • around 10 tests per class • 1 test runs around 1ms • total test suite runtime is about 3s Is it really slow? Let’s do 10 mutations per class • We get 3000 (300 * 10) mutations • runtime with all mutations is 150 minutes (3s * 3000)
47. Speeding it up Run only tests that cover the mutation • 300 classes • 10 tests per class • 10 mutations per class • 1ms test runtime • total mutation runtime 10 * 10 * 1 * 300 = 30s
48. Speeding it up During development run tests that cover only your current changes
49. • Continuous integration • TDD with mutation testing only on new changes • Add mutation testing to your legacy project, but do not fail a build - produce warning report Usage scenarios
50. Tools • Ruby - Mutant • Java - PIT • And many tools for other languages
51. Summary • Code coverage highlights code that is definitely not tested • Mutation testing highlights code that definitely is tested • Given non equivalent mutations, good test suite should work the same as a hash function
52. Vaidas Pilkauskas @liucijus • Vilnius JUG co-founder • Vilnius Scala leader • Coderetreat facilitator • Mountain bicycle rider • Snowboarder About us Tadas Ščerbinskas @tadassce • VilniusRB co-organizer • RubyConfLT co- organizer • RailsGirls Vilnius & Berlin coach • Various board sports’ enthusiast
53. Credits A lot of presentation content is based on work by these guys • Markus Schirp - author of Mutant • Henry Coles - author of PIT • Filip Van Laenen - working on a book
54. Q&A