Mutation testing with Humbug - Confoo Montréal 2016

Satisfied with your unit test code coverage? Are you sure that your code is thoroughly tested, and not merely executed by the tests? Learn what mutation testing is, and how Humbug can help you give your test suite a run for its money. Humbug is a tool that injects defects and regressions in your code and then checks if your tests noticed.

Transcript

1. MUTATION TESTING WITH HUMBUG @CONFOO 2016
2. $ whoami
3. WEB DEVELOPER @ DUPROPRIO MAAUBE / MARCAUBE HTTP://MARCAUBE.CA
5. UNIT TESTING
6. UNIT TESTING > prove the code works
7. function sum($a, $b) { return $a + $b; } $this->assertEquals(3, sum(1, 2));
8. $number = sum(1, 2.5)); // 3.5? // 3? // 4?
9. $number = sum(-42, 10); // probably -32, but...
10. $number = sum(2**63, 2**63); // float(1.844674407371E+19)? // int(0)?
11. UNIT TESTING > prove the code works > catch regressions
12. UNIT TESTING > prove the code works > catch regressions > drive away from bad design
13. UNIT TESTING > prove the code works > catch regressions > drive away from bad design > short feedback loop
14. UNIT TESTING > prove the code works > catch regressions > drive away from bad design > short feedback loop > mature tooling
15. THE CODE COVERAGE FALLACY
16. public function getPrice($tickets = 1, $hasCoupon = false) { $price = $tickets * 945.00; if ($tickets >= 10 || $hasCoupon) { $price *= 0.8; } return $price; }
17. public function testCanCalculateOrdersWithRegularPrice() { $this->assertEquals(945.00, $calculator->getPrice(1)); } public function testCanCalculateOrdersWithDiscountedPrice() { $this->assertEquals(756.00, $calculator->getPrice(1, true)); }
18. 100% COVERAGE!
19. SUCCESS: 26/26 (100%)
20. 100% LINE COVERAGE
21. public function getPrice($tickets = 1, $hasCoupon = false) { $price = $tickets * 945.00; if ($tickets >= 10 || $hasCoupon) { $price *= 0.8; } return $price; }
22. public function testCanCalculateThePriceOfARegularOrder() { $this->assertEquals(945.00, $calculator->getPrice()); } public function testAppliesADiscountForOrdersOfTenTicketsOrMore() { $this->assertEquals(11340.00, $calculator->getPrice(15)); } public function testAppliesADiscountForOrdersWithCoupon() { $this->assertEquals(756.00, $calculator->getPrice(1, true)); }
23. MUTATION TESTING
24. Code has bugs, Tests are code, tests have bugs. — Abraham Lincoln
25. MUTATION TESTING > identify weaknesses in unit tests
26. MUTATION TESTING > identify weaknesses in unit tests > most effective for high coverage
27. MUTATION TESTING > identify weaknesses in unit tests > most effective for high coverage > complements unit testing
28. HOW IT WORKS
29. ARITHMETIC MUTATORS
30. BOOLEAN MUTATORS
31. COMPARISON MUTATORS
32. A BIT OF LINGO...
33. Mutation Testing is commencing on 1 files... (.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out) ..M.M... 8 mutations were generated: 6 mutants were killed 0 mutants were not covered by tests 2 covered mutants were not detected 0 fatal errors were encountered 0 time outs were encountered
34. 2) \Humbug\Mutator\Number\IntegerValue Diff on \Confoo\PriceCalculator::getPrice() in src/PriceCalculator.php: --- Original +++ New @@ @@ - if ($tickets >= 10 || $hasCoupon) { + if ($tickets >= 11 || $hasCoupon) { $price *= 0.8; } return $price; } }
35. EQUIVALENT MUTANTS (FALSE POSITIVES) // ORIGINAL CODE // MUTANT $index = 0; $index = 0; while ($index != 10) { while ($index < 10) { // do stuff // do stuff $index++; $index++; } }
36. THE METRICS Mutation Score Indicator (MSI): 75% Mutation Code Coverage: 100% Covered Code MSI: 75%
37. INSTALLING HUMBUG $ composer require --dev "humbug/humbug:~1.0@dev" ./composer.json has been updated Loading composer repositories with package information Updating dependencies (including require-dev) - Installing humbug/humbug (dev-master d80edfc) Cloning d80edfcb3e14cd3afce4b0cf8169565af33e1657 Writing lock file Generating autoload files
38. CONFIGURING HUMBUG $ vendor/bin/humbug configure _ _ _ | || |_ _ _ __ | |__ _ _ __ _ | __ | || | ' \| '_ \ || / _` | |_||_|\_,_|_|_|_|_.__/\_,_\__, | |___/ Humbug version 1.0-dev Humbug configuration tool. It will guide you through Humbug configuration in few seconds.
39. HUMBUG.JSON.DIST { "source": { "directories": [ "src" ] }, "timeout": 10, "logs": { "text": "humbuglog.txt" } }
40. .GITIGNORE # Humbug config humbug.json # Humbug logs humbuglog.txt humbuglog.json
41. THE GOOD, THE BAD AND THE UGLY
42. { "require": { "php": ">=5.4.0", "phpunit/phpunit": "^4.5|^5.0" } }
43. IT IS SLOW
44. Program testing can be used to show the presence of bugs, but never to show their absence. — Edsger W. Dijkstra
45. THANK YOU!
46. QUESTIONS?
47. https://github.com/padraic/humbug