Mutation testing with Humbug - TrueNorth PHP 2016

Satisfied with your unit test code coverage? Are you sure that your code is thoroughly tested, and not merely executed by the tests? Learn what mutation testing is, and how Humbug can help you give your test suite a run for its money. Humbug is a tool that injects defects and regressions in your code and then checks if your tests noticed.

Transcript

1. Mutation testing with Humbug @TrueNorthPHP 2016
2. $ whoami
3. web developer @ DuProprio maaube / marcaube http://marcaube.ca
4. The Test Suit
6. unit testing
7. unit testing · prove the code works
8. function sum($a, $b) { return $a + $b; } $this->assertEquals(3, sum(1, 2));
9. $number = sum(1, 2.5); // 3.5? // 3? // 4?
10. $number = sum(-42, 10); // probably -32, but...
11. $number = sum(2**63, 2**63); // float(1.844674407371E+19)? // int(0)?
12. unit testing · prove the code works · catch regressions
13. unit testing · prove the code works · catch regressions · drive away from bad design
14. unit testing · prove the code works · catch regressions · drive away from bad design · short feedback loop
15. unit testing · prove the code works · catch regressions · drive away from bad design · short feedback loop · mature tooling
16. the code coverage fallacy
17. public function getPrice($tickets = 1, $hasCoupon = false) { $price = $tickets * 945.00; if ($tickets >= 10 || $hasCoupon) { $price *= 0.8; } return $price; }
18. public function testCanCalculateOrdersWithRegularPrice() { $this->assertEquals(945.00, $calculator->getPrice(1)); } public function testCanCalculateOrdersWithDiscountedPrice() { $this->assertEquals(756.00, $calculator->getPrice(1, true)); }
19. 100% coverage!
20. SUCCESS: 26/26 (100%)
21. 100% line coverage
22. public function getPrice($tickets = 1, $hasCoupon = false) { $price = $tickets * 945.00; if ($tickets >= 10 || $hasCoupon) { $price *= 0.8; } return $price; }
23. public function testCanCalculateThePriceOfARegularOrder() { $this->assertEquals(945.00, $calculator->getPrice()); } public function testAppliesADiscountForOrdersOfTenTicketsOrMore() { $this->assertEquals(11340.00, $calculator->getPrice(15)); } public function testAppliesADiscountForOrdersWithCoupon() { $this->assertEquals(756.00, $calculator->getPrice(1, true)); }
24. mutation testing
25. Code has bugs, Tests are code, tests have bugs. — Abraham Lincoln
26. mutation testing · identify weaknesses in unit tests
28. mutation testing · identify weaknesses in unit tests · most effective for high coverage
29. mutation testing · identify weaknesses in unit tests · most effective for high coverage · complements unit testing
30. how it works
31. Arithmetic Mutators
32. Boolean Mutators
33. Comparison Mutators
34. a bit of lingo...
35. Mutation Testing is commencing on 1 files... (.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out) ..M.M... 8 mutations were generated: 6 mutants were killed 0 mutants were not covered by tests 2 covered mutants were not detected 0 fatal errors were encountered 0 time outs were encountered
36. 2) \Humbug\Mutator\Number\IntegerValue Diff on \Confoo\PriceCalculator::getPrice() in src/PriceCalculator.php: --- Original +++ New @@ @@ - if ($tickets >= 10 || $hasCoupon) { + if ($tickets >= 11 || $hasCoupon) { $price *= 0.8; } return $price; } }
37. equivalent mutants (false positives) // ORIGINAL CODE // MUTANT $index = 0; $index = 0; while ($index != 10) { while ($index < 10) { // do stuff // do stuff $index++; $index++; } }
38. The metrics
39. Metrics: Mutation Score Indicator (MSI): 75% Mutation Code Coverage: 100% Covered Code MSI: 75% Remember that some mutants will inevitably be harmless (i.e. false positives). Time: 3.74 seconds Memory: 6.00MB Humbug results are being logged as TEXT to: humbuglog.txt
40. Mutation Score Indicator (MSI): 75%
41. Mutation Code Coverage: 100%
42. Covered Code MSI: 75%
43. Installing Humbug $ composer require --dev "humbug/humbug:^1.0@dev" ./composer.json has been updated Loading composer repositories with package information Updating dependencies (including require-dev) - Installing humbug/humbug (dev-master 06b1c05) Cloning 06b1c059e432dab8c22c36bc8b6e1ffc7e587c07 Writing lock file Generating autoload files
44. Configuring Humbug $ vendor/bin/humbug configure _ _ _ | || |_ _ _ __ | |__ _ _ __ _ | __ | || | ' \| '_ \ || / _` | |_||_|\_,_|_|_|_|_.__/\_,_\__, | |___/ Humbug version 1.0-dev Humbug configuration tool. It will guide you through Humbug configuration in few seconds.
45. humbug.json.dist { "source": { "directories": [ "src" ] }, "timeout": 10, "logs": { "text": "humbuglog.txt" } }
46. .gitignore # Humbug config humbug.json # Humbug logs humbuglog.txt humbuglog.json
47. Running Humbug $ vendor/bin/humbug
48. Options and parameters
49. Infinite loop timeout humbug --timeout=50
50. Restricting Files To Mutate humbug --file=NewClass.php --file=*Driver.php
51. Incremental Analysis humbug --incremental
52. The good, the bad and the ugly
53. Program testing can be used to show the presence of bugs, but never to show their absence. — Edsger W. Dijkstra
54. Questions?
55. Let's try it!
56. Let's try it!
57. Another interesting testing tool you should def check out Eris <?php use Eris\Generator; class ReadmeTest extends \PHPUnit_Framework_TestCase { use Eris\TestTrait; public function testNaturalNumbersMagnitude() { $this->forAll( Generator\choose(0, 1000) ) ->then(function($number) { $this->assertTrue( $number < 42, "$number is not less than 42 apparently" ); }); } }