Mutation testing with Humbug - TrueNorth PHP 2016

5e6bcf291601ee2e0faf35b30a839cb6?s=47 Marc Aubé
November 05, 2016

Mutation testing with Humbug - TrueNorth PHP 2016

Satisfied with your unit test code coverage? Are you sure that your code is thoroughly tested, and not merely executed by the tests? Learn what mutation testing is, and how Humbug can help you give your test suite a run for its money. Humbug is a tool that injects defects and regressions in your code and then checks if your tests noticed.

5e6bcf291601ee2e0faf35b30a839cb6?s=128

Marc Aubé

November 05, 2016
Tweet

Transcript

  1. Mutation testing with Humbug @TrueNorthPHP 2016

  2. $ whoami

  3. web developer @ DuProprio maaube / marcaube http://marcaube.ca

  4. The Test Suit

  5. None
  6. unit testing

  7. unit testing · prove the code works

  8. function sum($a, $b) { return $a + $b; } $this->assertEquals(3,

    sum(1, 2));
  9. $number = sum(1, 2.5); // 3.5? // 3? // 4?

  10. $number = sum(-42, 10); // probably -32, but...

  11. $number = sum(2**63, 2**63); // float(1.844674407371E+19)? // int(0)?

  12. unit testing · prove the code works · catch regressions

  13. unit testing · prove the code works · catch regressions

    · drive away from bad design
  14. unit testing · prove the code works · catch regressions

    · drive away from bad design · short feedback loop
  15. unit testing · prove the code works · catch regressions

    · drive away from bad design · short feedback loop · mature tooling
  16. the code coverage fallacy

  17. public function getPrice($tickets = 1, $hasCoupon = false) { $price

    = $tickets * 945.00; if ($tickets >= 10 || $hasCoupon) { $price *= 0.8; } return $price; }
  18. public function testCanCalculateOrdersWithRegularPrice() { $this->assertEquals(945.00, $calculator->getPrice(1)); } public function testCanCalculateOrdersWithDiscountedPrice()

    { $this->assertEquals(756.00, $calculator->getPrice(1, true)); }
  19. 100% coverage!

  20. SUCCESS: 26/26 (100%)

  21. 100% line coverage

  22. public function getPrice($tickets = 1, $hasCoupon = false) { $price

    = $tickets * 945.00; if ($tickets >= 10 || $hasCoupon) { $price *= 0.8; } return $price; }
  23. public function testCanCalculateThePriceOfARegularOrder() { $this->assertEquals(945.00, $calculator->getPrice()); } public function testAppliesADiscountForOrdersOfTenTicketsOrMore()

    { $this->assertEquals(11340.00, $calculator->getPrice(15)); } public function testAppliesADiscountForOrdersWithCoupon() { $this->assertEquals(756.00, $calculator->getPrice(1, true)); }
  24. mutation testing

  25. Code has bugs, Tests are code, tests have bugs. —

    Abraham Lincoln
  26. mutation testing · identify weaknesses in unit tests

  27. None
  28. mutation testing · identify weaknesses in unit tests · most

    effective for high coverage
  29. mutation testing · identify weaknesses in unit tests · most

    effective for high coverage · complements unit testing
  30. how it works

  31. Arithmetic Mutators

  32. Boolean Mutators

  33. Comparison Mutators

  34. a bit of lingo...

  35. Mutation Testing is commencing on 1 files... (.: killed, M:

    escaped, S: uncovered, E: fatal error, T: timed out) ..M.M... 8 mutations were generated: 6 mutants were killed 0 mutants were not covered by tests 2 covered mutants were not detected 0 fatal errors were encountered 0 time outs were encountered
  36. 2) \Humbug\Mutator\Number\IntegerValue Diff on \Confoo\PriceCalculator::getPrice() in src/PriceCalculator.php: --- Original +++

    New @@ @@ - if ($tickets >= 10 || $hasCoupon) { + if ($tickets >= 11 || $hasCoupon) { $price *= 0.8; } return $price; } }
  37. equivalent mutants (false positives) // ORIGINAL CODE // MUTANT $index

    = 0; $index = 0; while ($index != 10) { while ($index < 10) { // do stuff // do stuff $index++; $index++; } }
  38. The metrics

  39. Metrics: Mutation Score Indicator (MSI): 75% Mutation Code Coverage: 100%

    Covered Code MSI: 75% Remember that some mutants will inevitably be harmless (i.e. false positives). Time: 3.74 seconds Memory: 6.00MB Humbug results are being logged as TEXT to: humbuglog.txt
  40. Mutation Score Indicator (MSI): 75%

  41. Mutation Code Coverage: 100%

  42. Covered Code MSI: 75%

  43. Installing Humbug $ composer require --dev "humbug/humbug:^1.0@dev" ./composer.json has been

    updated Loading composer repositories with package information Updating dependencies (including require-dev) - Installing humbug/humbug (dev-master 06b1c05) Cloning 06b1c059e432dab8c22c36bc8b6e1ffc7e587c07 Writing lock file Generating autoload files
  44. Configuring Humbug $ vendor/bin/humbug configure _ _ _ | ||

    |_ _ _ __ | |__ _ _ __ _ | __ | || | ' \| '_ \ || / _` | |_||_|\_,_|_|_|_|_.__/\_,_\__, | |___/ Humbug version 1.0-dev Humbug configuration tool. It will guide you through Humbug configuration in few seconds.
  45. humbug.json.dist { "source": { "directories": [ "src" ] }, "timeout":

    10, "logs": { "text": "humbuglog.txt" } }
  46. .gitignore # Humbug config humbug.json # Humbug logs humbuglog.txt humbuglog.json

  47. Running Humbug $ vendor/bin/humbug

  48. Options and parameters

  49. Infinite loop timeout humbug --timeout=50

  50. Restricting Files To Mutate humbug --file=NewClass.php --file=*Driver.php

  51. Incremental Analysis humbug --incremental

  52. The good, the bad and the ugly

  53. Program testing can be used to show the presence of

    bugs, but never to show their absence. — Edsger W. Dijkstra
  54. Questions?

  55. Let's try it!

  56. Let's try it!

  57. Another interesting testing tool you should def check out Eris

    <?php use Eris\Generator; class ReadmeTest extends \PHPUnit_Framework_TestCase { use Eris\TestTrait; public function testNaturalNumbersMagnitude() { $this->forAll( Generator\choose(0, 1000) ) ->then(function($number) { $this->assertTrue( $number < 42, "$number is not less than 42 apparently" ); }); } }