Mutation testing with Humbug - Confoo Montréal 2016

5e6bcf291601ee2e0faf35b30a839cb6?s=47 Marc Aubé
February 26, 2016

Mutation testing with Humbug - Confoo Montréal 2016

Satisfied with your unit test code coverage? Are you sure that your code is thoroughly tested, and not merely executed by the tests? Learn what mutation testing is, and how Humbug can help you give your test suite a run for its money. Humbug is a tool that injects defects and regressions in your code and then checks if your tests noticed.

5e6bcf291601ee2e0faf35b30a839cb6?s=128

Marc Aubé

February 26, 2016
Tweet

Transcript

  1. MUTATION TESTING WITH HUMBUG @CONFOO 2016

  2. $ whoami

  3. WEB DEVELOPER @ DUPROPRIO MAAUBE / MARCAUBE HTTP://MARCAUBE.CA

  4. None
  5. UNIT TESTING

  6. UNIT TESTING > prove the code works

  7. function sum($a, $b) { return $a + $b; } $this->assertEquals(3,

    sum(1, 2));
  8. $number = sum(1, 2.5)); // 3.5? // 3? // 4?

  9. $number = sum(-42, 10); // probably -32, but...

  10. $number = sum(2**63, 2**63); // float(1.844674407371E+19)? // int(0)?

  11. UNIT TESTING > prove the code works > catch regressions

  12. UNIT TESTING > prove the code works > catch regressions

    > drive away from bad design
  13. UNIT TESTING > prove the code works > catch regressions

    > drive away from bad design > short feedback loop
  14. UNIT TESTING > prove the code works > catch regressions

    > drive away from bad design > short feedback loop > mature tooling
  15. THE CODE COVERAGE FALLACY

  16. public function getPrice($tickets = 1, $hasCoupon = false) { $price

    = $tickets * 945.00; if ($tickets >= 10 || $hasCoupon) { $price *= 0.8; } return $price; }
  17. public function testCanCalculateOrdersWithRegularPrice() { $this->assertEquals(945.00, $calculator->getPrice(1)); } public function testCanCalculateOrdersWithDiscountedPrice()

    { $this->assertEquals(756.00, $calculator->getPrice(1, true)); }
  18. 100% COVERAGE!

  19. SUCCESS: 26/26 (100%)

  20. 100% LINE COVERAGE

  21. public function getPrice($tickets = 1, $hasCoupon = false) { $price

    = $tickets * 945.00; if ($tickets >= 10 || $hasCoupon) { $price *= 0.8; } return $price; }
  22. public function testCanCalculateThePriceOfARegularOrder() { $this->assertEquals(945.00, $calculator->getPrice()); } public function testAppliesADiscountForOrdersOfTenTicketsOrMore()

    { $this->assertEquals(11340.00, $calculator->getPrice(15)); } public function testAppliesADiscountForOrdersWithCoupon() { $this->assertEquals(756.00, $calculator->getPrice(1, true)); }
  23. MUTATION TESTING

  24. Code has bugs, Tests are code, tests have bugs. —

    Abraham Lincoln
  25. MUTATION TESTING > identify weaknesses in unit tests

  26. MUTATION TESTING > identify weaknesses in unit tests > most

    effective for high coverage
  27. MUTATION TESTING > identify weaknesses in unit tests > most

    effective for high coverage > complements unit testing
  28. HOW IT WORKS

  29. ARITHMETIC MUTATORS

  30. BOOLEAN MUTATORS

  31. COMPARISON MUTATORS

  32. A BIT OF LINGO...

  33. Mutation Testing is commencing on 1 files... (.: killed, M:

    escaped, S: uncovered, E: fatal error, T: timed out) ..M.M... 8 mutations were generated: 6 mutants were killed 0 mutants were not covered by tests 2 covered mutants were not detected 0 fatal errors were encountered 0 time outs were encountered
  34. 2) \Humbug\Mutator\Number\IntegerValue Diff on \Confoo\PriceCalculator::getPrice() in src/PriceCalculator.php: --- Original +++

    New @@ @@ - if ($tickets >= 10 || $hasCoupon) { + if ($tickets >= 11 || $hasCoupon) { $price *= 0.8; } return $price; } }
  35. EQUIVALENT MUTANTS (FALSE POSITIVES) // ORIGINAL CODE // MUTANT $index

    = 0; $index = 0; while ($index != 10) { while ($index < 10) { // do stuff // do stuff $index++; $index++; } }
  36. THE METRICS Mutation Score Indicator (MSI): 75% Mutation Code Coverage:

    100% Covered Code MSI: 75%
  37. INSTALLING HUMBUG $ composer require --dev "humbug/humbug:~1.0@dev" ./composer.json has been

    updated Loading composer repositories with package information Updating dependencies (including require-dev) - Installing humbug/humbug (dev-master d80edfc) Cloning d80edfcb3e14cd3afce4b0cf8169565af33e1657 Writing lock file Generating autoload files
  38. CONFIGURING HUMBUG $ vendor/bin/humbug configure _ _ _ | ||

    |_ _ _ __ | |__ _ _ __ _ | __ | || | ' \| '_ \ || / _` | |_||_|\_,_|_|_|_|_.__/\_,_\__, | |___/ Humbug version 1.0-dev Humbug configuration tool. It will guide you through Humbug configuration in few seconds.
  39. HUMBUG.JSON.DIST { "source": { "directories": [ "src" ] }, "timeout":

    10, "logs": { "text": "humbuglog.txt" } }
  40. .GITIGNORE # Humbug config humbug.json # Humbug logs humbuglog.txt humbuglog.json

  41. THE GOOD, THE BAD AND THE UGLY

  42. { "require": { "php": ">=5.4.0", "phpunit/phpunit": "^4.5|^5.0" } }

  43. IT IS SLOW

  44. Program testing can be used to show the presence of

    bugs, but never to show their absence. — Edsger W. Dijkstra
  45. THANK YOU!

  46. QUESTIONS?

  47. https://github.com/padraic/humbug