Mutation testing with Humbug - Confoo Montréal 2016

5e6bcf291601ee2e0faf35b30a839cb6?s=47 Marc Aubé
February 26, 2016

Mutation testing with Humbug - Confoo Montréal 2016

Satisfied with your unit test code coverage? Are you sure that your code is thoroughly tested, and not merely executed by the tests? Learn what mutation testing is, and how Humbug can help you give your test suite a run for its money. Humbug is a tool that injects defects and regressions in your code and then checks if your tests noticed.

5e6bcf291601ee2e0faf35b30a839cb6?s=128

Marc Aubé

February 26, 2016
Tweet

Transcript

  1. 4.
  2. 13.

    UNIT TESTING > prove the code works > catch regressions

    > drive away from bad design > short feedback loop
  3. 14.

    UNIT TESTING > prove the code works > catch regressions

    > drive away from bad design > short feedback loop > mature tooling
  4. 16.

    public function getPrice($tickets = 1, $hasCoupon = false) { $price

    = $tickets * 945.00; if ($tickets >= 10 || $hasCoupon) { $price *= 0.8; } return $price; }
  5. 21.

    public function getPrice($tickets = 1, $hasCoupon = false) { $price

    = $tickets * 945.00; if ($tickets >= 10 || $hasCoupon) { $price *= 0.8; } return $price; }
  6. 22.

    public function testCanCalculateThePriceOfARegularOrder() { $this->assertEquals(945.00, $calculator->getPrice()); } public function testAppliesADiscountForOrdersOfTenTicketsOrMore()

    { $this->assertEquals(11340.00, $calculator->getPrice(15)); } public function testAppliesADiscountForOrdersWithCoupon() { $this->assertEquals(756.00, $calculator->getPrice(1, true)); }
  7. 27.

    MUTATION TESTING > identify weaknesses in unit tests > most

    effective for high coverage > complements unit testing
  8. 33.

    Mutation Testing is commencing on 1 files... (.: killed, M:

    escaped, S: uncovered, E: fatal error, T: timed out) ..M.M... 8 mutations were generated: 6 mutants were killed 0 mutants were not covered by tests 2 covered mutants were not detected 0 fatal errors were encountered 0 time outs were encountered
  9. 34.

    2) \Humbug\Mutator\Number\IntegerValue Diff on \Confoo\PriceCalculator::getPrice() in src/PriceCalculator.php: --- Original +++

    New @@ @@ - if ($tickets >= 10 || $hasCoupon) { + if ($tickets >= 11 || $hasCoupon) { $price *= 0.8; } return $price; } }
  10. 35.

    EQUIVALENT MUTANTS (FALSE POSITIVES) // ORIGINAL CODE // MUTANT $index

    = 0; $index = 0; while ($index != 10) { while ($index < 10) { // do stuff // do stuff $index++; $index++; } }
  11. 37.

    INSTALLING HUMBUG $ composer require --dev "humbug/humbug:~1.0@dev" ./composer.json has been

    updated Loading composer repositories with package information Updating dependencies (including require-dev) - Installing humbug/humbug (dev-master d80edfc) Cloning d80edfcb3e14cd3afce4b0cf8169565af33e1657 Writing lock file Generating autoload files
  12. 38.

    CONFIGURING HUMBUG $ vendor/bin/humbug configure _ _ _ | ||

    |_ _ _ __ | |__ _ _ __ _ | __ | || | ' \| '_ \ || / _` | |_||_|\_,_|_|_|_|_.__/\_,_\__, | |___/ Humbug version 1.0-dev Humbug configuration tool. It will guide you through Humbug configuration in few seconds.
  13. 39.
  14. 44.

    Program testing can be used to show the presence of

    bugs, but never to show their absence. — Edsger W. Dijkstra