Mutation testing with Humbug - TrueNorth PHP 2016

Slide 1

Slide 1 text

Mutation testing with Humbug @TrueNorthPHP 2016

Slide 2

Slide 2 text

$ whoami

Slide 3

Slide 3 text

web developer @ DuProprio maaube / marcaube http://marcaube.ca

Slide 4

Slide 4 text

The Test Suit

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

unit testing

Slide 7

Slide 7 text

unit testing · prove the code works

Slide 8

Slide 8 text

function sum($a, $b) { return $a + $b; } $this->assertEquals(3, sum(1, 2));

Slide 9

Slide 9 text

$number = sum(1, 2.5); // 3.5? // 3? // 4?

Slide 10

Slide 10 text

$number = sum(-42, 10); // probably -32, but...

Slide 11

Slide 11 text

$number = sum(2**63, 2**63); // float(1.844674407371E+19)? // int(0)?

Slide 12

Slide 12 text

unit testing · prove the code works · catch regressions

Slide 13

Slide 13 text

unit testing · prove the code works · catch regressions · drive away from bad design

Slide 14

Slide 14 text

unit testing · prove the code works · catch regressions · drive away from bad design · short feedback loop

Slide 15

Slide 15 text

unit testing · prove the code works · catch regressions · drive away from bad design · short feedback loop · mature tooling

Slide 16

Slide 16 text

the code coverage fallacy

Slide 17

Slide 17 text

public function getPrice($tickets = 1, $hasCoupon = false) { $price = $tickets * 945.00; if ($tickets >= 10 || $hasCoupon) { $price *= 0.8; } return $price; }

Slide 18

Slide 18 text

public function testCanCalculateOrdersWithRegularPrice() { $this->assertEquals(945.00, $calculator->getPrice(1)); } public function testCanCalculateOrdersWithDiscountedPrice() { $this->assertEquals(756.00, $calculator->getPrice(1, true)); }

Slide 19

Slide 19 text

100% coverage!

Slide 20

Slide 20 text

SUCCESS: 26/26 (100%)

Slide 21

Slide 21 text

100% line coverage

Slide 22

Slide 22 text

public function getPrice($tickets = 1, $hasCoupon = false) { $price = $tickets * 945.00; if ($tickets >= 10 || $hasCoupon) { $price *= 0.8; } return $price; }

Slide 23

Slide 23 text

public function testCanCalculateThePriceOfARegularOrder() { $this->assertEquals(945.00, $calculator->getPrice()); } public function testAppliesADiscountForOrdersOfTenTicketsOrMore() { $this->assertEquals(11340.00, $calculator->getPrice(15)); } public function testAppliesADiscountForOrdersWithCoupon() { $this->assertEquals(756.00, $calculator->getPrice(1, true)); }

Slide 24

Slide 24 text

mutation testing

Slide 25

Slide 25 text

Code has bugs, Tests are code, tests have bugs. — Abraham Lincoln

Slide 26

Slide 26 text

mutation testing · identify weaknesses in unit tests

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

mutation testing · identify weaknesses in unit tests · most effective for high coverage

Slide 29

Slide 29 text

mutation testing · identify weaknesses in unit tests · most effective for high coverage · complements unit testing

Slide 30

Slide 30 text

how it works

Slide 31

Slide 31 text

Arithmetic Mutators

Slide 32

Slide 32 text

Boolean Mutators

Slide 33

Slide 33 text

Comparison Mutators

Slide 34

Slide 34 text

a bit of lingo...

Slide 35

Slide 35 text

Mutation Testing is commencing on 1 files... (.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out) ..M.M... 8 mutations were generated: 6 mutants were killed 0 mutants were not covered by tests 2 covered mutants were not detected 0 fatal errors were encountered 0 time outs were encountered

Slide 36

Slide 36 text

2) \Humbug\Mutator\Number\IntegerValue Diff on \Confoo\PriceCalculator::getPrice() in src/PriceCalculator.php: --- Original +++ New @@ @@ - if ($tickets >= 10 || $hasCoupon) { + if ($tickets >= 11 || $hasCoupon) { $price *= 0.8; } return $price; } }

Slide 37

Slide 37 text

equivalent mutants (false positives) // ORIGINAL CODE // MUTANT $index = 0; $index = 0; while ($index != 10) { while ($index < 10) { // do stuff // do stuff $index++; $index++; } }

Slide 38

Slide 38 text

The metrics

Slide 39

Slide 39 text

Metrics: Mutation Score Indicator (MSI): 75% Mutation Code Coverage: 100% Covered Code MSI: 75% Remember that some mutants will inevitably be harmless (i.e. false positives). Time: 3.74 seconds Memory: 6.00MB Humbug results are being logged as TEXT to: humbuglog.txt

Slide 40

Slide 40 text

Mutation Score Indicator (MSI): 75%

Slide 41

Slide 41 text

Mutation Code Coverage: 100%

Slide 42

Slide 42 text

Covered Code MSI: 75%

Slide 43

Slide 43 text

Installing Humbug $ composer require --dev "humbug/humbug:^1.0@dev" ./composer.json has been updated Loading composer repositories with package information Updating dependencies (including require-dev) - Installing humbug/humbug (dev-master 06b1c05) Cloning 06b1c059e432dab8c22c36bc8b6e1ffc7e587c07 Writing lock file Generating autoload files

Slide 44

Slide 44 text

Configuring Humbug $ vendor/bin/humbug configure _ _ _ | || |_ _ _ __ | |__ _ _ __ _ | __ | || | ' \| '_ \ || / _` | |_||_|\_,_|_|_|_|_.__/\_,_\__, | |___/ Humbug version 1.0-dev Humbug configuration tool. It will guide you through Humbug configuration in few seconds.

Slide 45

Slide 45 text

humbug.json.dist { "source": { "directories": [ "src" ] }, "timeout": 10, "logs": { "text": "humbuglog.txt" } }

Slide 46

Slide 46 text

.gitignore # Humbug config humbug.json # Humbug logs humbuglog.txt humbuglog.json

Slide 47

Slide 47 text

Running Humbug $ vendor/bin/humbug

Slide 48

Slide 48 text

Options and parameters

Slide 49

Slide 49 text

Infinite loop timeout humbug --timeout=50

Slide 50

Slide 50 text

Restricting Files To Mutate humbug --file=NewClass.php --file=*Driver.php

Slide 51

Slide 51 text

Incremental Analysis humbug --incremental

Slide 52

Slide 52 text

The good, the bad and the ugly

Slide 53

Slide 53 text

Program testing can be used to show the presence of bugs, but never to show their absence. — Edsger W. Dijkstra

Slide 54

Slide 54 text

Questions?

Slide 55

Slide 55 text

Let's try it!

Slide 56

Slide 56 text

Let's try it!

Slide 57

Slide 57 text

Another interesting testing tool you should def check out Eris forAll( Generator\choose(0, 1000) ) ->then(function($number) { $this->assertTrue( $number < 42, "$number is not less than 42 apparently" ); }); } }