Marc Aubé
February 26, 2016
110

# Mutation testing with Humbug - Confoo Montréal 2016

Satisfied with your unit test code coverage? Are you sure that your code is thoroughly tested, and not merely executed by the tests? Learn what mutation testing is, and how Humbug can help you give your test suite a run for its money. Humbug is a tool that injects defects and regressions in your code and then checks if your tests noticed.

## Marc Aubé

February 26, 2016

## Transcript

1. MUTATION TESTING
WITH HUMBUG
@CONFOO 2016

2. \$ whoami

3. WEB DEVELOPER @ DUPROPRIO
MAAUBE / MARCAUBE
HTTP://MARCAUBE.CA

4. UNIT TESTING

5. UNIT TESTING
> prove the code works

6. function sum(\$a, \$b)
{
return \$a + \$b;
}
\$this->assertEquals(3, sum(1, 2));

7. \$number = sum(1, 2.5));
// 3.5?
// 3?
// 4?

8. \$number = sum(-42, 10);
// probably -32, but...

9. \$number = sum(2**63, 2**63);
// float(1.844674407371E+19)?
// int(0)?

10. UNIT TESTING
> prove the code works
> catch regressions

11. UNIT TESTING
> prove the code works
> catch regressions
> drive away from bad design

12. UNIT TESTING
> prove the code works
> catch regressions
> drive away from bad design
> short feedback loop

13. UNIT TESTING
> prove the code works
> catch regressions
> drive away from bad design
> short feedback loop
> mature tooling

14. THE CODE COVERAGE
FALLACY

15. public function getPrice(\$tickets = 1, \$hasCoupon = false)
{
\$price = \$tickets * 945.00;
if (\$tickets >= 10 || \$hasCoupon) {
\$price *= 0.8;
}
return \$price;
}

16. public function testCanCalculateOrdersWithRegularPrice()
{
\$this->assertEquals(945.00, \$calculator->getPrice(1));
}
public function testCanCalculateOrdersWithDiscountedPrice()
{
\$this->assertEquals(756.00, \$calculator->getPrice(1, true));
}

17. 100% COVERAGE!

18. SUCCESS: 26/26 (100%)

19. 100% LINE COVERAGE

20. public function getPrice(\$tickets = 1, \$hasCoupon = false)
{
\$price = \$tickets * 945.00;
if (\$tickets >= 10 || \$hasCoupon) {
\$price *= 0.8;
}
return \$price;
}

21. public function testCanCalculateThePriceOfARegularOrder()
{
\$this->assertEquals(945.00, \$calculator->getPrice());
}
{
\$this->assertEquals(11340.00, \$calculator->getPrice(15));
}
{
\$this->assertEquals(756.00, \$calculator->getPrice(1, true));
}

22. MUTATION TESTING

23. Code has bugs,
Tests are code,
tests have bugs.
— Abraham Lincoln

24. MUTATION TESTING
> identify weaknesses in unit tests

25. MUTATION TESTING
> identify weaknesses in unit tests
> most effective for high coverage

26. MUTATION TESTING
> identify weaknesses in unit tests
> most effective for high coverage
> complements unit testing

27. HOW IT WORKS

28. ARITHMETIC
MUTATORS

29. BOOLEAN MUTATORS

30. COMPARISON
MUTATORS

31. A BIT OF LINGO...

32. Mutation Testing is commencing on 1 files...
(.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out)
..M.M...
8 mutations were generated:
6 mutants were killed
0 mutants were not covered by tests
2 covered mutants were not detected
0 fatal errors were encountered
0 time outs were encountered

33. 2) \Humbug\Mutator\Number\IntegerValue
Diff on \Confoo\PriceCalculator::getPrice() in src/PriceCalculator.php:
--- Original
+++ New
@@ @@
- if (\$tickets >= 10 || \$hasCoupon) {
+ if (\$tickets >= 11 || \$hasCoupon) {
\$price *= 0.8;
}
return \$price;
}
}

34. EQUIVALENT MUTANTS
(FALSE POSITIVES)
// ORIGINAL CODE // MUTANT
\$index = 0; \$index = 0;
while (\$index != 10) { while (\$index < 10) {
// do stuff // do stuff
\$index++; \$index++;
} }

35. THE METRICS
Mutation Score Indicator (MSI): 75%
Mutation Code Coverage: 100%
Covered Code MSI: 75%

36. INSTALLING HUMBUG
\$ composer require --dev "humbug/humbug:~1.0@dev"
./composer.json has been updated
Updating dependencies (including require-dev)
- Installing humbug/humbug (dev-master d80edfc)
Cloning d80edfcb3e14cd3afce4b0cf8169565af33e1657
Writing lock file

37. CONFIGURING HUMBUG
\$ vendor/bin/humbug configure
_ _ _
| || |_ _ _ __ | |__ _ _ __ _
| __ | || | ' \| '_ \ || / _` |
|_||_|\_,_|_|_|_|_.__/\_,_\__, |
|___/
Humbug version 1.0-dev
Humbug configuration tool.
It will guide you through Humbug configuration in few seconds.

38. HUMBUG.JSON.DIST
{
"source": {
"directories": [
"src"
]
},
"timeout": 10,
"logs": {
"text": "humbuglog.txt"
}
}

39. .GITIGNORE
# Humbug config
humbug.json
# Humbug logs
humbuglog.txt
humbuglog.json

40. THE GOOD, THE BAD AND
THE UGLY

41. {
"require": {
"php": ">=5.4.0",
"phpunit/phpunit": "^4.5|^5.0"
}
}

42. IT IS SLOW

43. Program testing can be used to
show the presence of bugs, but
never to show their absence.
— Edsger W. Dijkstra

44. THANK YOU!

45. QUESTIONS?