Mutation testing with Humbug - Confoo Montréal 2016

Satisfied with your unit test code coverage? Are you sure that your code is thoroughly tested, and not merely executed by the tests? Learn what mutation testing is, and how Humbug can help you give your test suite a run for its money. Humbug is a tool that injects defects and regressions in your code and then checks if your tests noticed.

Transcript

1. MUTATION TESTING
   WITH HUMBUG
   @CONFOO 2016
   

   View Slide

2. $ whoami
   

   View Slide

3. WEB DEVELOPER @ DUPROPRIO
   MAAUBE / MARCAUBE
   HTTP://MARCAUBE.CA
   

   View Slide

4. View Slide

5. UNIT TESTING
   

   View Slide

6. UNIT TESTING
   > prove the code works
   

   View Slide

7. function sum($a, $b)
   {
   return $a + $b;
   }
   $this->assertEquals(3, sum(1, 2));
   

   View Slide

8. $number = sum(1, 2.5));
   // 3.5?
   // 3?
   // 4?
   

   View Slide

9. $number = sum(-42, 10);
   // probably -32, but...
   

   View Slide

10. $number = sum(2**63, 2**63);
    // float(1.844674407371E+19)?
    // int(0)?
    

    View Slide

11. UNIT TESTING
    > prove the code works
    > catch regressions
    

    View Slide

12. UNIT TESTING
    > prove the code works
    > catch regressions
    > drive away from bad design
    

    View Slide

13. UNIT TESTING
    > prove the code works
    > catch regressions
    > drive away from bad design
    > short feedback loop
    

    View Slide

14. UNIT TESTING
    > prove the code works
    > catch regressions
    > drive away from bad design
    > short feedback loop
    > mature tooling
    

    View Slide

15. THE CODE COVERAGE
    FALLACY
    

    View Slide

16. public function getPrice($tickets = 1, $hasCoupon = false)
    {
    $price = $tickets * 945.00;
    if ($tickets >= 10 || $hasCoupon) {
    $price *= 0.8;
    }
    return $price;
    }
    

    View Slide

17. public function testCanCalculateOrdersWithRegularPrice()
    {
    $this->assertEquals(945.00, $calculator->getPrice(1));
    }
    public function testCanCalculateOrdersWithDiscountedPrice()
    {
    $this->assertEquals(756.00, $calculator->getPrice(1, true));
    }
    

    View Slide

18. 100% COVERAGE!
    

    View Slide

19. SUCCESS: 26/26 (100%)
    

    View Slide

20. 100% LINE COVERAGE
    

    View Slide

21. public function getPrice($tickets = 1, $hasCoupon = false)
    {
    $price = $tickets * 945.00;
    if ($tickets >= 10 || $hasCoupon) {
    $price *= 0.8;
    }
    return $price;
    }
    

    View Slide

22. public function testCanCalculateThePriceOfARegularOrder()
    {
    $this->assertEquals(945.00, $calculator->getPrice());
    }
    public function testAppliesADiscountForOrdersOfTenTicketsOrMore()
    {
    $this->assertEquals(11340.00, $calculator->getPrice(15));
    }
    public function testAppliesADiscountForOrdersWithCoupon()
    {
    $this->assertEquals(756.00, $calculator->getPrice(1, true));
    }
    

    View Slide

23. MUTATION TESTING
    

    View Slide

24. Code has bugs,
    Tests are code,
    tests have bugs.
    — Abraham Lincoln
    

    View Slide

25. MUTATION TESTING
    > identify weaknesses in unit tests
    

    View Slide

26. MUTATION TESTING
    > identify weaknesses in unit tests
    > most effective for high coverage
    

    View Slide

27. MUTATION TESTING
    > identify weaknesses in unit tests
    > most effective for high coverage
    > complements unit testing
    

    View Slide

28. HOW IT WORKS
    

    View Slide

29. ARITHMETIC
    MUTATORS
    

    View Slide

30. BOOLEAN MUTATORS
    

    View Slide

31. COMPARISON
    MUTATORS
    

    View Slide

32. A BIT OF LINGO...
    

    View Slide

33. Mutation Testing is commencing on 1 files...
    (.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out)
    ..M.M...
    8 mutations were generated:
    6 mutants were killed
    0 mutants were not covered by tests
    2 covered mutants were not detected
    0 fatal errors were encountered
    0 time outs were encountered
    

    View Slide

34. 2) \Humbug\Mutator\Number\IntegerValue
    Diff on \Confoo\PriceCalculator::getPrice() in src/PriceCalculator.php:
    --- Original
    +++ New
    @@ @@
    - if ($tickets >= 10 || $hasCoupon) {
    + if ($tickets >= 11 || $hasCoupon) {
    $price *= 0.8;
    }
    return $price;
    }
    }
    

    View Slide

35. EQUIVALENT MUTANTS
    (FALSE POSITIVES)
    // ORIGINAL CODE // MUTANT
    $index = 0; $index = 0;
    while ($index != 10) { while ($index < 10) {
    // do stuff // do stuff
    $index++; $index++;
    } }
    

    View Slide

36. THE METRICS
    Mutation Score Indicator (MSI): 75%
    Mutation Code Coverage: 100%
    Covered Code MSI: 75%
    

    View Slide

37. INSTALLING HUMBUG
    $ composer require --dev "humbug/humbug:~1.0@dev"
    ./composer.json has been updated
    Loading composer repositories with package information
    Updating dependencies (including require-dev)
    - Installing humbug/humbug (dev-master d80edfc)
    Cloning d80edfcb3e14cd3afce4b0cf8169565af33e1657
    Writing lock file
    Generating autoload files
    

    View Slide

38. CONFIGURING HUMBUG
    $ vendor/bin/humbug configure
    _ _ _
    | || |_ _ _ __ | |__ _ _ __ _
    | __ | || | ' \| '_ \ || / _` |
    |_||_|\_,_|_|_|_|_.__/\_,_\__, |
    |___/
    Humbug version 1.0-dev
    Humbug configuration tool.
    It will guide you through Humbug configuration in few seconds.
    

    View Slide

39. HUMBUG.JSON.DIST
    {
    "source": {
    "directories": [
    "src"
    ]
    },
    "timeout": 10,
    "logs": {
    "text": "humbuglog.txt"
    }
    }
    

    View Slide

40. .GITIGNORE
    # Humbug config
    humbug.json
    # Humbug logs
    humbuglog.txt
    humbuglog.json
    

    View Slide

41. THE GOOD, THE BAD AND
    THE UGLY
    

    View Slide

42. {
    "require": {
    "php": ">=5.4.0",
    "phpunit/phpunit": "^4.5|^5.0"
    }
    }
    

    View Slide

43. IT IS SLOW
    

    View Slide

44. Program testing can be used to
    show the presence of bugs, but
    never to show their absence.
    — Edsger W. Dijkstra
    

    View Slide

45. THANK YOU!
    

    View Slide

46. QUESTIONS?
    

    View Slide

47. https://github.com/padraic/humbug
    

    View Slide