Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Who watches the watchmen: testing our tests

Who watches the watchmen: testing our tests

The talk was given at JS Experience 2018. https://eventos.imasters.com.br/jsexperience

Maria Clara Santana

July 05, 2018
Tweet

More Decks by Maria Clara Santana

Other Decks in Programming

Transcript

  1. “WHO WATCHES THE
    WATCHMEN”
    Testing our tests
    Maria C. Santana
    JS Experience ‘18

    View full-size slide

  2. Maria Clara
    • Front-end Developer @ Picter
    • Live Coding Instructor @ Udacity
    • Likes dogs and sparkling water

    View full-size slide

  3. Types of software
    errors
    • Construction errors;
    • Specification errors;
    • Design errors;
    • Requirements errors;
    Toward a theory of test data selection, Softech Inc.

    View full-size slide

  4. BLACK-BOX
    TESTING

    View full-size slide

  5. Input Output

    View full-size slide

  6. •Pro: Allows to check the functionality of a program
    without worrying with the structure behind it;
    •Con: Does not offer insights about the tested program’s
    structure;

    View full-size slide

  7. High code coverage
    !==
    Effective Test Suite

    View full-size slide

  8. Source: https://gph.is/1jE2Wzg

    View full-size slide

  9. Take it with a grain of salt.

    View full-size slide

  10. “Quis
    custodiet ipsos
    custodes?”
    Satires, 2a.c.

    View full-size slide

  11. WHITE-BOX
    TESTING

    View full-size slide

  12. Input Output

    View full-size slide

  13. •Pro: More thorough tests;
    •Con: Can be complex and require a lot of work due to
    implementation changes;

    View full-size slide

  14. MUTATION
    TESTING

    View full-size slide

  15. #1 STEP
    Test data Original code

    View full-size slide

  16. •If it fails, then the original program is wrong;
    •If it passes, then we should proceed with creating
    mutants;

    View full-size slide

  17. #2 STEP
    Test data N modified versions
    of the original code

    View full-size slide

  18. •#1: given the test data, the modified versions have
    different outputs from the original one;
    •#2: some (or all of them) versions have the same output
    of the original one;

    View full-size slide

  19. MUTATORS TYPES

    View full-size slide

  20. Not this kind, though…

    View full-size slide

  21. AAR (array for array replacement)

    View full-size slide

  22. ABS (absolute value insertion)

    View full-size slide

  23. AOR (arithmetic operator replacement)

    View full-size slide

  24. UOI (unary operation insertion)

    View full-size slide

  25. Many others…
    • Array constant replacement;
    • Logical connector replacement;
    • Relational operator replacement;
    • Constant replacement;
    • Return statement replacement;
    • Statement deletion;
    • …

    View full-size slide

  26. Original program

    View full-size slide

  27. Mutant #1 - Conditional operator replacement

    View full-size slide

  28. Mutant #1 - Test suite results
    Mutant was killed!

    View full-size slide

  29. Mutant #2 - Conditional operator replacement

    View full-size slide

  30. Mutant #2 - Test suite results
    Mutant has survived… ☹

    View full-size slide

  31. EQUIVALENT
    MUTANTS

    View full-size slide

  32. Mutant #3 - Return statement replacement

    View full-size slide

  33. Mutant #3 - Test suite results
    Mutant was killed!

    View full-size slide

  34. mutation score = mutants killed / total of
    mutants; -> 100%

    View full-size slide

  35. https://stryker-mutator.io/

    View full-size slide

  36. Trade-offs
    •Open-source;
    •Active maintenance and improvements;
    •Supports most of the mainstream runners (karma,
    jasmine, jest, …);
    •Does not support React;

    View full-size slide

  37. Other languages…
    • Python: https://github.com/mutpy/mutpy;
    • Ruby: https://github.com/mbj/mutant;
    • Java: https://github.com/hcoles/pitest;

    View full-size slide

  38. •Application safety;
    •Evaluation of edge cases;
    •Better test design;

    View full-size slide

  39. •High computational costs;
    •Filtering equivalent mutants is still a non-optimal task;

    View full-size slide

  40. THANKS!
    olarclara.github.io

    View full-size slide