Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Who watches the watchmen: testing our tests

Who watches the watchmen: testing our tests

The talk was given at JS Experience 2018. https://eventos.imasters.com.br/jsexperience

Maria Clara Santana

July 05, 2018
Tweet

More Decks by Maria Clara Santana

Other Decks in Programming

Transcript

  1. “WHO WATCHES THE
    WATCHMEN”
    Testing our tests
    Maria C. Santana
    JS Experience ‘18

    View Slide

  2. Maria Clara
    • Front-end Developer @ Picter
    • Live Coding Instructor @ Udacity
    • Likes dogs and sparkling water

    View Slide

  3. Types of software
    errors
    • Construction errors;
    • Specification errors;
    • Design errors;
    • Requirements errors;
    Toward a theory of test data selection, Softech Inc.

    View Slide

  4. BLACK-BOX
    TESTING

    View Slide

  5. Input Output

    View Slide

  6. View Slide

  7. View Slide

  8. •Pro: Allows to check the functionality of a program
    without worrying with the structure behind it;
    •Con: Does not offer insights about the tested program’s
    structure;

    View Slide

  9. High code coverage
    !==
    Effective Test Suite

    View Slide

  10. Source: https://gph.is/1jE2Wzg

    View Slide

  11. View Slide

  12. View Slide

  13. Take it with a grain of salt.

    View Slide

  14. “Quis
    custodiet ipsos
    custodes?”
    Satires, 2a.c.

    View Slide

  15. WHITE-BOX
    TESTING

    View Slide

  16. Input Output

    View Slide

  17. View Slide

  18. View Slide

  19. •Pro: More thorough tests;
    •Con: Can be complex and require a lot of work due to
    implementation changes;

    View Slide

  20. MUTATION
    TESTING

    View Slide

  21. CONCEPT

    View Slide

  22. #1 STEP
    Test data Original code

    View Slide

  23. •If it fails, then the original program is wrong;
    •If it passes, then we should proceed with creating
    mutants;

    View Slide

  24. #2 STEP
    Test data N modified versions
    of the original code

    View Slide

  25. •#1: given the test data, the modified versions have
    different outputs from the original one;
    •#2: some (or all of them) versions have the same output
    of the original one;

    View Slide

  26. MUTATORS TYPES

    View Slide

  27. Not this kind, though…

    View Slide

  28. AAR (array for array replacement)

    View Slide

  29. ABS (absolute value insertion)

    View Slide

  30. AOR (arithmetic operator replacement)

    View Slide

  31. UOI (unary operation insertion)

    View Slide

  32. Many others…
    • Array constant replacement;
    • Logical connector replacement;
    • Relational operator replacement;
    • Constant replacement;
    • Return statement replacement;
    • Statement deletion;
    • …

    View Slide

  33. EXAMPLE

    View Slide

  34. Original program

    View Slide

  35. Test suite

    View Slide

  36. Mutant #1 - Conditional operator replacement

    View Slide

  37. Mutant #1 - Test suite results
    Mutant was killed!

    View Slide

  38. Mutant #2 - Conditional operator replacement

    View Slide

  39. Mutant #2 - Test suite results
    Mutant has survived… ☹

    View Slide

  40. EQUIVALENT
    MUTANTS

    View Slide

  41. View Slide

  42. Mutant #3 - Return statement replacement

    View Slide

  43. Mutant #3 - Test suite results
    Mutant was killed!

    View Slide

  44. mutation score = mutants killed / total of
    mutants; -> 100%

    View Slide

  45. TOOLS

    View Slide

  46. https://stryker-mutator.io/

    View Slide

  47. Trade-offs
    •Open-source;
    •Active maintenance and improvements;
    •Supports most of the mainstream runners (karma,
    jasmine, jest, …);
    •Does not support React;

    View Slide

  48. Other languages…
    • Python: https://github.com/mutpy/mutpy;
    • Ruby: https://github.com/mbj/mutant;
    • Java: https://github.com/hcoles/pitest;

    View Slide

  49. PROS

    View Slide

  50. •Application safety;
    •Evaluation of edge cases;
    •Better test design;

    View Slide

  51. CONS

    View Slide

  52. •High computational costs;
    •Filtering equivalent mutants is still a non-optimal task;

    View Slide

  53. Q&A

    View Slide

  54. THANKS!
    olarclara.github.io

    View Slide