Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Who watches the watchmen: testing our tests

Who watches the watchmen: testing our tests

The talk was given at JS Experience 2018. https://eventos.imasters.com.br/jsexperience

Maria Clara Santana

July 05, 2018
Tweet

More Decks by Maria Clara Santana

Other Decks in Programming

Transcript

  1. “WHO WATCHES THE WATCHMEN” Testing our tests Maria C. Santana

    JS Experience ‘18
  2. Maria Clara • Front-end Developer @ Picter • Live Coding

    Instructor @ Udacity • Likes dogs and sparkling water
  3. Types of software errors • Construction errors; • Specification errors;

    • Design errors; • Requirements errors; Toward a theory of test data selection, Softech Inc.
  4. BLACK-BOX TESTING

  5. Input Output

  6. None
  7. None
  8. •Pro: Allows to check the functionality of a program without

    worrying with the structure behind it; •Con: Does not offer insights about the tested program’s structure;
  9. High code coverage !== Effective Test Suite

  10. Source: https://gph.is/1jE2Wzg

  11. None
  12. None
  13. Take it with a grain of salt.

  14. “Quis custodiet ipsos custodes?” Satires, 2a.c.

  15. WHITE-BOX TESTING

  16. Input Output

  17. None
  18. None
  19. •Pro: More thorough tests; •Con: Can be complex and require

    a lot of work due to implementation changes;
  20. MUTATION TESTING

  21. CONCEPT

  22. #1 STEP Test data Original code

  23. •If it fails, then the original program is wrong; •If

    it passes, then we should proceed with creating mutants;
  24. #2 STEP Test data N modified versions of the original

    code
  25. •#1: given the test data, the modified versions have different

    outputs from the original one; •#2: some (or all of them) versions have the same output of the original one;
  26. MUTATORS TYPES

  27. Not this kind, though…

  28. AAR (array for array replacement)

  29. ABS (absolute value insertion)

  30. AOR (arithmetic operator replacement)

  31. UOI (unary operation insertion)

  32. Many others… • Array constant replacement; • Logical connector replacement;

    • Relational operator replacement; • Constant replacement; • Return statement replacement; • Statement deletion; • …
  33. EXAMPLE

  34. Original program

  35. Test suite

  36. Mutant #1 - Conditional operator replacement

  37. Mutant #1 - Test suite results Mutant was killed!

  38. Mutant #2 - Conditional operator replacement

  39. Mutant #2 - Test suite results Mutant has survived… ☹

  40. EQUIVALENT MUTANTS

  41. None
  42. Mutant #3 - Return statement replacement

  43. Mutant #3 - Test suite results Mutant was killed!

  44. mutation score = mutants killed / total of mutants; ->

    100%
  45. TOOLS

  46. https://stryker-mutator.io/

  47. Trade-offs •Open-source; •Active maintenance and improvements; •Supports most of the

    mainstream runners (karma, jasmine, jest, …); •Does not support React;
  48. Other languages… • Python: https://github.com/mutpy/mutpy; • Ruby: https://github.com/mbj/mutant; • Java:

    https://github.com/hcoles/pitest;
  49. PROS

  50. •Application safety; •Evaluation of edge cases; •Better test design;

  51. CONS

  52. •High computational costs; •Filtering equivalent mutants is still a non-optimal

    task;
  53. Q&A

  54. THANKS! olarclara.github.io