Who watches the watchmen: testing our tests

Transcript

1. 1.

   “WHO WATCHES THE WATCHMEN” Testing our tests Maria C. Santana

   JS Experience ‘18
2. 2.

   Maria Clara • Front-end Developer @ Picter • Live Coding

   Instructor @ Udacity • Likes dogs and sparkling water
3. 3.

   Types of software errors • Construction errors; • Specification errors;

   • Design errors; • Requirements errors; Toward a theory of test data selection, Softech Inc.
4. 4.

   BLACK-BOX TESTING

5. 5.

   Input Output

6. 6.

   None
7. 7.

   None
8. 8.

   •Pro: Allows to check the functionality of a program without

   worrying with the structure behind it; •Con: Does not offer insights about the tested program’s structure;
9. 9.

   High code coverage !== Effective Test Suite

10. 10.

    Source: https://gph.is/1jE2Wzg

11. 11.

    None
12. 12.

    None
13. 13.

    Take it with a grain of salt.

14. 14.

    “Quis custodiet ipsos custodes?” Satires, 2a.c.

15. 15.

    WHITE-BOX TESTING

16. 16.

    Input Output

17. 17.

    None
18. 18.

    None
19. 19.

    •Pro: More thorough tests; •Con: Can be complex and require

    a lot of work due to implementation changes;
20. 20.

    MUTATION TESTING

21. 21.

    CONCEPT

22. 22.

    #1 STEP Test data Original code

23. 23.

    •If it fails, then the original program is wrong; •If

    it passes, then we should proceed with creating mutants;
24. 24.

    #2 STEP Test data N modified versions of the original

    code
25. 25.

    •#1: given the test data, the modified versions have different

    outputs from the original one; •#2: some (or all of them) versions have the same output of the original one;
26. 26.

    MUTATORS TYPES

27. 27.

    Not this kind, though…

28. 28.

    AAR (array for array replacement)

29. 29.

    ABS (absolute value insertion)

30. 30.

    AOR (arithmetic operator replacement)

31. 31.

    UOI (unary operation insertion)

32. 32.

    Many others… • Array constant replacement; • Logical connector replacement;

    • Relational operator replacement; • Constant replacement; • Return statement replacement; • Statement deletion; • …
33. 33.

    EXAMPLE

34. 34.

    Original program

35. 35.

    Test suite

36. 36.

    Mutant #1 - Conditional operator replacement

37. 37.

    Mutant #1 - Test suite results Mutant was killed!

38. 38.

    Mutant #2 - Conditional operator replacement

39. 39.

    Mutant #2 - Test suite results Mutant has survived… ☹

40. 40.

    EQUIVALENT MUTANTS

41. 41.

    None
42. 42.

    Mutant #3 - Return statement replacement

43. 43.

    Mutant #3 - Test suite results Mutant was killed!

44. 44.

    mutation score = mutants killed / total of mutants; ->

    100%
45. 45.

    TOOLS

46. 46.

    https://stryker-mutator.io/

47. 47.

    Trade-offs •Open-source; •Active maintenance and improvements; •Supports most of the

    mainstream runners (karma, jasmine, jest, …); •Does not support React;
48. 48.

    Other languages… • Python: https://github.com/mutpy/mutpy; • Ruby: https://github.com/mbj/mutant; • Java:

    https://github.com/hcoles/pitest;
49. 49.

    PROS

50. 50.

    •Application safety; •Evaluation of edge cases; •Better test design;

51. 51.

    CONS

52. 52.

    •High computational costs; •Filtering equivalent mutants is still a non-optimal

    task;
53. 53.

    Q&A

54. 54.

    THANKS! olarclara.github.io