Who watches the watchmen: testing our tests

The talk was given at JS Experience 2018. https://eventos.imasters.com.br/jsexperience

Transcript

1. “WHO WATCHES THE
   WATCHMEN”
   Testing our tests
   Maria C. Santana
   JS Experience ‘18
   

   View Slide

2. Maria Clara
   • Front-end Developer @ Picter
   • Live Coding Instructor @ Udacity
   • Likes dogs and sparkling water
   

   View Slide

3. Types of software
   errors
   • Construction errors;
   • Specification errors;
   • Design errors;
   • Requirements errors;
   Toward a theory of test data selection, Softech Inc.
   

   View Slide

4. BLACK-BOX
   TESTING
   

   View Slide

5. Input Output
   

   View Slide

6. View Slide

7. View Slide

8. •Pro: Allows to check the functionality of a program
   without worrying with the structure behind it;
   •Con: Does not offer insights about the tested program’s
   structure;
   

   View Slide

9. High code coverage
   !==
   Effective Test Suite
   

   View Slide

10. Source: https://gph.is/1jE2Wzg
    

    View Slide

11. View Slide

12. View Slide

13. Take it with a grain of salt.
    

    View Slide

14. “Quis
    custodiet ipsos
    custodes?”
    Satires, 2a.c.
    

    View Slide

15. WHITE-BOX
    TESTING
    

    View Slide

16. Input Output
    

    View Slide

17. View Slide

18. View Slide

19. •Pro: More thorough tests;
    •Con: Can be complex and require a lot of work due to
    implementation changes;
    

    View Slide

20. MUTATION
    TESTING
    

    View Slide

21. CONCEPT
    

    View Slide

22. #1 STEP
    Test data Original code
    

    View Slide

23. •If it fails, then the original program is wrong;
    •If it passes, then we should proceed with creating
    mutants;
    

    View Slide

24. #2 STEP
    Test data N modified versions
    of the original code
    

    View Slide

25. •#1: given the test data, the modified versions have
    different outputs from the original one;
    •#2: some (or all of them) versions have the same output
    of the original one;
    

    View Slide

26. MUTATORS TYPES
    

    View Slide

27. Not this kind, though…
    

    View Slide

28. AAR (array for array replacement)
    

    View Slide

29. ABS (absolute value insertion)
    

    View Slide

30. AOR (arithmetic operator replacement)
    

    View Slide

31. UOI (unary operation insertion)
    

    View Slide

32. Many others…
    • Array constant replacement;
    • Logical connector replacement;
    • Relational operator replacement;
    • Constant replacement;
    • Return statement replacement;
    • Statement deletion;
    • …
    

    View Slide

33. EXAMPLE
    

    View Slide

34. Original program
    

    View Slide

35. Test suite
    

    View Slide

36. Mutant #1 - Conditional operator replacement
    

    View Slide

37. Mutant #1 - Test suite results
    Mutant was killed!
    

    View Slide

38. Mutant #2 - Conditional operator replacement
    

    View Slide

39. Mutant #2 - Test suite results
    Mutant has survived… ☹
    

    View Slide

40. EQUIVALENT
    MUTANTS
    

    View Slide

41. View Slide

42. Mutant #3 - Return statement replacement
    

    View Slide

43. Mutant #3 - Test suite results
    Mutant was killed!
    

    View Slide

44. mutation score = mutants killed / total of
    mutants; -> 100%
    

    View Slide

45. TOOLS
    

    View Slide

46. https://stryker-mutator.io/
    

    View Slide

47. Trade-offs
    •Open-source;
    •Active maintenance and improvements;
    •Supports most of the mainstream runners (karma,
    jasmine, jest, …);
    •Does not support React;
    

    View Slide

48. Other languages…
    • Python: https://github.com/mutpy/mutpy;
    • Ruby: https://github.com/mbj/mutant;
    • Java: https://github.com/hcoles/pitest;
    

    View Slide

49. PROS
    

    View Slide

50. •Application safety;
    •Evaluation of edge cases;
    •Better test design;
    

    View Slide

51. CONS
    

    View Slide

52. •High computational costs;
    •Filtering equivalent mutants is still a non-optimal task;
    

    View Slide

53. Q&A
    

    View Slide

54. THANKS!
    olarclara.github.io
    

    View Slide