Who watches the watchmen: testing our tests

The talk was given at JS Experience 2018. https://eventos.imasters.com.br/jsexperience

Transcript

1. “WHO WATCHES THE WATCHMEN” Testing our tests Maria C. Santana JS Experience ‘18
2. Maria Clara • Front-end Developer @ Picter • Live Coding Instructor @ Udacity • Likes dogs and sparkling water
3. Types of software errors • Construction errors; • Specification errors; • Design errors; • Requirements errors; Toward a theory of test data selection, Softech Inc.
4. BLACK-BOX TESTING
5. Input Output
8. •Pro: Allows to check the functionality of a program without worrying with the structure behind it; •Con: Does not offer insights about the tested program’s structure;
9. High code coverage !== Effective Test Suite
10. Source: https://gph.is/1jE2Wzg
13. Take it with a grain of salt.
14. “Quis custodiet ipsos custodes?” Satires, 2a.c.
15. WHITE-BOX TESTING
16. Input Output
19. •Pro: More thorough tests; •Con: Can be complex and require a lot of work due to implementation changes;
20. MUTATION TESTING
21. CONCEPT
22. #1 STEP Test data Original code
23. •If it fails, then the original program is wrong; •If it passes, then we should proceed with creating mutants;
24. #2 STEP Test data N modified versions of the original code
25. •#1: given the test data, the modified versions have different outputs from the original one; •#2: some (or all of them) versions have the same output of the original one;
26. MUTATORS TYPES
27. Not this kind, though…
28. AAR (array for array replacement)
29. ABS (absolute value insertion)
30. AOR (arithmetic operator replacement)
31. UOI (unary operation insertion)
32. Many others… • Array constant replacement; • Logical connector replacement; • Relational operator replacement; • Constant replacement; • Return statement replacement; • Statement deletion; • …
33. EXAMPLE
34. Original program
35. Test suite
36. Mutant #1 - Conditional operator replacement
37. Mutant #1 - Test suite results Mutant was killed!
38. Mutant #2 - Conditional operator replacement
39. Mutant #2 - Test suite results Mutant has survived… ☹
40. EQUIVALENT MUTANTS
42. Mutant #3 - Return statement replacement
43. Mutant #3 - Test suite results Mutant was killed!
44. mutation score = mutants killed / total of mutants; -> 100%
45. TOOLS
46. https://stryker-mutator.io/
47. Trade-offs •Open-source; •Active maintenance and improvements; •Supports most of the mainstream runners (karma, jasmine, jest, …); •Does not support React;
48. Other languages… • Python: https://github.com/mutpy/mutpy; • Ruby: https://github.com/mbj/mutant; • Java: https://github.com/hcoles/pitest;
49. PROS
50. •Application safety; •Evaluation of edge cases; •Better test design;
51. CONS
52. •High computational costs; •Filtering equivalent mutants is still a non-optimal task;
53. Q&A
54. THANKS! olarclara.github.io