If You Can't Kill a Supermutant, You Have a Problem

Mutation Workshop 2018

Transcript

1. 9. April 2018 Rahul Gopinath, Björn Mathis and Andreas Zeller If You Can't Kill a Supermutant You Have a Problem 1
2. Mutant Kill Matrix 2
3. Mutant Kill Matrix 2
4. Mutant Kill Matrix 2
5. Mutant Kill Matrix 2
6. Mutant Kill Matrix 2
7. Mutant Kill Matrix 2
8. Mutant Kill Matrix 2
9. Mutant Kill Matrix 2
10. Mutant Kill Matrix 2 Enormous Amount of Test Executions
11. Supermutants - Coverage Adequate Test Suite 3
12. Supermutants - Coverage Adequate Test Suite 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 3
13. Supermutants - Coverage Adequate Test Suite 0 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 93 % 0 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 93 % 3
14. Supermutants - Coverage Adequate Test Suite 1 Coverage Adequate Test 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 70 % 93 % 1 Coverage Adequate Test 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 70 % 93 % 3
15. Supermutants - Coverage Adequate Test Suite 2 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 93 % 93 % 2 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 93 % 93 % 3
16. Supermutants - Coverage Adequate Test Suite 4
17. Supermutants - Coverage Adequate Test Suite 4
18. Supermutants - Coverage Adequate Test Suite 4
19. Supermutants - Coverage Adequate Test Suite 4
20. Supermutant Execution 5
21. Supermutant Execution 5 7 Tests
22. Supermutant Execution 5 7 Tests
23. Supermutant Execution 5 7 Tests
24. Supermutant Execution 5 7 Tests
25. Supermutant Execution 5 3 Tests 3 Tests 7 Tests
26. Supermutant Execution 5 3 Tests 3 Tests 7 Tests
27. Supermutant Execution 5 3 Tests 3 Tests 7 Tests
28. Supermutant Execution 5 3 Tests 3 Tests 7 Tests
29. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3 Tests 7 Tests
30. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3 Tests 7 Tests
31. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3 Tests 7 Tests
32. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test 7 Tests
33. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test 7 Tests
34. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test 7 Tests 34 vs 42 Test Executions
35. Supermutants - Mutant Kill Matrix 6 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test
36. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test
37. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test
38. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test
39. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test
40. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test
41. Case Study - Urlparse 7
42. Case Study - Urlparse 7 0 exec. 10000 exec. 20000 exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite
43. Case Study - Urlparse 7 0 exec. 10000 exec. 20000 exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 32,000
44. Case Study - Urlparse 7 0 exec. 10000 exec. 20000 exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 13,009 32,000
45. Case Study - Urlparse 7 0 exec. 10000 exec. 20000 exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 9,037 13,009 32,000
46. Case Study - Urlparse 7 0 exec. 10000 exec. 20000 exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 9,037 13,009 32,000 Over 70% of Test Executions Are Saved
47. When Are Supermutants Super? 8
48. When Are Supermutants Super? 8 Many and Weak Tests
49. When Are Supermutants Super? 8 Many and Weak Tests Stubborn Mutants
50. When Are Supermutants Super? 8 Many and Weak Tests Stubborn Mutants No Fault Masking
51. When Are Supermutants Super? 8 Many and Weak Tests Stubborn Mutants No Fault Masking Automatically Generated Test Suites Can Be Efficiently Analyzed
52. 9 Summary
53. 9 Summary
54. 9 Summary
55. 9 Summary
56. 9 Summary