If You Can't Kill a Supermutant, You Have a Problem

Mutation Workshop 2018

Transcript

1. 9. April 2018
   Rahul Gopinath, Björn Mathis and Andreas Zeller
   If You Can't Kill a Supermutant You Have a
   Problem
   1
   

   View Slide

2. Mutant Kill Matrix
   2
   

   View Slide

3. Mutant Kill Matrix
   2
   

   View Slide

4. Mutant Kill Matrix
   2
   

   View Slide

5. Mutant Kill Matrix
   2
   

   View Slide

6. Mutant Kill Matrix
   2
   

   View Slide

7. Mutant Kill Matrix
   2
   

   View Slide

8. Mutant Kill Matrix
   2
   

   View Slide

9. Mutant Kill Matrix
   2
   

   View Slide

10. Mutant Kill Matrix
    2
    Enormous Amount of Test Executions
    

    View Slide

11. Supermutants - Coverage Adequate Test Suite
    3
    

    View Slide

12. Supermutants - Coverage Adequate Test Suite
    0 %
    25 %
    50 %
    75 %
    100 %
    Full Suite (7 Tests) Coverage Adequate
    0 %
    25 %
    50 %
    75 %
    100 %
    Full Suite (7 Tests) Coverage Adequate
    3
    

    View Slide

13. Supermutants - Coverage Adequate Test Suite
    0 Coverage Adequate Tests
    0 %
    25 %
    50 %
    75 %
    100 %
    Full Suite (7 Tests) Coverage Adequate
    0 %
    93 %
    0 Coverage Adequate Tests
    0 %
    25 %
    50 %
    75 %
    100 %
    Full Suite (7 Tests) Coverage Adequate
    0 %
    93 %
    3
    

    View Slide

14. Supermutants - Coverage Adequate Test Suite
    1 Coverage Adequate Test
    0 %
    25 %
    50 %
    75 %
    100 %
    Full Suite (7 Tests) Coverage Adequate
    70 %
    93 %
    1 Coverage Adequate Test
    0 %
    25 %
    50 %
    75 %
    100 %
    Full Suite (7 Tests) Coverage Adequate
    70 %
    93 %
    3
    

    View Slide

15. Supermutants - Coverage Adequate Test Suite
    2 Coverage Adequate Tests
    0 %
    25 %
    50 %
    75 %
    100 %
    Full Suite (7 Tests) Coverage Adequate
    93 %
    93 %
    2 Coverage Adequate Tests
    0 %
    25 %
    50 %
    75 %
    100 %
    Full Suite (7 Tests) Coverage Adequate
    93 %
    93 %
    3
    

    View Slide

16. Supermutants - Coverage Adequate Test Suite
    4
    

    View Slide

17. Supermutants - Coverage Adequate Test Suite
    4
    

    View Slide

18. Supermutants - Coverage Adequate Test Suite
    4
    

    View Slide

19. Supermutants - Coverage Adequate Test Suite
    4
    

    View Slide

20. Supermutant Execution
    5
    

    View Slide

21. Supermutant Execution
    5
    7 Tests
    

    View Slide

22. Supermutant Execution
    5
    7 Tests
    

    View Slide

23. Supermutant Execution
    5
    7 Tests
    

    View Slide

24. Supermutant Execution
    5
    7 Tests
    

    View Slide

25. Supermutant Execution
    5
    3 Tests 3 Tests
    7 Tests
    

    View Slide

26. Supermutant Execution
    5
    3 Tests 3 Tests
    7 Tests
    

    View Slide

27. Supermutant Execution
    5
    3 Tests 3 Tests
    7 Tests
    

    View Slide

28. Supermutant Execution
    5
    3 Tests 3 Tests
    7 Tests
    

    View Slide

29. Supermutant Execution
    5
    3 Tests 3 Tests
    3 Tests
    3 Tests
    7 Tests
    

    View Slide

30. Supermutant Execution
    5
    3 Tests 3 Tests
    3 Tests
    3 Tests
    7 Tests
    

    View Slide

31. Supermutant Execution
    5
    3 Tests 3 Tests
    3 Tests
    3 Tests
    7 Tests
    

    View Slide

32. Supermutant Execution
    5
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test
    7 Tests
    

    View Slide

33. Supermutant Execution
    5
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test
    7 Tests
    

    View Slide

34. Supermutant Execution
    5
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test
    7 Tests
    34 vs 42 Test Executions
    

    View Slide

35. Supermutants - Mutant Kill Matrix
    6
    7 Tests
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test
    

    View Slide

36. Supermutants - Mutant Kill Matrix
    6
    Test 1
    Test 2
    Test 3
    Test 4
    Test 5
    Test 6
    Test 7
    7 Tests
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test
    

    View Slide

37. Supermutants - Mutant Kill Matrix
    6
    Test 1
    Test 2
    Test 3
    Test 4
    Test 5
    Test 6
    Test 7
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    7 Tests
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test
    

    View Slide

38. Supermutants - Mutant Kill Matrix
    6
    Test 1
    Test 2
    Test 3
    Test 4
    Test 5
    Test 6
    Test 7
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✓
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    7 Tests
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test
    

    View Slide

39. Supermutants - Mutant Kill Matrix
    6
    Test 1
    Test 2
    Test 3
    Test 4
    Test 5
    Test 6
    Test 7
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✓
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✓
    ✕
    ✕
    ✓
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✓
    ✓
    ✓
    ✓
    ✓
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    7 Tests
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test
    

    View Slide

40. Supermutants - Mutant Kill Matrix
    6
    Test 1
    Test 2
    Test 3
    Test 4
    Test 5
    Test 6
    Test 7
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✓
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✓
    ✕
    ✕
    ✓
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✓
    ✓
    ✓
    ✓
    ✓
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    ✕
    7 Tests
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test
    

    View Slide

41. Case Study - Urlparse
    7
    

    View Slide

42. Case Study - Urlparse
    7
    0 exec.
    10000 exec.
    20000 exec.
    30000 exec.
    40000 exec.
    Full Matrix Strong Testsuite Weak Testsuite
    

    View Slide

43. Case Study - Urlparse
    7
    0 exec.
    10000 exec.
    20000 exec.
    30000 exec.
    40000 exec.
    Full Matrix Strong Testsuite Weak Testsuite
    32,000
    

    View Slide

44. Case Study - Urlparse
    7
    0 exec.
    10000 exec.
    20000 exec.
    30000 exec.
    40000 exec.
    Full Matrix Strong Testsuite Weak Testsuite
    13,009
    32,000
    

    View Slide

45. Case Study - Urlparse
    7
    0 exec.
    10000 exec.
    20000 exec.
    30000 exec.
    40000 exec.
    Full Matrix Strong Testsuite Weak Testsuite
    9,037
    13,009
    32,000
    

    View Slide

46. Case Study - Urlparse
    7
    0 exec.
    10000 exec.
    20000 exec.
    30000 exec.
    40000 exec.
    Full Matrix Strong Testsuite Weak Testsuite
    9,037
    13,009
    32,000
    Over 70% of Test Executions Are Saved
    

    View Slide

47. When Are Supermutants Super?
    8
    

    View Slide

48. When Are Supermutants Super?
    8
    Many and Weak Tests
    

    View Slide

49. When Are Supermutants Super?
    8
    Many and Weak Tests Stubborn Mutants
    

    View Slide

50. When Are Supermutants Super?
    8
    Many and Weak Tests Stubborn Mutants No Fault Masking
    

    View Slide

51. When Are Supermutants Super?
    8
    Many and Weak Tests Stubborn Mutants No Fault Masking
    Automatically Generated Test Suites
    Can Be Efficiently Analyzed
    

    View Slide

52. 9
    Summary
    

    View Slide

53. 9
    Summary
    

    View Slide

54. 9
    Summary
    

    View Slide

55. 9
    Summary
    

    View Slide

56. 9
    Summary
    

    View Slide