Upgrade to Pro — share decks privately, control downloads, hide ads and more …

If You Can't Kill a Supermutant, You Have a Problem

If You Can't Kill a Supermutant, You Have a Problem

Mutation Workshop 2018

Rahul Gopinath

April 09, 2018
Tweet

More Decks by Rahul Gopinath

Other Decks in Research

Transcript

  1. 9. April 2018
    Rahul Gopinath, Björn Mathis and Andreas Zeller
    If You Can't Kill a Supermutant You Have a
    Problem
    1

    View Slide

  2. Mutant Kill Matrix
    2

    View Slide

  3. Mutant Kill Matrix
    2

    View Slide

  4. Mutant Kill Matrix
    2

    View Slide

  5. Mutant Kill Matrix
    2

    View Slide

  6. Mutant Kill Matrix
    2

    View Slide

  7. Mutant Kill Matrix
    2

    View Slide

  8. Mutant Kill Matrix
    2

    View Slide

  9. Mutant Kill Matrix
    2

    View Slide

  10. Mutant Kill Matrix
    2
    Enormous Amount of Test Executions

    View Slide

  11. Supermutants - Coverage Adequate Test Suite
    3

    View Slide

  12. Supermutants - Coverage Adequate Test Suite
    0 %
    25 %
    50 %
    75 %
    100 %
    Full Suite (7 Tests) Coverage Adequate
    0 %
    25 %
    50 %
    75 %
    100 %
    Full Suite (7 Tests) Coverage Adequate
    3

    View Slide

  13. Supermutants - Coverage Adequate Test Suite
    0 Coverage Adequate Tests
    0 %
    25 %
    50 %
    75 %
    100 %
    Full Suite (7 Tests) Coverage Adequate
    0 %
    93 %
    0 Coverage Adequate Tests
    0 %
    25 %
    50 %
    75 %
    100 %
    Full Suite (7 Tests) Coverage Adequate
    0 %
    93 %
    3

    View Slide

  14. Supermutants - Coverage Adequate Test Suite
    1 Coverage Adequate Test
    0 %
    25 %
    50 %
    75 %
    100 %
    Full Suite (7 Tests) Coverage Adequate
    70 %
    93 %
    1 Coverage Adequate Test
    0 %
    25 %
    50 %
    75 %
    100 %
    Full Suite (7 Tests) Coverage Adequate
    70 %
    93 %
    3

    View Slide

  15. Supermutants - Coverage Adequate Test Suite
    2 Coverage Adequate Tests
    0 %
    25 %
    50 %
    75 %
    100 %
    Full Suite (7 Tests) Coverage Adequate
    93 %
    93 %
    2 Coverage Adequate Tests
    0 %
    25 %
    50 %
    75 %
    100 %
    Full Suite (7 Tests) Coverage Adequate
    93 %
    93 %
    3

    View Slide

  16. Supermutants - Coverage Adequate Test Suite
    4

    View Slide

  17. Supermutants - Coverage Adequate Test Suite
    4

    View Slide

  18. Supermutants - Coverage Adequate Test Suite
    4

    View Slide

  19. Supermutants - Coverage Adequate Test Suite
    4

    View Slide

  20. Supermutant Execution
    5

    View Slide

  21. Supermutant Execution
    5
    7 Tests

    View Slide

  22. Supermutant Execution
    5
    7 Tests

    View Slide

  23. Supermutant Execution
    5
    7 Tests

    View Slide

  24. Supermutant Execution
    5
    7 Tests

    View Slide

  25. Supermutant Execution
    5
    3 Tests 3 Tests
    7 Tests

    View Slide

  26. Supermutant Execution
    5
    3 Tests 3 Tests
    7 Tests

    View Slide

  27. Supermutant Execution
    5
    3 Tests 3 Tests
    7 Tests

    View Slide

  28. Supermutant Execution
    5
    3 Tests 3 Tests
    7 Tests

    View Slide

  29. Supermutant Execution
    5
    3 Tests 3 Tests
    3 Tests
    3 Tests
    7 Tests

    View Slide

  30. Supermutant Execution
    5
    3 Tests 3 Tests
    3 Tests
    3 Tests
    7 Tests

    View Slide

  31. Supermutant Execution
    5
    3 Tests 3 Tests
    3 Tests
    3 Tests
    7 Tests

    View Slide

  32. Supermutant Execution
    5
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test
    7 Tests

    View Slide

  33. Supermutant Execution
    5
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test
    7 Tests

    View Slide

  34. Supermutant Execution
    5
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test
    7 Tests
    34 vs 42 Test Executions

    View Slide

  35. Supermutants - Mutant Kill Matrix
    6
    7 Tests
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test

    View Slide

  36. Supermutants - Mutant Kill Matrix
    6
    Test 1
    Test 2
    Test 3
    Test 4
    Test 5
    Test 6
    Test 7
    7 Tests
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test

    View Slide

  37. Supermutants - Mutant Kill Matrix
    6
    Test 1
    Test 2
    Test 3
    Test 4
    Test 5
    Test 6
    Test 7







    7 Tests
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test

    View Slide

  38. Supermutants - Mutant Kill Matrix
    6
    Test 1
    Test 2
    Test 3
    Test 4
    Test 5
    Test 6
    Test 7














    7 Tests
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test

    View Slide

  39. Supermutants - Mutant Kill Matrix
    6
    Test 1
    Test 2
    Test 3
    Test 4
    Test 5
    Test 6
    Test 7










































    7 Tests
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test

    View Slide

  40. Supermutants - Mutant Kill Matrix
    6
    Test 1
    Test 2
    Test 3
    Test 4
    Test 5
    Test 6
    Test 7










































    7 Tests
    3 Tests 3 Tests
    3 Tests
    3 Tests
    1 Test
    1 Test

    View Slide

  41. Case Study - Urlparse
    7

    View Slide

  42. Case Study - Urlparse
    7
    0 exec.
    10000 exec.
    20000 exec.
    30000 exec.
    40000 exec.
    Full Matrix Strong Testsuite Weak Testsuite

    View Slide

  43. Case Study - Urlparse
    7
    0 exec.
    10000 exec.
    20000 exec.
    30000 exec.
    40000 exec.
    Full Matrix Strong Testsuite Weak Testsuite
    32,000

    View Slide

  44. Case Study - Urlparse
    7
    0 exec.
    10000 exec.
    20000 exec.
    30000 exec.
    40000 exec.
    Full Matrix Strong Testsuite Weak Testsuite
    13,009
    32,000

    View Slide

  45. Case Study - Urlparse
    7
    0 exec.
    10000 exec.
    20000 exec.
    30000 exec.
    40000 exec.
    Full Matrix Strong Testsuite Weak Testsuite
    9,037
    13,009
    32,000

    View Slide

  46. Case Study - Urlparse
    7
    0 exec.
    10000 exec.
    20000 exec.
    30000 exec.
    40000 exec.
    Full Matrix Strong Testsuite Weak Testsuite
    9,037
    13,009
    32,000
    Over 70% of Test Executions Are Saved

    View Slide

  47. When Are Supermutants Super?
    8

    View Slide

  48. When Are Supermutants Super?
    8
    Many and Weak Tests

    View Slide

  49. When Are Supermutants Super?
    8
    Many and Weak Tests Stubborn Mutants

    View Slide

  50. When Are Supermutants Super?
    8
    Many and Weak Tests Stubborn Mutants No Fault Masking

    View Slide

  51. When Are Supermutants Super?
    8
    Many and Weak Tests Stubborn Mutants No Fault Masking
    Automatically Generated Test Suites
    Can Be Efficiently Analyzed

    View Slide

  52. 9
    Summary

    View Slide

  53. 9
    Summary

    View Slide

  54. 9
    Summary

    View Slide

  55. 9
    Summary

    View Slide

  56. 9
    Summary

    View Slide