If You Can't Kill a Supermutant, You Have a Problem

D27cb84e0d30e2778e9b66d6a5f42106?s=47 Rahul Gopinath
April 09, 2018
Research
0
73

If You Can't Kill a Supermutant, You Have a Problem

Mutation Workshop 2018

D27cb84e0d30e2778e9b66d6a5f42106?s=128

Rahul Gopinath

April 09, 2018
Tweet

Want more?

D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
29
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
220
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
35
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
1
130
A9a491b0fcbe0fbce3d64063a37add99?s=48 rmw
1
67
766b9746b59e6f09e280cd33cf4ed419?s=48 skipperchong
0
67
61857dafbd287b3027c4dcea9008ad3c?s=48 carmenhchung
4
130
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
3
150
C0390e8b11079f8761c0cd3274ed61cb?s=48 productmarketing
2
220
C35e01544a0dd94a2cf1619ee8a42ebb?s=48 clairelew
2
410
6bb82b13cda86d3b821fa44100335ddf?s=48 thoeni
1
110
B3ace07412a5178ea778e7eec161fc0a?s=48 jcasabona
4
200

Transcript

  1. 1.

    9. April 2018 Rahul Gopinath, Björn Mathis and Andreas Zeller

    If You Can't Kill a Supermutant You Have a Problem 1
  2. 2.

    Mutant Kill Matrix 2

  3. 3.

    Mutant Kill Matrix 2

  4. 4.

    Mutant Kill Matrix 2

  5. 5.

    Mutant Kill Matrix 2

  6. 6.

    Mutant Kill Matrix 2

  7. 7.

    Mutant Kill Matrix 2

  8. 8.

    Mutant Kill Matrix 2

  9. 9.

    Mutant Kill Matrix 2

  10. 10.

    Mutant Kill Matrix 2 Enormous Amount of Test Executions

  11. 11.

    Supermutants - Coverage Adequate Test Suite 3

  12. 12.

    Supermutants - Coverage Adequate Test Suite 0 % 25 %

    50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 3
  13. 13.

    Supermutants - Coverage Adequate Test Suite 0 Coverage Adequate Tests

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 93 % 0 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 93 % 3
  14. 14.

    Supermutants - Coverage Adequate Test Suite 1 Coverage Adequate Test

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 70 % 93 % 1 Coverage Adequate Test 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 70 % 93 % 3
  15. 15.

    Supermutants - Coverage Adequate Test Suite 2 Coverage Adequate Tests

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 93 % 93 % 2 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 93 % 93 % 3
  16. 16.

    Supermutants - Coverage Adequate Test Suite 4

  17. 17.

    Supermutants - Coverage Adequate Test Suite 4

  18. 18.

    Supermutants - Coverage Adequate Test Suite 4

  19. 19.

    Supermutants - Coverage Adequate Test Suite 4

  20. 20.

    Supermutant Execution 5

  21. 21.

    Supermutant Execution 5 7 Tests

  22. 22.

    Supermutant Execution 5 7 Tests

  23. 23.

    Supermutant Execution 5 7 Tests

  24. 24.

    Supermutant Execution 5 7 Tests

  25. 25.

    Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  26. 26.

    Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  27. 27.

    Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  28. 28.

    Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  29. 29.

    Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests
  30. 30.

    Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests
  31. 31.

    Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests
  32. 32.

    Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests
  33. 33.

    Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests
  34. 34.

    Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests 34 vs 42 Test Executions
  35. 35.

    Supermutants - Mutant Kill Matrix 6 7 Tests 3 Tests

    3 Tests 3 Tests 3 Tests 1 Test 1 Test
  36. 36.

    Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test
  37. 37.

    Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test
  38. 38.

    Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test
  39. 39.

    Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test
  40. 40.

    Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test
  41. 41.

    Case Study - Urlparse 7

  42. 42.

    Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite
  43. 43.

    Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 32,000
  44. 44.

    Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 13,009 32,000
  45. 45.

    Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 9,037 13,009 32,000
  46. 46.

    Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 9,037 13,009 32,000 Over 70% of Test Executions Are Saved
  47. 47.

    When Are Supermutants Super? 8

  48. 48.

    When Are Supermutants Super? 8 Many and Weak Tests

  49. 49.

    When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants
  50. 50.

    When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants No Fault Masking
  51. 51.

    When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants No Fault Masking Automatically Generated Test Suites Can Be Efficiently Analyzed
  52. 52.

    9 Summary

  53. 53.

    9 Summary

  54. 54.

    9 Summary

  55. 55.

    9 Summary

  56. 56.

    9 Summary