If You Can't Kill a Supermutant, You Have a Problem

D27cb84e0d30e2778e9b66d6a5f42106?s=47 Rahul Gopinath
April 09, 2018
Research
0
75

If You Can't Kill a Supermutant, You Have a Problem

Mutation Workshop 2018

D27cb84e0d30e2778e9b66d6a5f42106?s=128

Rahul Gopinath

April 09, 2018
Tweet

More Decks by Rahul Gopinath

See All by Rahul Gopinath
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
35
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
240
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
36
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
22
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
140
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
58
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
52
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
33
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
29

Other Decks in Research

See All in Research
24b82e535177c6be997ca4ec43d8df6b?s=48 tomoyanonymous
1
310
951ca4f70b5884e18cf3f7e88c441d6a?s=48 issdu
0
1.2k
9e7d522690b2ea97d452a0e09d25d2ff?s=48 et771011
0
230
7b3e0dbc0d712ad5df602d9f9e5e4209?s=48 sei88888
0
350
30a79337fdc6926a768e2f65139dc656?s=48 hijojo_partners
1
1.3k
C6b97a47d5406cfdef50a5c755751c16?s=48 sorami
0
280
Cd3d2cb2dadf5488935fe0ddaea7938a?s=48 monochromegane
0
240
4f7b64c10f3f95c6f6c99423380ad6f3?s=48 sobamchan
1
380
7aff8f547184534da3ca2e14e63a68a8?s=48 mgiraldo
0
130
F0aa5f405baf92b0242549266327f267?s=48 elerac
0
130
B91bb8ea34ba992953bfcdf5a51ea457?s=48 mns54
0
830
6325a9b34da54d5cbddb814c3987a2fe?s=48 chokkan
0
2.3k

Featured

See All Featured
4394ceb8b277f35ee3da7030384efb5d?s=48 davidbonilla
68
3.4k
D8fc2580cfaca035f666d9e4ee79a7f7?s=48 mongodb
23
3.7k
D67fff74178013024d833b3eec6cf27f?s=48 lynnandtonic
266
16k
719435d98d452de7ac367c828266cf01?s=48 erikaheidi
10
3.8k
96270e4c3e5e9806cf7245475c00b275?s=48 addyosmani
308
21k
170b760e0147792d0140e59ec78e9893?s=48 eitanlees
106
9.5k
A415db3ac9e9668acbc1fb36eead84ac?s=48 mthomps
37
2.2k
462dad0dd24c568a32dcdb0ae895ae1b?s=48 rasmusluckow
315
18k
C7393b7ba7ec9c8890dd77d209fbb3c9?s=48 maltzj
499
36k
20bfe76b3d6105641f879fe45cfc9272?s=48 bkeepers
319
53k
C620790ae5bf5b50c245b2e0ef95f338?s=48 deanohume
294
27k
C9b18d9dff88a9dd2393364c2b3b21bd?s=48 colly
65
2.9k

Transcript

  1. 9. April 2018 Rahul Gopinath, Björn Mathis and Andreas Zeller

    If You Can't Kill a Supermutant You Have a Problem 1

  2. Mutant Kill Matrix 2

  3. Mutant Kill Matrix 2

  4. Mutant Kill Matrix 2

  5. Mutant Kill Matrix 2

  6. Mutant Kill Matrix 2

  7. Mutant Kill Matrix 2

  8. Mutant Kill Matrix 2

  9. Mutant Kill Matrix 2

  10. Mutant Kill Matrix 2 Enormous Amount of Test Executions

  11. Supermutants - Coverage Adequate Test Suite 3

  12. Supermutants - Coverage Adequate Test Suite 0 % 25 %

    50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 3

  13. Supermutants - Coverage Adequate Test Suite 0 Coverage Adequate Tests

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 93 % 0 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 93 % 3

  14. Supermutants - Coverage Adequate Test Suite 1 Coverage Adequate Test

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 70 % 93 % 1 Coverage Adequate Test 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 70 % 93 % 3

  15. Supermutants - Coverage Adequate Test Suite 2 Coverage Adequate Tests

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 93 % 93 % 2 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 93 % 93 % 3

  16. Supermutants - Coverage Adequate Test Suite 4

  17. Supermutants - Coverage Adequate Test Suite 4

  18. Supermutants - Coverage Adequate Test Suite 4

  19. Supermutants - Coverage Adequate Test Suite 4

  20. Supermutant Execution 5

  21. Supermutant Execution 5 7 Tests

  22. Supermutant Execution 5 7 Tests

  23. Supermutant Execution 5 7 Tests

  24. Supermutant Execution 5 7 Tests

  25. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  26. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  27. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  28. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  29. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  30. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  31. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  32. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests

  33. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests

  34. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests 34 vs 42 Test Executions

  35. Supermutants - Mutant Kill Matrix 6 7 Tests 3 Tests

    3 Tests 3 Tests 3 Tests 1 Test 1 Test

  36. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  37. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  38. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  39. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  40. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  41. Case Study - Urlparse 7

  42. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite

  43. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 32,000

  44. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 13,009 32,000

  45. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 9,037 13,009 32,000

  46. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 9,037 13,009 32,000 Over 70% of Test Executions Are Saved

  47. When Are Supermutants Super? 8

  48. When Are Supermutants Super? 8 Many and Weak Tests

  49. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants

  50. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants No Fault Masking

  51. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants No Fault Masking Automatically Generated Test Suites Can Be Efficiently Analyzed

  52. 9 Summary

  53. 9 Summary

  54. 9 Summary

  55. 9 Summary

  56. 9 Summary