Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

If You Can't Kill a Supermutant, You Have a Pr...

Avatar for Rahul Gopinath Rahul Gopinath
April 09, 2018
Research
360
0

If You Can't Kill a Supermutant, You Have a Problem

Mutation Workshop 2018

Avatar for Rahul Gopinath

Rahul Gopinath

April 09, 2018

More Decks by Rahul Gopinath

See All by Rahul Gopinath
Better Distributional Fit Does Not Improve Reachable Coverage Estimation in Fuzzing
Avatar for Rahul Gopinath rahulgopinath
0
6
Dr. DD 1-Minimal Isolation of Failure Causes via Deferred Restarts
Avatar for Rahul Gopinath rahulgopinath
0
6
Maximal Format-free Data Repair
Avatar for Rahul Gopinath rahulgopinath
0
13
Specification for Free: Behavior-Driven Fuzzing with Inferred Specifications
Avatar for Rahul Gopinath rahulgopinath
0
36
Evaluating Impact of Coverage Feedback on Estimators for Maximum Reachability in Fuzzing (Registered Report)
Avatar for Rahul Gopinath rahulgopinath
0
62
Automatic Data Repair without Format Specifications
Avatar for Rahul Gopinath rahulgopinath
0
71
From Theory to Practice: Blackbox Testing in an Industrial Telecom Environment
Avatar for Rahul Gopinath rahulgopinath
0
83
Assessing Reliability of Statistical Maximum Coverage Estimators in Fuzzing
Avatar for Rahul Gopinath rahulgopinath
0
83
Fuzzing Without Specifications: Learning Structure from Behaviour Part I
Avatar for Rahul Gopinath rahulgopinath
0
96

Other Decks in Research

See All in Research
論文読み会 SNLP2026 Tau2-Bench: Evaluating Conversational Agents in a Dual-Control Environment
Avatar for S s_mizuki_nlp
0
320
Prithvi-EO-2.0
Avatar for SatAI.challenge satai
3
110
Hitting the buffers
Avatar for Emma beale
0
360
湯村研究室の紹介2026 / yumulab2026
Avatar for yumulab yumulab
0
190
山中市政が招いた組織崩壊：よこはま安心ボックスを例に
Avatar for sishi sishi
0
370
Evaluating LLM Reliability Across Facts, Evidence, and Cultures
Avatar for Yuki Arase yukiar
0
220
某助成金プロジェクト採択に向けて企業研究所のアウトリーチ専任者がやったこと
Avatar for afroscript afroscript
0
230
フィジカルAIの社会実装に向けて -AIエージェントからAgentic Roboticsへ-
Avatar for Hironobu Fujiyoshi hf149
1
2.6k
Détection des lésions podales par thermographie infrarouge chez les bovins et les ovins
Avatar for Idele institutdelelevage
PRO
0
110
Pesée et pointage par imagerie 3D : PHENO3D entre en phase de test terrains
Avatar for Idele institutdelelevage
PRO
0
140
JPA2026_NetworkTutorial_JunKashihara
Avatar for JunKashihara junkashihara
0
250
J-STAGEの現況と全文XML登載必須化について
Avatar for XSPA xspa2012
0
360

Featured

See All Featured
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
31
3.4k
10 Git Anti Patterns You Should be Aware of
Avatar for Lemi Orhan Ergin lemiorhan
PRO
659
62k
WENDY [Excerpt]
Avatar for Tessa Abrams tessaabrams
14
40k
Principles of Awesome APIs and How to Build Them.
Avatar for Keavy McMinn keavy
128
18k
Navigating the moral maze — ethical principles for Al-driven product design
Avatar for Skipper Chong Warson skipperchong
2
660
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
Avatar for Tammy Everts tammyeverts
56
3.5k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
38
3.1k
エンジニアに許された特別な時間の終わり
Avatar for watany watany
109
250k
Ecommerce SEO: The Keys for Success Now & Beyond - #SERPConf2024
Avatar for Aleyda Solis aleyda
1
2.2k
For a Future-Friendly Web
Avatar for Brad Frost brad_frost
184
11k
Getting science done with accelerated Python computing platforms
Avatar for Jacob Tomlinson jacobtomlinson
2
560
Agile Actions for Facilitating Distributed Teams - ADO2019
Avatar for Mark Kilby mkilby
0
330

Transcript

  1. 9. April 2018 Rahul Gopinath, Björn Mathis and Andreas Zeller

    If You Can't Kill a Supermutant You Have a Problem 1

  2. Mutant Kill Matrix 2

  3. Mutant Kill Matrix 2

  4. Mutant Kill Matrix 2

  5. Mutant Kill Matrix 2

  6. Mutant Kill Matrix 2

  7. Mutant Kill Matrix 2

  8. Mutant Kill Matrix 2

  9. Mutant Kill Matrix 2

  10. Mutant Kill Matrix 2 Enormous Amount of Test Executions

  11. Supermutants - Coverage Adequate Test Suite 3

  12. Supermutants - Coverage Adequate Test Suite 0 % 25 %

    50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 3

  13. Supermutants - Coverage Adequate Test Suite 0 Coverage Adequate Tests

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 93 % 0 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 93 % 3

  14. Supermutants - Coverage Adequate Test Suite 1 Coverage Adequate Test

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 70 % 93 % 1 Coverage Adequate Test 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 70 % 93 % 3

  15. Supermutants - Coverage Adequate Test Suite 2 Coverage Adequate Tests

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 93 % 93 % 2 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 93 % 93 % 3

  16. Supermutants - Coverage Adequate Test Suite 4

  17. Supermutants - Coverage Adequate Test Suite 4

  18. Supermutants - Coverage Adequate Test Suite 4

  19. Supermutants - Coverage Adequate Test Suite 4

  20. Supermutant Execution 5

  21. Supermutant Execution 5 7 Tests

  22. Supermutant Execution 5 7 Tests

  23. Supermutant Execution 5 7 Tests

  24. Supermutant Execution 5 7 Tests

  25. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  26. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  27. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  28. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  29. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  30. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  31. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  32. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests

  33. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests

  34. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests 34 vs 42 Test Executions

  35. Supermutants - Mutant Kill Matrix 6 7 Tests 3 Tests

    3 Tests 3 Tests 3 Tests 1 Test 1 Test

  36. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  37. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  38. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  39. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  40. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  41. Case Study - Urlparse 7

  42. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite

  43. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 32,000

  44. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 13,009 32,000

  45. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 9,037 13,009 32,000

  46. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 9,037 13,009 32,000 Over 70% of Test Executions Are Saved

  47. When Are Supermutants Super? 8

  48. When Are Supermutants Super? 8 Many and Weak Tests

  49. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants

  50. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants No Fault Masking

  51. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants No Fault Masking Automatically Generated Test Suites Can Be Efficiently Analyzed

  52. 9 Summary

  53. 9 Summary

  54. 9 Summary

  55. 9 Summary

  56. 9 Summary