Upgrade to Pro — share decks privately, control downloads, hide ads and more …

If You Can't Kill a Supermutant, You Have a Pr...

Rahul Gopinath
April 09, 2018
Research
0
250

If You Can't Kill a Supermutant, You Have a Problem

Mutation Workshop 2018

Rahul Gopinath

April 09, 2018
Tweet

More Decks by Rahul Gopinath

See All by Rahul Gopinath
Look Ma No Hands! Testing Software Without Specifications
rahulgopinath
0
38
How to Talk to Strange Programs and Find Bugs
rahulgopinath
0
68
Dancing to an Unknown Music: Grammar Inferrence with Prefix Queries
rahulgopinath
0
54
Systematic Assessment of Fuzzers using Mutation Analysis
rahulgopinath
0
130
Building Blocks for Fuzzing
rahulgopinath
0
170
"Synthesizing Input Grammars" A Replication Study
rahulgopinath
0
210
CLIFuzzer: Mining Grammars for Command-Line Invocations
rahulgopinath
0
100
Learning And Refining Input Grammars For Effective Fuzzing
rahulgopinath
0
170
Input Languages for 
Effective and Focused Fuzzing
rahulgopinath
0
190

Other Decks in Research

See All in Research
20240719_第2回熊本の交通を語る会
trafficbrain
0
390
SSII2024 [OS2] 大規模言語モデルと基盤モデルの射程
ssii
PRO
0
460
「確率的なオウム」にできること、またそれがなぜできるのかについて
eumesy
PRO
7
2.9k
日本語医療LLM評価ベンチマークの構築と性能分析
fta98
3
380
Weekly AI Agents News! 7月号 論文のアーカイブ
masatoto
1
160
SSII2024 [OS1] 研究紹介100連発（オープンニング）
ssii
PRO
0
470
SSII2024 [OS2] 画像、その先へ 〜モーション解析への誘い〜
ssii
PRO
1
1.2k
LINEチャットボット「全力肯定彼氏くん(LuC4)」の 1年を振り返る
o_ob
0
1.4k
SSII2024 [TS3] 画像認識におけるマルチモーダル基盤モデル ～基盤モデル、あなたのタスクに役立つかも？～
ssii
PRO
0
1.1k
言語処理学会30周年記念事業留学支援交流会＠YANS2024：「学生のための短期留学」
a1da4
1
180
自動運転・AIシステムの問題を賢く探す・賢く直す / Smart Search & Repair Techniques for Automated Driving Systems and AI Systems
ishikawafyu
0
180
第60回名古屋CV・PRMU勉強会：CVPR2024論文紹介（Vision Transformer）
waka_90b
1
150

Featured

See All Featured
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
89
16k
Music & Morning Musume
bryan
46
6k
[RailsConf 2023] Rails as a piece of cake
palkan
48
4.6k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
166
48k
Reflections from 52 weeks, 52 projects
jeffersonlam
346
20k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
8.9k
Why Our Code Smells
bkeepers
PRO
334
56k
Code Reviewing Like a Champion
maltzj
517
39k
Producing Creativity
orderedlist
PRO
340
39k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
26
2k
Debugging Ruby Performance
tmm1
72
12k
Designing the Hi-DPI Web
ddemaree
278
34k

Transcript

  1. 9. April 2018 Rahul Gopinath, Björn Mathis and Andreas Zeller

    If You Can't Kill a Supermutant You Have a Problem 1

  2. Mutant Kill Matrix 2

  3. Mutant Kill Matrix 2

  4. Mutant Kill Matrix 2

  5. Mutant Kill Matrix 2

  6. Mutant Kill Matrix 2

  7. Mutant Kill Matrix 2

  8. Mutant Kill Matrix 2

  9. Mutant Kill Matrix 2

  10. Mutant Kill Matrix 2 Enormous Amount of Test Executions

  11. Supermutants - Coverage Adequate Test Suite 3

  12. Supermutants - Coverage Adequate Test Suite 0 % 25 %

    50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 3

  13. Supermutants - Coverage Adequate Test Suite 0 Coverage Adequate Tests

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 93 % 0 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 93 % 3

  14. Supermutants - Coverage Adequate Test Suite 1 Coverage Adequate Test

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 70 % 93 % 1 Coverage Adequate Test 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 70 % 93 % 3

  15. Supermutants - Coverage Adequate Test Suite 2 Coverage Adequate Tests

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 93 % 93 % 2 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 93 % 93 % 3

  16. Supermutants - Coverage Adequate Test Suite 4

  17. Supermutants - Coverage Adequate Test Suite 4

  18. Supermutants - Coverage Adequate Test Suite 4

  19. Supermutants - Coverage Adequate Test Suite 4

  20. Supermutant Execution 5

  21. Supermutant Execution 5 7 Tests

  22. Supermutant Execution 5 7 Tests

  23. Supermutant Execution 5 7 Tests

  24. Supermutant Execution 5 7 Tests

  25. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  26. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  27. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  28. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  29. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  30. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  31. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  32. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests

  33. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests

  34. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests 34 vs 42 Test Executions

  35. Supermutants - Mutant Kill Matrix 6 7 Tests 3 Tests

    3 Tests 3 Tests 3 Tests 1 Test 1 Test

  36. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  37. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  38. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  39. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  40. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  41. Case Study - Urlparse 7

  42. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite

  43. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 32,000

  44. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 13,009 32,000

  45. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 9,037 13,009 32,000

  46. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 9,037 13,009 32,000 Over 70% of Test Executions Are Saved

  47. When Are Supermutants Super? 8

  48. When Are Supermutants Super? 8 Many and Weak Tests

  49. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants

  50. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants No Fault Masking

  51. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants No Fault Masking Automatically Generated Test Suites Can Be Efficiently Analyzed

  52. 9 Summary

  53. 9 Summary

  54. 9 Summary

  55. 9 Summary

  56. 9 Summary