Upgrade to Pro — share decks privately, control downloads, hide ads and more …

If You Can't Kill a Supermutant, You Have a Problem

D27cb84e0d30e2778e9b66d6a5f42106?s=47 Rahul Gopinath
April 09, 2018
Research
0
130

If You Can't Kill a Supermutant, You Have a Problem

Mutation Workshop 2018

D27cb84e0d30e2778e9b66d6a5f42106?s=128

Rahul Gopinath

April 09, 2018
Tweet

More Decks by Rahul Gopinath

See All by Rahul Gopinath
Learning And Refining Input Grammars For Effective Fuzzing
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
4
Input Languages for 
Effective and Focused Fuzzing
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
42
Fuzzing -- From Alchemy to a Science
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
71
Input Algebras
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
84
Inducing Subtle Mutations with Program Repair
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
64
Mining Input Grammars from Dynamic Control Flow
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
90
Revisiting the Relationship Between Fault Detection, Test Adequacy Criteria, and Test Set Size
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
62
Abstracting Failure Inducing Inputs
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
65
Learning Grammars without Samples
D27cb84e0d30e2778e9b66d6a5f42106?s=48 rahulgopinath
0
120

Other Decks in Research

See All in Research
第10回チャンピオンズミーティング・アクエリアス杯ラウンド2集計 / Umamusume Aquarius 2022 Round2
9782dd351a8f1737d4ef02d839e821f9?s=48 kitachan_black
0
1.1k
CompilerGym CGO 2022 Tutorial Part 2
0d2bf4a55ccf5ca212897c2c09e18c94?s=48 chriscummins
0
460
機械学習ベースの動画像処理における近似計算手法の検討 (CPSY 2022/03)
290a66d9796825297c2cdb438a372f6a?s=48 hashi0203
0
150
ICCV 2021 ただただVision and Languageの論文を読んでみた
Be0f86176276318b4b9775d795278f7e?s=48 yushiku
3
1.7k
CVIM_2022_01
Cdf066fce1247cc62daf6aec83b78b54?s=48 keiichirokagawa
0
440
時間情報表現抽出とルールベース解析器のこれから / Temporal Expression Analysis in Japanese and Future of Rule-based Approach
64fb973c47087ece7fb9b45d5b8593a4?s=48 yag_ays
PRO
0
740
深層学習を用いた三次元点群処理入門
1aefc563d6967f558b1ae295cc1d56de?s=48 nnchiba
4
2.3k
AIOps研究録―SREのための
システム障害の自動原因診断 / SRE NEXT 2022
A658ec7f1badf73819dfa501165016c1?s=48 yuukit
5
1.1k
ライティング支援のための文法誤り訂正
2d4315a0cfbfeb12fdc48bbadd955cd9?s=48 chemical_tree
1
1k
ICSE22: An Exploratory Study of Deep Learning Supply Chain
2644bdfd7318fdc10672be113fb39133?s=48 sunflowerpku
0
200
民間企業におけるデータ整備の課題と工夫 / 20220305
Fb925cff494f6a5158a461493be48066?s=48 yuzutas0
4
3.1k
Iterative source steering を用いたオンライン補助関数型独立ベクトル分析に基づくブラインド音源分離 / Blind source separation using online auxiliary-function-based independent vector analysis with iterative source steering
Fba2bc6aeb14d00ab84710be1903da49?s=48 taishi
0
140

Featured

See All Featured
A better future with KSS
5f2da528927a2ec9ba4fec2069cbc958?s=48 kneath
225
15k
What's in a price? How to price your products and services
Dad095ea7038f89f760419ce475d5d14?s=48 michaelherold
229
9.3k
Clear Off the Table
Ef32e0b1ac5082450dc8c1fca3a26b5c?s=48 cherdarchuk
79
280k
Helping Users Find Their Own Way: Creating Modern Search Experiences
3d4519fd34b76fb265fc0237f3792bd4?s=48 danielanewman
7
1k
Embracing the Ebb and Flow
C9b18d9dff88a9dd2393364c2b3b21bd?s=48 colly
73
3.3k
Large-scale JavaScript Application Architecture
96270e4c3e5e9806cf7245475c00b275?s=48 addyosmani
499
110k
How to train your dragon (web standard)
053e75a5b48b44d6dd0612795dfb326d?s=48 notwaldorf
57
3.8k
Stop Working from a Prison Cell
C1daf20e5c49ff910745198ef9869ac2?s=48 hatefulcrawdad
261
17k
Side Projects
027d1ebf66cc039a0bd3b55eeadbe75d?s=48 sachag
449
37k
Visualization
170b760e0147792d0140e59ec78e9893?s=48 eitanlees
124
11k
Designing Dashboards & Data Visualisations in Web Apps
Fde6c3a50ca518a909ef35c22c0ee0e4?s=48 destraynor
224
49k
Building a Scalable Design System with Sketch
1177e050db6bafe62885362edf6e3537?s=48 lauravandoore
447
30k

Transcript

  1. 9. April 2018 Rahul Gopinath, Björn Mathis and Andreas Zeller

    If You Can't Kill a Supermutant You Have a Problem 1

  2. Mutant Kill Matrix 2

  3. Mutant Kill Matrix 2

  4. Mutant Kill Matrix 2

  5. Mutant Kill Matrix 2

  6. Mutant Kill Matrix 2

  7. Mutant Kill Matrix 2

  8. Mutant Kill Matrix 2

  9. Mutant Kill Matrix 2

  10. Mutant Kill Matrix 2 Enormous Amount of Test Executions

  11. Supermutants - Coverage Adequate Test Suite 3

  12. Supermutants - Coverage Adequate Test Suite 0 % 25 %

    50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 3

  13. Supermutants - Coverage Adequate Test Suite 0 Coverage Adequate Tests

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 93 % 0 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 93 % 3

  14. Supermutants - Coverage Adequate Test Suite 1 Coverage Adequate Test

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 70 % 93 % 1 Coverage Adequate Test 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 70 % 93 % 3

  15. Supermutants - Coverage Adequate Test Suite 2 Coverage Adequate Tests

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 93 % 93 % 2 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 93 % 93 % 3

  16. Supermutants - Coverage Adequate Test Suite 4

  17. Supermutants - Coverage Adequate Test Suite 4

  18. Supermutants - Coverage Adequate Test Suite 4

  19. Supermutants - Coverage Adequate Test Suite 4

  20. Supermutant Execution 5

  21. Supermutant Execution 5 7 Tests

  22. Supermutant Execution 5 7 Tests

  23. Supermutant Execution 5 7 Tests

  24. Supermutant Execution 5 7 Tests

  25. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  26. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  27. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  28. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  29. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  30. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  31. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  32. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests

  33. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests

  34. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests 34 vs 42 Test Executions

  35. Supermutants - Mutant Kill Matrix 6 7 Tests 3 Tests

    3 Tests 3 Tests 3 Tests 1 Test 1 Test

  36. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  37. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  38. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  39. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  40. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  41. Case Study - Urlparse 7

  42. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite

  43. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 32,000

  44. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 13,009 32,000

  45. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 9,037 13,009 32,000

  46. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 9,037 13,009 32,000 Over 70% of Test Executions Are Saved

  47. When Are Supermutants Super? 8

  48. When Are Supermutants Super? 8 Many and Weak Tests

  49. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants

  50. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants No Fault Masking

  51. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants No Fault Masking Automatically Generated Test Suites Can Be Efficiently Analyzed

  52. 9 Summary

  53. 9 Summary

  54. 9 Summary

  55. 9 Summary

  56. 9 Summary