Upgrade to Pro — share decks privately, control downloads, hide ads and more …

If You Can't Kill a Supermutant, You Have a Pr...

Avatar for Rahul Gopinath Rahul Gopinath
April 09, 2018
Research
0
310

If You Can't Kill a Supermutant, You Have a Problem

Mutation Workshop 2018

Avatar for Rahul Gopinath

Rahul Gopinath

April 09, 2018
Tweet

More Decks by Rahul Gopinath

See All by Rahul Gopinath
Fuzzing Without Specifications: Learning Structure from Behaviour Part I
Avatar for Rahul Gopinath rahulgopinath
0
5
Fuzzing Without Specifications: Learning Structure from Behaviour Part II
Avatar for Rahul Gopinath rahulgopinath
0
2
How to Compare Fuzzers
Avatar for Rahul Gopinath rahulgopinath
0
83
Mutation Analysis: Answering the Fuzzing Challenge
Avatar for Rahul Gopinath rahulgopinath
0
43
An Empirical Evaluation of Frequency Based Statistical Models for Estimating Killable Mutants
Avatar for Rahul Gopinath rahulgopinath
0
51
Look Ma No Hands! Testing Software Without Specifications
Avatar for Rahul Gopinath rahulgopinath
0
110
How to Talk to Strange Programs and Find Bugs
Avatar for Rahul Gopinath rahulgopinath
0
140
Dancing to an Unknown Music: Grammar Inferrence with Prefix Queries
Avatar for Rahul Gopinath rahulgopinath
0
120
Systematic Assessment of Fuzzers using Mutation Analysis
Avatar for Rahul Gopinath rahulgopinath
0
200

Other Decks in Research

See All in Research
LLM-as-a-Judge: 文章をLLMで評価する@教育機関DXシンポ
Avatar for Kouta Nakayama k141303
3
800
Type Theory as a Formal Basis of Natural Language Semantics
Avatar for Daiki Matsuoka daikimatsuoka
1
220
ｳｯﾄﾞｽﾀｯｸﾁｬﾝ：木材を用いた小型エージェントロボットの開発と印象評価 / ec75-sato
Avatar for yumulab yumulab
1
390
Cross-Media Information Spaces and Architectures
Avatar for Beat Signer signer
PRO
0
220
CHaserWeb:ブラウザ上で動作する対戦型プログラミング学習環境の提案と評価 / i2025-inoue
Avatar for yumulab yumulab
0
190
SSII2025 [SS1] レンズレスカメラ
Avatar for 画像センシングシンポジウム ssii
PRO
2
920
Weekly AI Agents News!
Avatar for masatoto masatoto
33
67k
Self-supervised audiovisual representation learning for remote sensing data
Avatar for SatAI.challenge satai
3
210
Pix2Poly: A Sequence Prediction Method for End-to-end Polygonal Building Footprint Extraction from Remote Sensing Imagery
Avatar for SatAI.challenge satai
3
420
電力システム最適化入門
Avatar for MIKIO KUBO mickey_kubo
1
620
データサイエンティストの採用に関するアンケート
Avatar for The Japan DataScientist Society datascientistsociety
PRO
0
940
90 分で学ぶ P 対 NP 問題
Avatar for E869120 e869120
17
7.4k

Featured

See All Featured
Designing for Performance
Avatar for Lara Hogan lara
609
69k
Bash Introduction
Avatar for André Augusto Costa Santos 62gerente
614
210k
Into the Great Unknown - MozCon
Avatar for Noah Learner thekraken
39
1.8k
Dealing with People You Can't Stand - Big Design 2015
Avatar for Cassini Nazir cassininazir
367
26k
Principles of Awesome APIs and How to Build Them.
Avatar for Keavy McMinn keavy
126
17k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
Avatar for Vitaly Friedman smashingmag
252
21k
Code Reviewing Like a Champion
Avatar for maltzj maltzj
524
40k
Improving Core Web Vitals using Speculation Rules API
Avatar for Sergey Chernyshev sergeychernyshev
16
930
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
36
2.7k
Designing Experiences People Love
Avatar for Jonathan Moore moore
142
24k
Fight the Zombie Pattern Library - RWD Summit 2016
Avatar for Marcelo Somers marcelosomers
233
17k
Testing 201, or: Great Expectations
Avatar for Joseph Mastey jmmastey
42
7.5k

Transcript

  1. 9. April 2018 Rahul Gopinath, Björn Mathis and Andreas Zeller

    If You Can't Kill a Supermutant You Have a Problem 1

  2. Mutant Kill Matrix 2

  3. Mutant Kill Matrix 2

  4. Mutant Kill Matrix 2

  5. Mutant Kill Matrix 2

  6. Mutant Kill Matrix 2

  7. Mutant Kill Matrix 2

  8. Mutant Kill Matrix 2

  9. Mutant Kill Matrix 2

  10. Mutant Kill Matrix 2 Enormous Amount of Test Executions

  11. Supermutants - Coverage Adequate Test Suite 3

  12. Supermutants - Coverage Adequate Test Suite 0 % 25 %

    50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 3

  13. Supermutants - Coverage Adequate Test Suite 0 Coverage Adequate Tests

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 93 % 0 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 93 % 3

  14. Supermutants - Coverage Adequate Test Suite 1 Coverage Adequate Test

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 70 % 93 % 1 Coverage Adequate Test 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 70 % 93 % 3

  15. Supermutants - Coverage Adequate Test Suite 2 Coverage Adequate Tests

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 93 % 93 % 2 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 93 % 93 % 3

  16. Supermutants - Coverage Adequate Test Suite 4

  17. Supermutants - Coverage Adequate Test Suite 4

  18. Supermutants - Coverage Adequate Test Suite 4

  19. Supermutants - Coverage Adequate Test Suite 4

  20. Supermutant Execution 5

  21. Supermutant Execution 5 7 Tests

  22. Supermutant Execution 5 7 Tests

  23. Supermutant Execution 5 7 Tests

  24. Supermutant Execution 5 7 Tests

  25. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  26. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  27. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  28. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  29. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  30. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  31. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  32. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests

  33. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests

  34. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests 34 vs 42 Test Executions

  35. Supermutants - Mutant Kill Matrix 6 7 Tests 3 Tests

    3 Tests 3 Tests 3 Tests 1 Test 1 Test

  36. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  37. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  38. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  39. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  40. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  41. Case Study - Urlparse 7

  42. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite

  43. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 32,000

  44. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 13,009 32,000

  45. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 9,037 13,009 32,000

  46. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 9,037 13,009 32,000 Over 70% of Test Executions Are Saved

  47. When Are Supermutants Super? 8

  48. When Are Supermutants Super? 8 Many and Weak Tests

  49. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants

  50. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants No Fault Masking

  51. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants No Fault Masking Automatically Generated Test Suites Can Be Efficiently Analyzed

  52. 9 Summary

  53. 9 Summary

  54. 9 Summary

  55. 9 Summary

  56. 9 Summary