Upgrade to Pro — share decks privately, control downloads, hide ads and more …

If You Can't Kill a Supermutant, You Have a Pr...

Avatar for Rahul Gopinath Rahul Gopinath
April 09, 2018
Research
0
320

If You Can't Kill a Supermutant, You Have a Problem

Mutation Workshop 2018

Avatar for Rahul Gopinath

Rahul Gopinath

April 09, 2018
Tweet

More Decks by Rahul Gopinath

See All by Rahul Gopinath
Assessing Reliability of Statistical Maximum Coverage Estimators in Fuzzing
Avatar for Rahul Gopinath rahulgopinath
0
2
Fuzzing Without Specifications: Learning Structure from Behaviour Part I
Avatar for Rahul Gopinath rahulgopinath
0
23
Fuzzing Without Specifications: Learning Structure from Behaviour Part II
Avatar for Rahul Gopinath rahulgopinath
0
19
How to Compare Fuzzers
Avatar for Rahul Gopinath rahulgopinath
0
100
Mutation Analysis: Answering the Fuzzing Challenge
Avatar for Rahul Gopinath rahulgopinath
0
59
An Empirical Evaluation of Frequency Based Statistical Models for Estimating Killable Mutants
Avatar for Rahul Gopinath rahulgopinath
0
75
Look Ma No Hands! Testing Software Without Specifications
Avatar for Rahul Gopinath rahulgopinath
0
120
How to Talk to Strange Programs and Find Bugs
Avatar for Rahul Gopinath rahulgopinath
0
140
Dancing to an Unknown Music: Grammar Inferrence with Prefix Queries
Avatar for Rahul Gopinath rahulgopinath
0
130

Other Decks in Research

See All in Research
大域マッチングコスト最小化とLiDAR-IMUタイトカップリングに基づく三次元地図生成 / GLIM @ Robotics symposia 2022
Avatar for koide3 koide3
0
100
投資戦略202508
Avatar for Pragmaworks pw
0
560
とあるSREの博士「過程」 / A Certain SRE’s Ph.D. Journey
Avatar for Yuuki Tsubouchi (yuuk1) yuukit
11
4.3k
最適決定木を用いた処方的価格最適化
Avatar for MIKIO KUBO mickey_kubo
4
1.9k
時系列データに対する解釈可能な 決定木クラスタリング
Avatar for MIKIO KUBO mickey_kubo
2
1k
J-RAGBench: 日本語RAGにおける Generator評価ベンチマークの構築
Avatar for Koki Itai koki_itai
0
620
Learning to (Learn at Test Time): RNNs with Expressive Hidden States
Avatar for Hiroto Kurita kurita
1
250
2025年度人工知能学会全国大会チュートリアル講演「深層基盤モデルの数理」
Avatar for Taiji Suzuki taiji_suzuki
25
19k
電通総研の生成AI・エージェントの取り組み エンジニアリング業務向けAI活用事例紹介
Avatar for AITC - DENTSU SOKEN isidaitc
1
1k
言語モデルの地図：確率分布と 情報幾何による類似性の可視化
Avatar for Hidetoshi Shimodaira shimosan
5
1.6k
スキマバイトサービスにおける現場起点でのデザインアプローチ
Avatar for 眞行寺由郎 yoshioshingyouji
0
240
EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues
Avatar for SatAI.challenge satai
3
210

Featured

See All Featured
Agile that works and the tools we love
Avatar for Rasmus Luckow-Nielsen rasmusluckow
331
21k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
232
18k
GitHub's CSS Performance
Avatar for Jon Rohan jonrohan
1032
460k
Navigating Team Friction
Avatar for Lara Hogan lara
189
15k
Building Applications with DynamoDB
Avatar for Matt Wood mza
96
6.6k
Building Better People: How to give real-time feedback that sticks.
Avatar for Will Jessup wjessup
368
20k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
84
9.2k
Docker and Python
Avatar for Tania Allard trallard
46
3.6k
JavaScript: Past, Present, and Future - NDC Porto 2020
Avatar for David Neal reverentgeek
52
5.6k
BBQ
Avatar for Matthew Crist matthewcrist
89
9.8k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
Avatar for Chris Coyier chriscoyier
507
140k
Making the Leap to Tech Lead
Avatar for Ryan Cromwell cromwellryan
135
9.5k

Transcript

  1. 9. April 2018 Rahul Gopinath, Björn Mathis and Andreas Zeller

    If You Can't Kill a Supermutant You Have a Problem 1

  2. Mutant Kill Matrix 2

  3. Mutant Kill Matrix 2

  4. Mutant Kill Matrix 2

  5. Mutant Kill Matrix 2

  6. Mutant Kill Matrix 2

  7. Mutant Kill Matrix 2

  8. Mutant Kill Matrix 2

  9. Mutant Kill Matrix 2

  10. Mutant Kill Matrix 2 Enormous Amount of Test Executions

  11. Supermutants - Coverage Adequate Test Suite 3

  12. Supermutants - Coverage Adequate Test Suite 0 % 25 %

    50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 3

  13. Supermutants - Coverage Adequate Test Suite 0 Coverage Adequate Tests

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 93 % 0 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 0 % 93 % 3

  14. Supermutants - Coverage Adequate Test Suite 1 Coverage Adequate Test

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 70 % 93 % 1 Coverage Adequate Test 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 70 % 93 % 3

  15. Supermutants - Coverage Adequate Test Suite 2 Coverage Adequate Tests

    0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 93 % 93 % 2 Coverage Adequate Tests 0 % 25 % 50 % 75 % 100 % Full Suite (7 Tests) Coverage Adequate 93 % 93 % 3

  16. Supermutants - Coverage Adequate Test Suite 4

  17. Supermutants - Coverage Adequate Test Suite 4

  18. Supermutants - Coverage Adequate Test Suite 4

  19. Supermutants - Coverage Adequate Test Suite 4

  20. Supermutant Execution 5

  21. Supermutant Execution 5 7 Tests

  22. Supermutant Execution 5 7 Tests

  23. Supermutant Execution 5 7 Tests

  24. Supermutant Execution 5 7 Tests

  25. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  26. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  27. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  28. Supermutant Execution 5 3 Tests 3 Tests 7 Tests

  29. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  30. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  31. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 7 Tests

  32. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests

  33. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests

  34. Supermutant Execution 5 3 Tests 3 Tests 3 Tests 3

    Tests 1 Test 1 Test 7 Tests 34 vs 42 Test Executions

  35. Supermutants - Mutant Kill Matrix 6 7 Tests 3 Tests

    3 Tests 3 Tests 3 Tests 1 Test 1 Test

  36. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  37. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  38. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  39. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  40. Supermutants - Mutant Kill Matrix 6 Test 1 Test 2

    Test 3 Test 4 Test 5 Test 6 Test 7 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ 7 Tests 3 Tests 3 Tests 3 Tests 3 Tests 1 Test 1 Test

  41. Case Study - Urlparse 7

  42. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite

  43. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 32,000

  44. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 13,009 32,000

  45. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 9,037 13,009 32,000

  46. Case Study - Urlparse 7 0 exec. 10000 exec. 20000

    exec. 30000 exec. 40000 exec. Full Matrix Strong Testsuite Weak Testsuite 9,037 13,009 32,000 Over 70% of Test Executions Are Saved

  47. When Are Supermutants Super? 8

  48. When Are Supermutants Super? 8 Many and Weak Tests

  49. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants

  50. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants No Fault Masking

  51. When Are Supermutants Super? 8 Many and Weak Tests Stubborn

    Mutants No Fault Masking Automatically Generated Test Suites Can Be Efficiently Analyzed

  52. 9 Summary

  53. 9 Summary

  54. 9 Summary

  55. 9 Summary

  56. 9 Summary