An Empirical Evaluation of Frequency Based Statistical Models for Estimating Killable Mutants

ESEM 2024

Transcript

1. An Empirical Evaluation of Frequency Based Statistical Models for Estimating Killable Mutants Konstantin Kuznetsov, Alessio Gambi, Saikrishna Dhiddi, Julia Hess, Rahul Gopinath Or: Can We Estimate Equivalent Mutants in a Program?
2. Bugs 2
4. 4 Input ✓ ✘ Testing @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/ {}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500
5. 5 When To Stop Testing? @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/ {}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 Inputs
6. 6 When To Stop Testing? @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/ {}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 Inputs When you have very few defects left.
7. 7 How To Estimate Residual Defects? @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/ {}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 Use Fault Seeding!
8. 8 How To Estimate Residual Defects? @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/ {}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 Fault Seeding : Mutation Analysis
9. Mutation analysis Target program Test suite Problem: Equivalent Mutants!
10. Mutation analysis Target program Test suite Problem: Equivalent Mutants!
11. Probabilistic Model - detection probability of mutant Mi - e ff ect of test Xj with prob density
12. Probabilistic Model - detection probability of mutant Mi - e ff ect of test Xj with prob density frequency of 𝑀 being killed Q0 = # undetected mutants frequency counts Qk
13. Frequency based estimators Marcel Böhme. STADS: Software testing as species discovery., TSE, 2018 Used for estimating reachable but yet to be covered elements remaining for coverage evaluation. The idea in species richness estimation is that the number of species that did not show up in any sampling unit we can estimated by considering the next rarest species, such as species detected only once (singleton), twice (doubleton), and so on.
14. Incidence estimators - Chao - iChao - Jackknife (of order up to fi ve) - Incidence coverage estimator (ICE) - Bootstrap - Zelterman Abundance estimators - Chao-Bungie - PCG Poisson-compound Gamma model - Unconditional nonparametric maximum likelihood estimator (UNPMLE) - Penalized nonparametric maximum likelihood estimator (PNPMLE) Frequency Based Estimators
15. Frequency Based Estimators
16. Mutant generation PIT v 1.4.9 “Default” mutators: Conditional boundary Increments Negative Conditional Math Invert Negatives Void Method Calls Empty returns False returns True returns Null returns Primitive returns Executed with parallelized 10 seconds timeout per unit test
17. Test subjects Project Release kLOC # mutants commons-net 3.6 20 5764 commons-math 3.6_RC2 100 47881 commons-lang 3.8_RC1 28 13061 commons-io 2.6-RC3 10 3273 commons-imaging 1.0-alpha1-RC3 31 11597 commons-dbcp 2.6.0 14 4230 commons-csv 1.7 2 635 commons-con fi guration 2.5 28 6279 commons-compress 1.18 24 9566 commons-collections 4.4 29 8309
18. Test Suites Random DynaMOSA Original biased? • generate unit tests for each class for 60 seconds • generate assertions for 120 seconds • minimize the tests for 300 seconds
 EvoSuite Manual unit tests
19. Manual Mutants Classification
20. Manual Mutants Classification 100 random samples
21. Manual Mutants Classification 100 random samples
22. Manual Mutants Classification A B C Cohen’s Kappa Population proportion p = x / N 0.48 100 random samples Validated
23. Manual Mutants Classification A B C Cohen’s Kappa Proportions 0.48 100 random samples
24. What is the accuracy of frequency based estimators in 
 predicting killable mutants? Chao estimator for Original, Random, and DynaMOSA, methods                    
25. What is the accuracy of frequency based estimators in 
 predicting killable mutants? Original test suite, method sampling units What is the accuracy of frequency based estimators in 
 predicting killable mutants?
26. Are frequency based estimators affected by sampling strategies? Random test suite, method sampling units
27. DynaMOSA test suite, method sampling units Are frequency based estimators affected by sampling strategies?
28.                     Chao estimator for Original test suite method, class, manual Are frequency-based estimators affected by sampling units in mutation analysis?
29. Are frequency-based estimators affected by sampling units in mutation analysis? Original test suite, method vs class sampling units
30. • None of the estimators could consistently produce estimates that are close to the manual estimates • The estimates produced with di ff erent sampling units were inconsistent –> violated assumptions?
 • The estimators are not yet ready for use by developers. Result Summary