Are Comprehensive Quality Models Necessary for Evaluating Software Quality?

by Klaus Lochmann, Jasmin Ramadani and Stefan Wagner

Transcript

1. www.uni-stuttgart.de Stefan Wagner PROMISE 2013 Baltimore, USA 9 October 2013 Comprehensive Quality Models Necessary for Are @prof_wagnerst joined work with Klaus Lochmann and Jasmin Ramadani Evaluating ? Software Quality
2. "We deployed a bug prediction algorithm across Google, and found no identifiable change in developer behavior." Lewis et al., ICSE'13
3. "Quality is a complex and multi-faceted concept... it is also the source of great confusion." –David A. Garvin
4. Comment ratio ISO 9126 ISO 25010 Quality attribute Clone coverage Measure Cyclomatic complexity
5. Comment ratio Quality attribute Clone coverage Measure Cyclomatic complexity ? ISO 9126 ISO 25010
6. The Benchmark for Software Quality http://www.quamoco.de
7. Comment ratio Quality attribute Clone coverage Measure Cyclomatic complexity ? ISO 9126 ISO 25010
8. Comment ratio Quality attribute Clone coverage Measure Cyclomatic complexity Product factor ISO 9126 ISO 25010
9. Statically unused method Analyzability Quality attribute Measure Usefulness of method Product factor Maintainability Gendarme: Avoid Uncalled Private Code PMD: Unused Private Method Instrument
10. root C GUI object-oriented Java Base Model C# C++
12. 1 Software Product Quality Control Stefan Wagner
13. Quality Model Maintenance
14. Research Strategy
15. RQ 1: What is the performance of focused quality models built using machine learning algorithms? RQ 2: What is the performance of the focused quality models including additional expert-based measures?
16. Predictor Models Used • Random guessing • Linear regression (forward selection) • Linear regression (backward elimination) • Linear regression (bidirectional elimination) • Classification tree • Random forest
17. • Mean absolute residual (MAR) • Standardised accuracy measure (SA) • Effect size Model Comparison MAR = Pn 1 |(yi ˆ yi)| n SApi = 1 MARpi MARp0 = MARpi MARp0 sp0
18. Study Objects • 1994 Java systems from SDS repository • 15 Java systems for which we have manual measures
19. Procedure • Collection of all measures and evaluations for maintainability • Building of predictors (4-fold cross validation) • Calculation of model comparison measures
20. Results
21. 0 10 20 30 40 50 60 35 40 45 50 55 60 65 SA (percentage of improvement over random guessing) Random Forest (Forward Selection) Classification Tree (Forward Selection) Classification Tree (different complexity param.) Regression (Forward Selection) Regression (Bidirectional Elimination) Regression (Backward Elimination) SA / # of Variables # of variables
22. 0 10 20 30 40 50 50 55 60 65 Number of Variables Random Forest (Forward Selection) SA / # of Variables
23. 0 5 10 15 20 25 30 0 10 20 30 40 50 60 SA Systems without experXďFEWIHQIEWYVIW Systems with experXďFEWIHQIEWYVIW With and Without Manual Measures # of variables
24. Threats to Validity • Expert measures not included in RQ 1 • For RQ 2 only 15 systems • Set of predictors and comparison measures • Only maintainability • Only Java systems
25. Conclusions
26. • Comprehensive models to capture all the different aspects and quality factors • More focused models measuring only few measures • Focused model with 61% accuracy but only 10 measures (compared to 378) • Expert-based measures reduce accuracy • So what should we use?