SQAPlanner: Generating Data-Informed Software Quality Improvement Plans -- A Journal-First Presentation

Transcript

1. 1 SQAPlanner: Generating Data-informed Software Quality Improvement Plans @klainfo http://chakkrit.com Jirayus Jiarpakdee Kla 
 Tantithamthavorn John Grundy Accepted at IEEE Transactions on Software Engineering (TSE) Dilini Rajapaksha Christoph Bergmeir Wray Buntine
2. Challenges in SQA Practices Software defects are costly and prevalent, 
 but hard to predict and to prevent. With millions LOCs/project, practitioners do not know: 
 Which files are likely to be defective in the future? SQA practices (pre-testing, code review) are adopted, 
 but very time-consuming and expensive.
3. 3 Defect Prediction Models: An Overview A classifier to predict if a file will be defective in the future Prediction Help developers find defects faster Timeline V2 Testing Data Defect 
 Models Training Data V1 Important Factors Help managers develop quality improvement plans
4. Microsoft’s Code Defect AI 4 Reference: https://www.microsoft.com/en-us/ai/ai-lab-code-defect Insight: lines of code contributed the most to the prediction. Explainable, But Not Actionable!!
5. Limitations: Explainable, But Not Actionable!! 5 Reference: https://www.microsoft.com/en-us/ai/ai-lab-code-defect First, practitioners still don’t know what should they do to decrease the risk of having defects. Second, practitioners still don’t know a risk threshold for each metric (e.g., how small is a file size should be?)
6. 6 Study Design
7. 7 RQ1-RQ2: Practitioners’ Perceptions A data-informed SQA planning tool is needed to support QA teams make better data-informed decision- and policy-making. R34: “Lessons learnt from projects are documented and common mistakes are included in review checklists to ensure that they are not repeated.”) • They used documentation and review checklists. • Some activities involve human heuristics in decision-making. How do developers conduct SQA Planning?
8. SQAPlanner Model Instance {LOC > 100 & Ownership < 0.8} DEFECT A rule based explanation 8 Defect Prediction Buggy Clean Why a file is predicted as defective? “A file is predicted as defective, 
 since LOC > 100 and Ownership < 0.8” SQAPlanner: Making Defect Prediction Actionable A rule-based model-agnostic approach to generate explanations and actionable guidance Help developers understand the most important aspects that are associated with defects. Help developers understand the risk threshold (how small it should be?). Generate Explanations Magnum 
 Opus Build SQAPlanner Crossover+ 
 Mutation Generate
9. 9 RQ3-RQ5: Experimental Evaluation Our SQAPlanner generates explanations that achieve the highest coverage, confidence, lift, and consistency when compared to LORE and Anchor. Consistency 
 How stable are the rule-based explanations? Effectiveness 
 How accurate are the rule-based explanations?
10. 10 RQ6-RQ7: Practitioners’ Perceptions 80% of our respondents agree that the visualization of our SQAPlanner is better for providing actionable guidance when compared to the visualization of Microsoft’s Code Defect AI
11. 11 Take-Away Messages Actionability is much needed in Software Engineering. 1 SQAPlanner is one of the importance advancements towards Actionable Software Analytics. 2 • Follow-up after the talk: chakkrit@monash.edu • Website: http://chakkrit.com • Twitter: @klainfo • Materials: http://xai4se.github.io