Share
Embed
Too trivial to test? An inverse view on defect prediction to identify methods with low fault risk
Presentation of the corresponding paper published in PeerJ Computer Science at the German-speaking Software Engineering Conference in Innsbruck, Austria.
Stefan Wagner
February 28, 2020
Transcript
38 slides
-
SE 2020 Innsbruck, Austria Stefan Wagner Too trivial to test?
An inverse view on defect prediction to identify methods with low fault risk -
You can copy, share and change, film and photograph, blog,
live-blog and tweet this presentation given that you attribute it to its author and respect the rights and licences of its parts. based on slides by @SMEasterbrook und @ethanwhite -
• Test resources are limited ➞ Need for prioritising testing
activities • Defect prediction overall with little practical usage • Cross-project defect prediction even less 4 Motivation -
Identify code regions with low fault risk 5 Idea: Inverse
Defect Prediction (IDP) FAULTY NON-FAULTY (Traditional) Defect Prediction OTHER LOW FAULT RISK Inverse Defect Prediction -
10 All Methods Compute Metrics Remove Faulty Methods Faulty Methods
Non-Faulty Methods Compute Metrics Merge Multiple Occurrences Method Fixes -
11 All Methods Compute Metrics Remove Faulty Methods Faulty Methods
Non-Faulty Methods Compute Metrics Merge Multiple Occurrences Method Fixes -
12 All Methods Compute Metrics Remove Faulty Methods Faulty Methods
Non-Faulty Methods Compute Metrics Merge Multiple Occurrences Method Fixes Address Data Imbalance -
13 All Methods Compute Metrics Remove Faulty Methods Faulty Methods
Non-Faulty Methods Compute Metrics Merge Multiple Occurrences Method Fixes Compute Association Rules Address Data Imbalance -
• Identify rules in a large dataset of transactions •
Rules describe implications between items { antecedent } ! { consequent } • Properties of rules • support (significance) • confidence (precision) 14 Association Rule Mining -
15 Method Properties (Items) Status setX(…) { IsSetter, NoNullChecks, NoLoops,
NoConditions, … } Not faulty toString() { IsToString, NoNullChecks, NoLoops, … } Not faulty computeResult() { NoStringLiterals, NoSynchronizedBlocks, … } Faulty convert(…) { NoLoops, NoAssignments, NoArrayAccesses, … } Faulty getX() { IsGetter, NoLoops, NoConditions, NoAssignments, … } Not faulty getY() { IsGetter, NoLoops, NoConditions, NoAssignments, … } Not faulty checkArgs(…) { NoAssignments, NoSynchronizedBlocks, … } Not faulty Transactions -
16 Rule { NoLoops } ! { Not faulty} {
NoLoops, NoConditions, NoAssignments } ! { Not faulty } { IsSetter, NoConditions, NoAssignments, NoStringLiterals } ! { Not faulty } { LessThan3Sloc, NoConditions } ! { Not faulty } … Rules -
17 Method Properties (Items) Status setX(…) { IsSetter, NoNullChecks, NoLoops,
NoConditions, … } Not faulty toString() { IsToString, NoNullChecks, NoLoops, … } Not faulty computeResult() { NoStringLiterals, NoSynchronizedBlocks, … } Faulty convert(…) { NoLoops, NoAssignments, NoArrayAccesses, … } Faulty getX() { IsGetter, NoLoops, NoConditions, NoAssignments, … } Not faulty getY() { IsGetter, NoLoops, NoConditions, NoAssignments, … } Not faulty checkArgs(…) { NoAssignments, NoSynchronizedBlocks, … } Not faulty … Rule: { NoLoops } ! { Not faulty } Support: 4/7 = 57.1 % Confidence: 4/5 = 80.0 % -
18 All Methods Compute Metrics Remove Faulty Methods Faulty Methods
Non-Faulty Methods Compute Metrics Merge Multiple Occurrences Method Fixes IDP Classifier Select top n rules Compute Association Rules Address Data Imbalance -
RQ 1 RQ 2 RQ 3 21 Research Questions How
many faults do methods classified as «low fault risk» contain? How large is the fraction of the code base consisting of methods classified as «low fault risk»? Is a trained classifier for methods with low fault risk generalizable to other projects? -
RQ 1 RQ 2 RQ 3 22 Research Questions How
many faults do methods classified as «low fault risk» contain? How large is the fraction of the code base consisting of methods classified as «low fault risk»? Is a trained classifier for methods with low fault risk generalizable to other projects? -
25 Proportion of Low-Fault-Risk Methods Within-Project Predictions: Chart Closure Lang
Math Mockito Time 80% 34% 17% 29% 29% 44% Method- based Chart Closure Lang Math Mockito Time 78% 25% 5% 14% 11% 16% SLOC -
RQ 1 RQ 2 RQ 3 26 Research Questions How
many faults do methods classified as «low fault risk» contain? How large is the fraction of the code base consisting of methods classified as «low fault risk»? Is a trained classifier for methods with low fault risk generalizable to other projects? -
27 Faulty Methods in Low-Fault-Risk Methods Within-Project Predictions: Chart Closure
Lang Math Mockito Time 0.1% 0.6% 1.1% 0.5% 0.2% 0.1% -
28 Fault-Density Reduction Factor Within-Project Predictions: proportion of low-fault-risk methods
out of all methods proportion of faulty low-fault-risk methods out of all faulty methods Example: 40% of all methods are classified as «low fault risk» and 10% of all methods are faulty low-fault-risk methods ! factor = 4 -
29 Fault-Density Reduction Factor Within-Project Predictions: 0 5 10 15
20 Chart Closure Lang Math Mockito Time 1.5 2.6 3.4 3.1 3.2 4.4 4.3 7.1 7 10.9 4.4 4.5 Methods SLOC Threshold -
RQ 1 RQ 2 RQ 3 30 Research Questions How
many faults do methods classified as «low fault risk» contain? How large is the fraction of the code base consisting of methods classified as «low fault risk»? Is a trained classifier for methods with low fault risk generalizable to other projects? -
32 Proportion of Low-Fault-Risk Methods Cross-Project Predictions: Chart Closure Lang
Math Mockito Time 18% 22% 27% 23% 25% 32% Chart Closure Lang Math Mockito Time 7% 7% 9% 8% 8% 11% Methods SLOC -
33 Fault-Density Reduction Factor Cross-Project Predictions: 0 5 10 15
20 Chart Closure Lang Math Mockito Time 1.4 5.8 6.1 1.6 4.4 3.3 4.2 18.3 16.3 4 13.6 8.3 Methods SLOC Threshold -
34 Fault-Density Reduction Factor Cross-Project Predictions: 0 5 10 15
20 Chart Closure Lang Math Mockito Time 1.4 5.8 6.1 1.6 4.4 3.3 4.2 18.3 16.3 4 13.6 8.3 Methods (cross) SLOC (cross) Methods (within) SLOC (within) Threshold -
• Inverse Defect Prediction can successfully predict low-fault-risk methods •
~20–40% of the methods (5–20% of SLOC) classified as «low fault risk» • Applicable for cross-project predictions 35 Conclusion -
# Rule Support Confidence # 1 { NoMethodInvocations, SlocLessThan4, NoArithmeticOperations,
NoNullLiterals } ) { NotFaulty } 10.43% 96.76% 2 { NoMethodInvocations, SlocLessThan4, NoArithmeticOperations } ) { NotFaulty } 11.03% 96.09% 3 { NoMethodInvocations, SlocLessThan4, NoCastExpressions, NoNullLiterals } ) { NotFaulty } 10.43% 95.43% 4 { NoMethodInvocations, SlocLessThan4, NoCastExpressions, NoInstantiations } ) { NotFaulty } 10.13% 95.31% 5 { NoMethodInvocations, SlocLessThan4, NoCastExpressions } ) { NotFaulty } 11.03% 94.85% -
Prof. Dr. Stefan Wagner e-mail [email protected] phone +49 (0) 711
685-88455 WWW www.iste.uni-stuttgart.de/se Twitter prof_wagnerst ORCID 0000-0002-5256-8429 Institute of Software Technology Slides are available at www.stefan-wagner.biz. Joint work with Rainer Niedermayr and Tobias Röhm -
Pictures used in this slide deck Trivial Pursuit by Bram
Cymet published under CC BY-NC 2.0 (https://flic.kr/p/ 67AFso)