Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Set Covers, Knapsacks, and Regression Testing Techniques Gregory M. Kapfhammer† Department of Computer Science Allegheny College, Pennsylvania, USA http://www.cs.allegheny.edu/~gkapfham/ Madras Christian College, February 2008 †In Conjunction with Mary Lou Soffa, Kristen Walcott (UVa/CS) Suvarshi Bhadra, Joshua Geiger, Adam Smith, Gavilan Steinman, Yuting Zhang (Allegheny/CS) Featuring images from the WikiMedia Commons 1 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions The Challenge of Software Testing I shall not deny that the construction of these testing programs has been a major intellectual effort: to convince oneself that one has not overlooked “a relevant state” and to convince oneself that the testing programs generate them all is no simple matter. The encouraging thing is that (as far as we know!) it could be done. Edsger W. Dijkstra, Communications of the ACM, 1968 Additional Challenge: understanding the fundamental difficulties associated with practical testing techniques 3 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions NP-Complete Problems Minimal Set Cover 0/1 Knapsack Question: what are the connections between the theory and practice of computer science? 4 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Halting Problem Program Halting Detector Inputs Yes! No! Question: What approach can we take in order to completely implement the halting detector? You can assume any existing hardware platform (e.g., fast multi-core processor) or software application (e.g., compiler). 5 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Minimal Set Cover Problem S1 E1 E2 E3 S2 E4 E5 E6 S3 E7 E8 E9 S4 Sets contain elements (e.g., S2 → E4 means that set S2 contains the element E4 ) Question: Can you find a subset of the sets that will contain all of the elements? This problem is NP-complete (see Garey and Johnson) and yet it has many practical applications in software testing 6 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions 0/1 Knapsack Problem Knapsack Capacity = 12 Cost = 4 Benefit = 6 Cost = 10 Benefit = 9 Cost = 7 Benefit = 4 Cost = 9 Benefit = 8 Question: Can you select items so that you maximize the benefit while ensuring that the cost does not exceed the capacity? This problem is NP-complete (see Garey and Johnson) and yet it also has many practical applications in both software and finance 7 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions 0/1 Knapsack Problem Knapsack Capacity = 12 Cost = 4 Benefit = 6 Cost = 10 Benefit = 9 Cost = 7 Benefit = 4 Cost = 9 Benefit = 8 Question: Can you select items so that you maximize the benefit while ensuring that the cost does not exceed the capacity? This problem is NP-complete (see Garey and Johnson) and yet it also has many practical applications in both software and finance 8 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Approaches to Software Testing Specification Testing Testing Random MTTF(P) = k Reliability Software Input Freq Testing Structural Input Space Testing isolates defects and establishes a confidence in the correctness of a software application 10 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Where are Defects Located? System Operating File System Virtual P Machine Graphical Interface Database Byte Code Input a 5 print ... exit Final Result: 45 Output Defects may exist in the individual components or the interactions 11 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions What is a Test Case? Input Method Under Test Output Test Oracle Expected Output Verdict Tear Down Set Up Test suite executor runs each test case independently Each test invokes a method within the program and then compares the actual and expected output values 12 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Using Tests to Find Faults Test Case Faults f1 f2 f3 f4 f5 T1 × × T2 × × T3 × × × T4 × × × T5 × × The Importance of Test Ordering: T3 , T4 , T1 , T2 , T5 detects faults more rapidly than T1 , T2 , T3 , T4 , T5 Since we do not have a priori knowledge of the faults that exist within a program, we must use a proxy like coverage 13 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Test Coverage Monitoring Program Instrumentation Adequacy Criterion Residual Test Requirements Cummulative Adequacy Calculator Instrumented Program Test Suite Execution Covered Requirements Structural adequacy criteria focus on the coverage of nodes, edges, paths, and definition-use associations Instrumentation probes track the coverage of test requirements 14 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Finding the Overlap in Coverage T2 T3 T6 T9 R R1 R2 T4 T8 T11 R3 T12 R4 T1 R4 R4 R4 T5 R4 R4 T10 R4 T11 R5 T5 R5 R6 R6 T10 R7 R7 T7 R7 T10 Rj → Ti means that requirement Rj is covered by test Ti T = T2 , T3 , T6 , T9 covers all of the test requirements Test suite reduction discards the test cases that redundantly cover the test requirements 15 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Regression Test Suite Prioritization Begin Coverage Report End VSRT Repeat Program Test Suite Prioritization Original Test Suite Modified Test Suite Test Suite Execution Testing Results GRT Repeat Overview Prioritization re-orders the tests so that they cover the requirements more effectively GRT uses the same prioritization across multiple runs of the test suite whereas VSRT creates a new prioritization for each test run 16 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Time-Aware Orderings (Fault Data) f1 f2 f3 f4 f5 f6 f7 f8 T1 X X X X X X X T2 X T3 X X T4 X X X T5 X X X T6 X X X It is very common to confront a testing time budget Question: If fault information is known and there is a testing time limit, then what is the best ordering? 19 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Time-Aware Orderings (Faults and Costs) # Faults Time Cost Avg. Faults per Min. T1 7 9 0.778 T2 1 1 1.000 T3 2 3 0.667 T4 3 4 0.750 T5 3 4 0.750 T6 3 4 0.750 When test case cost varies, then some tests are able to detect fault more rapidly than the others Question: What is the best ordering for this test suite? 20 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Time-Aware Orderings (Comparison) Time Limit: 12 minutes Fault Time APFD Intelligent T1 T2 T2 T5 T3 T1 T4 T4 T3 T5 Tot. Faults 7 8 7 8 Tot. Time 9 12 10 11 The existence of a time limit prevents the use of traditional minimal set cover solvers that only look at overlap When fault information is not available, we can use coverage 21 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Comparing Time-Aware Prioritizers Traditional 0/1 Knapsack Solvers Genetic Algorithm Techniques Empirical Results: prioritizers that consider coverage overlap take longer to re-order, but they arrive at good orderings 22 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Search-Based Test Suite Prioritization 0 20 40 60 80 100 120 0.50 0.55 0.60 0.65 0.70 0.75 Test Prioritization Index Coverage Effectiveness Value Coverage Effectiveness Values Use heuristic search (HC, SANN, GA) to prioritize the test suite 24 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Detailed Empirical Evaluations Synthetic Test Suites Real World Programs Conduct experiments to systematically study the trade-offs associated with testing techniques by using both synthetic coverage and real world applications 25 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Concluding Remarks Minimal Set Cover 0/1 Knapsack Establishes a connection between practical regression testing challenges and NP-complete problems Many approaches to testing are now ready for integration into frameworks such as JUnit 27 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions Personal Reflections So now, come back to your God! Act on the principles of love and justice, and always live in confident dependence on your God. Hosea 12:6 (New Living Translation) Please keep in touch! http://www.cs.allegheny.edu/~gkapfham/ 28 / 28 Set Covers, Knapsacks, and , Regression Testing Techniques
gkapfham
lukestringer90
psj59129
jpt_corp_official
fornewid
nacatl
workspace93
m_coder
dalinaum
devoc
murabayashi
alpacatc
akarin
marcduiker
dougneiner
phodgson
hannesfritz
productmarketing
aki_iinuma
reverentgeek
jacobian
samanthasiow
brettharned
bermonpainter
scottboms