Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi The Theory and Practice of Software Testing: Can we Test it? Yes we Can! Gregory M. Kapfhammer† Department of Computer Science Allegheny College, Pennsylvania, USA http://www.cs.allegheny.edu/~gkapfham/ SGT Global, February 2008 †In Conjunction with Mary Lou Soffa, Kristen Walcott (UVa/CS) Suvarshi Bhadra, Joshua Geiger, Adam Smith, Gavilan Steinman, Yuting Zhang (Allegheny/CS) Featuring images from Embroidery and Tapestry Weaving, Grace Christie (Project Gutenberg) 1 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Presentation Outline 1 Software Testing Challenges 2 Structural Testing 3 Regression Testing 4 Mutation Testing 5 Future Work 6 Conclusion 2 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi The Challenge of Software Testing I shall not deny that the construction of these testing programs has been a major intellectual effort: to convince oneself that one has not overlooked “a relevant state” and to convince oneself that the testing programs generate them all is no simple matter. The encouraging thing is that (as far as we know!) it could be done. Edsger W. Dijkstra, Communications of the ACM, 1968 Important Question: What are your software development and testing challenges? What are your best solutions? 3 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Modern Software is Complex Complex source code, database, files, and network communication Can we increase reliability by simplifying software? 4 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Defect Locations Virtual P Machine Graphical Interface Database File System System Operating Byte Code Input a 5 print ... exit Final Result: 45 Output Defects may exist in the individual components or the interactions 5 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Approaches to Software Testing Testing Structural Testing Specification Testing Regression Testing Mutation Testing isolates defects and establishes a confidence in the correctness of a software application 6 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Presentation Outline 1 Software Testing Challenges 2 Structural Testing 3 Regression Testing 4 Mutation Testing 5 Future Work 6 Conclusion 7 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi What is a Test Case? Input Method Under Test Output Test Oracle Expected Output Verdict Tear Down Set Up Overview Test suite executor (JUnit) runs each test case independently Each test invokes a method within the program and then compares the actual and expected output values 8 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Test Coverage Monitoring Program Instrumentation Adequacy Criterion Residual Test Requirements Cummulative Adequacy Calculator Instrumented Program Test Suite Execution Covered Requirements Overview Structural adequacy criteria focus on the coverage of nodes, edges, paths, and definition-use associations Instrumentation probes track the coverage of test requirements 9 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Calculating the Coverage of a Test Program Instrumentation Test Suite Adequacy Criterion Instrumented Program Test Coverage Monitoring Instrumented Test Suite Coverage Results Adequacy Calculation Test Requirements Adequacy Measurements Database Calculating Coverage Use instrumentation probes to capture and analyze a test suite’s coverage of the program Regression Testing The adequacy measurements can be used to support both test suite reduction and prioritization 10 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Presentation Outline 1 Software Testing Challenges 2 Structural Testing 3 Regression Testing 4 Mutation Testing 5 Future Work 6 Conclusion 11 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Finding the Overlap in Coverage T2 T3 T6 T9 R R1 R2 T4 T8 T11 R3 T12 R4 T1 R4 R4 R4 T5 R4 R4 T10 R4 T11 R5 T5 R5 R6 R6 T10 R7 R7 T7 R7 T10 Rj → Ti means that requirement Rj is covered by test Ti T = T2, T3, T6, T9 covers all of the test requirements May include the remaining tests so that they can redundantly cover the requirements 12 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Reducing and Prioritizing the Tests Begin Coverage Report End VSRT Repeat Program Reduction or Prioritization Original Test Suite Modified Test Suite Test Suite Execution Testing Results GRT Repeat Regression Testing Overview Reduction creates a smaller test suite that covers the same requirements as the original suite. Prioritization re-orders the tests so that they cover the requirements more effectively. Techniques use heuristics to solve NP-complete problems. 13 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Evaluating a Test Prioritization Testing Time . . . Covered Test Reqs acements T1 Done Tn−1 Done Tn Done Cover R(T1 ) Cover n−1 i=1 R(Ti ) Cover R(T) Area t(n) 0 C(T, t) C(T , t) (t) Prioritize to increase the CE of a test suite CE = Actual Ideal ∈ [0, 1] 14 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Characterizing a Test Suite Test Information Test Case Cost (sec) Requirements R1 R2 R3 R4 R5 T1 5 T2 10 T3 4 Total Testing Time = 19 seconds Formulating the Metrics CE considers the execution time of each test while CEu assumes that all test cases execute for a unit cost 15 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Coverage Effectiveness Values Calculating CE and CEu Ordering CE CEu T1 T2 T3 .3789 .4 T1 T3 T2 .5053 .4 T2 T1 T3 .3789 .5333 T2 T3 T1 .4316 .6 T3 T1 T2 .5789 .4557 T3 T2 T1 .5789 .5333 Observations Including test case costs does impact the CE metric Depending upon the characteristics of the test suite, we may see CE = CEu , CE > CEu , or CE < CEu 16 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Presentation Outline 1 Software Testing Challenges 2 Structural Testing 3 Regression Testing 4 Mutation Testing 5 Future Work 6 Conclusion 17 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Mutation Testing Techniques Program Mutant Generator Mutant Programs Test Suite Execution Mutation Score Mutant Creation A mutation testing tool (e.g., µJava or Jumble) inserts defects into the program under test Question: Why are we inserting faults into the the programs that we are testing? Test Quality Goal: measure the quality of the test suite by determining whether or not it can differentiate between faulty and non-faulty programs 18 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Average Percentage of Faults Detected Program Fault Seeding Faulty Program Test Suite Execution Test Results APFD Calculator Test Suite APFD Score Fault Table Fault Seeding Use known faults or a mutation testing tool (e.g., µJava or Jumble) to insert defects into the program Determine which test(s) are able to detect the seeded faults and construct a fault table APFD Calculation A test ordering has a higher APFD score if it rapidly detects the faults 19 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Presentation Outline 1 Software Testing Challenges 2 Structural Testing 3 Regression Testing 4 Mutation Testing 5 Future Work 6 Conclusion 20 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Search-Based Test Suite Prioritization 0 20 40 60 80 100 120 0.50 0.55 0.60 0.65 0.70 0.75 Test Prioritization Index Coverage Effectiveness Value Coverage Effectiveness Values Use heuristic search (HC, SANN, GA) to prioritize the test suite 21 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Detailed Empirical Evaluations New Testing Techniques Real World Programs Systematically study the efficiency and effectiveness trade-offs of different software testing techniques 22 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
Software Testing Challenges Structural Testing Regression Testing Mutation Testing Future Work Conclusi Conclusions Concluding Remarks Software development and testing is fun and exciting! There are many new developments in research and practical tools – some of which are ready for use today! What are your favorite software testing tools and techniques? Resources Conferences: ICSE, FSE, ISSTA, ASE, ICSM, ISSRE Journals: TSE, TOSEM, IST, JSS, JSME Many articles are available online from Google Scholar http://www.cs.allegheny.edu/~gkapfham/ 23 / 23 The Theory and Practice of Software Testing: , Can we Test it? Yes we Can!
gkapfham
hirokunimaeta
yosuke_furukawa
ksudate
syumai
microcms
maaretp
ffjlabo
stewemetal
octu0
dalinaum
ivargrimstad
yukana
lara
sstephenson
dougneiner
chrislema
danielanewman
jponch
tanoku
trishagee
speakerdeck
jacobian
addyosmani
panda_program