M. Kapfhammer∗ and Jonathan Miller Kauffman† Department of Computer Science Allegheny College ∗http://www.cs.allegheny.edu/∼gkapfham/ †https://sites.google.com/a/allegheny.edu/jonathan-kauffman/ 15th International Conference on Software Engineering and Applications December 14 - 16, 2011
Accessing the Presentation Scan this QR Code with your smartphone! ... or, visit this Web site: http://www.cs.allegheny.edu/˜gkapfham/sea2011.pdf ... or, ask us for a USB drive! Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Everywhere Computer Server Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Everywhere Computer Server Program Computer Server Program Desktop Computer Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Everywhere Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Everywhere Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Everywhere Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program Scientific Device Program Household Appliance Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Everywhere Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program Scientific Device Program Household Appliance Program Network Router Program Scientific Device Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Complex Computer Software Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Complex Computer Software Lines of Code Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Complex Computer Software Lines of Code Numerous Features Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Complex Computer Software Lines of Code Numerous Features Feature Interactions Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Complex Computer Software Lines of Code Numerous Features Feature Interactions Execution Environments Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Complex Computer Software Lines of Code Numerous Features Feature Interactions Execution Environments Software entities are more complex for their size than per- haps any other human construct - Frederick P. Brooks, Jr. Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Evolving Execution Environment Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Evolving Execution Environment Program Execution Environment Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Evolving Execution Environment Program Execution Environment Program Program Changed because of the addition of a new feature or the correction of a defect Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Evolving Execution Environment Program Execution Environment Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Evolving Execution Environment Program Execution Environment Program Execution Environment Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Software is Evolving Execution Environment Program Execution Environment Program Execution Environment Program Execution Environment Changed due to an up- grade in a kernel, device driver, or virtual machine Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Regression Testing to the Rescue Computer Software Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Regression Testing to the Rescue Computer Software Pervasive Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Regression Testing to the Rescue Computer Software Pervasive Complex Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Regression Testing to the Rescue Computer Software Pervasive Complex Evolving Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Regression Testing to the Rescue Computer Software Pervasive Complex Evolving Regression Testing supports the efficient construction of pervasive software that is complex and rapidly evolving Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Case? Method Under Test Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Case? Method Under Test Input Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Case? Method Under Test Input Output Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Case? Method Under Test Test Set Up Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Case? Method Under Test Test Set Up Input Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Case? Method Under Test Test Set Up Input Output Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Test Verdict The test case passes and the code is correct! Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Test Verdict The test case fails and a defect is found! Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Suite? T1 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Suite? T1 T2 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Suite? T1 T2 T3 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Suite? T1 T2 T3 T4 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Suite? T1 T2 T3 T4 . . . Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Suite? T1 T2 T3 T4 . . . Tn Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Suite? T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Suite? T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Suite? T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Suite? T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit Apache Ant Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
What is a Test Suite? T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit Apache Ant Eclipse Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Management T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique What if Some Test Cases are More Effective? T3 Tn Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique What if Some Test Cases are More Effective? T3 Tn Prioritization T3 Tn T1 T4 . . . T2 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique What if Some Test Cases are More Effective? T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn What if Some Test Cases are Redundant? T1 T2 T3 T4 . . . Tn Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn What if Some Test Cases are Redundant? T1 T2 T3 T4 . . . Tn Reduction T4 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn What if Some Test Cases are Redundant? T1 T2 T3 T4 . . . Tn Reduction T4 T1 T2 T3 T4 . . . Tn Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn T1 T2 T3 T4 . . . Tn Reduction T4 T1 T2 T3 T4 . . . Tn What if Only Certain Tests are Needed? T2 Tn Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn T1 T2 T3 T4 . . . Tn Reduction T4 T1 T2 T3 T4 . . . Tn What if Only Certain Tests are Needed? T2 Tn Selection T1 T2 T3 T4 . . . Tn Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Model of Regression Testing Start Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Model of Regression Testing Start Coverage Report Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Use the Coverage Report During the Next Round of Regression Testing Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Use the Same Test Suite for the Next Round of Regression Testing Standard Repeat Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Standard Repeat Make a New Test Suite for the Next Round of Regression Testing Version Specific Repeat Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Standard Repeat Make a New Test Suite for the Next Round of Regression Testing Version Specific Repeat Our Tools Support All of the Phases in this Model! Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Adequacy T1 T2 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Adequacy T1 T2 T3 T4 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Adequacy T1 T2 T3 T4 T5 T6 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 Run Test Case Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 Passing Test Case: OA = OE Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 Run Test Case Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Stop Running T Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Stop Running T T8 T9 T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Stop Running T T8 T9 T10 T8 T9 T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Stop Running T T8 T9 T10 T8 T9 T10 Continue Running T Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Stop Running T T8 T9 T10 T8 T9 T10 Continue Running T T8 T9 T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework Cobertura Test Coverage Monitor Proteja Test Suite Manager Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework Cobertura Test Coverage Monitor Proteja Test Suite Manager T1 Run Test Case Collect Per-Test Case Coverage Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hands-on: Preparing the Applications and Tools 1 Compile the Sudoku Case Study Application cd proteja-installation-directory cd apps/SK ant init ant compile cd ../../ Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hands-on: Preparing the Applications and Tools 1 Compile the Sudoku Case Study Application cd proteja-installation-directory cd apps/SK ant init ant compile cd ../../ 2 Compile the Proteja Test Coverage Monitor cd ../../ ant init ant build Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hands-on: Executing Tests and Monitoring Coverage 1 Run Sudoku’s Test Suite Without Coverage Monitoring cp data/settingsFiles/SK/* . ant addAnnotations ant proteja Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hands-on: Executing Tests and Monitoring Coverage 1 Run Sudoku’s Test Suite Without Coverage Monitoring cp data/settingsFiles/SK/* . ant addAnnotations ant proteja 2 Run Sudoku’s Test Suite With Coverage Monitoring ant proteja -Dcobertura=true -DtestTimer=true -DapplicationName=Sudoku Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: Configuring Test Suite Execution Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: Configuring the Test Coverage Monitor Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: Running the Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: Performing Test Coverage Monitoring Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: Viewing a Test Coverage Report Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: Viewing a Test Case Timings Report Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 1, 2, 3 CE = 0.3789 Prioritize to increase the CE of a test suite CE = Actual Ideal ∈ [0, 1] Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 1, 2, 3 CE = 0.3789 1,2,3 Test Orderings Original ordering exhibits poor effectiveness score - CE = 0.3789 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 1, 3, 2 CE = 0.5053 1,2,3 Test Orderings 1,2,3 1,3,2 Different ordering improves the effectiveness score - CE = 0.5053 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 2, 3, 1 CE = 0.4316 1,2,3 Test Orderings 1,2,3 1,3,2 1,2,3 1,3,2 2,3,1 Some orderings have less improved scores - CE = 0.4316 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 3, 1, 2 CE = 0.5789 1,2,3 Test Orderings 1,2,3 1,3,2 1,2,3 1,3,2 2,3,1 1,2,3 1,3,2 2,3,1 3,1,2 Search-based techniques may have some desirable characteristics Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 2.66 Proceed incrementally, picking the test case with the lowest ratio value for the uncovered requirements Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hands-on: Running the Greedy Algorithms 1 Load the Regression Testing Techniques cd modificare-installation-directory R source(‘‘FDP Start.R’’) zLoadFit() zLoadGRD() zLoadHC() zLoadGA() zLoadProduceOrderingReductionDataFile() Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hands-on: Running the Greedy Algorithms 3 Run the Greedy Prioritization Algorithm x2<-GRD(coverageMatrixName= "reqMatrices/StatementCoverageMatrices/ Sudoku 1 line true Coverage.dat", timingsFileName="reqMatrices/ TimingsFiles/Sudoku Timing.dat") Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hands-on: Running the Greedy Algorithms 4 Map the Test Indices to Test Names y1<-produceOrderingReductionDataFile (testSuite=x1$Ord, timingsFile="reqMatrices/ TimingsFiles/Sudoku Timing.dat") y2<-produceOrderingReductionDataFile (testSuite=x2$Ord, timingsFile="reqMatrices/ TimingsFiles/Sudoku Timing.dat") Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hands-on: Running the Greedy Algorithms 5 Store the Test Case Timings write.table(y1,file="path-to-proteja/ SK ordering1.dat", row.names=FALSE, col.names=FALSE,quote=FALSE) write.table(y2,file="path-to-proteja/ SK ordering2.dat", row.names=FALSE, col.names=FALSE,quote=FALSE) Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hands-on: Running the Greedy Algorithms 6 Execute the Reduced Test Suite cd proteja-installation-directory ant modifyTestSuite -DmodificationFile= "SK ordering1.dat" ant proteja Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hands-on: Running the Greedy Algorithms 7 Execute the Prioritized Test Suite cd proteja-installation-directory ant modifyTestSuite -DmodificationFile= "SK ordering2.dat" ant proteja Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: Running the Greedy Reducer Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: Running the Greedy Prioritizer Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: Test Index to Test Name Mapping Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: Storing the Test Case Timings Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: The Reduced Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: The Prioritized Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: Preparing the Reduced Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: Viewing the Reduced Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: Running the Reduced Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Limitations of Greedy Algorithms T1 T3 T2 T4 Possible configuration of the coverage report Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 Possible configuration of the coverage report Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 Possible configuration of the coverage report Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 Execution time of the test cases may mislead greedy Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 Execution time of the test cases may mislead greedy Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point T1, T2, T3, T4, T5 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point T1, T2, T3, T4, T5 T2, T1, T3, T4, T5 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point T1, T2, T3, T4, T5 T2, T1, T3, T4, T5 T3, T2, T1, T4, T5 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point T1, T2, T3, T4, T5 T2, T1, T3, T4, T5 T3, T2, T1, T4, T5 T4, T2, T3, T1, T5 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hill Climbing Algorithm fitness time Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hill Climbing Algorithm fitness time T Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hill Climbing Algorithm fitness time T T Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hill Climbing Algorithm fitness time T T T Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hill Climbing Algorithm fitness time T T T T Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hill Climbing Algorithm fitness time T T T T T Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Hands-on: Running the Search-Based Algorithms 1 Run the Hill Climbing Prioritization Algorithm HC FA(lFM=makeLogFM(read.table( "reqMatrices/StatementCoverageMatrices/ Sudoku 1 line true Coverage.dat")), NG="NG FS",Seed=100) Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: Running the Hill Climbing Algorithm Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Screenshot: Running the Genetic Algorithm Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Fault Localization mid() { int x,y,z,m; T1 T2 T3 T4 T5 T6 T7 T8 1: read(”Enter 3 numbers:”,x,y,z); 2: m=z; 3: if(y<z) 4: if(x< y) 5: m=y; 6: else if(x<z) 7: m=y; // *** bug *** 8: else 9: if(x>y) 10: m=y; 11: else if (x>z) 12: m=z; 13: print(”Middle number is:”,m); } Pass/Fail Status P P P P P P F F Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Experimentation How Do I Evaluate Regression Testing Methods? Start Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models Iteratively Perform Visualization and Statistical Analysis Repeat End Conduct Experiments with Additional Programs, Test Suites, and Techniques Repeat Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models Repeat End Conduct Experiments with Additional Programs, Test Suites, and Techniques Repeat Our tools support all of these tasks! Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Greedy and hill climbing produce comparable orderings Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime However, hill climbing is slightly more efficient than greedy Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Greedy produces a slightly better ordering than hill climbing Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime But the hill climbing algorithm executes over four times faster! Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime A small fitness increase may result in a large runtime increase Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Future Work Concluding Remarks • Comprehensive framework for regression testing • Interesting empirical results demonstrate trade-offs • Free/open source tools are available for download • http://proteja.googlecode.com • http://modificare.googlecode.com Future Work • Add new algorithms for regression testing • Conduct experiments with more case study applications • Develop statistically meaningful empirical results Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
Future Work Concluding Remarks • Comprehensive framework for regression testing • Interesting empirical results demonstrate trade-offs • Free/open source tools are available for download • http://proteja.googlecode.com • http://modificare.googlecode.com Future Work • Add new algorithms for regression testing • Conduct experiments with more case study applications • Develop statistically meaningful empirical results Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
M. Kapfhammer∗ and Jonathan Miller Kauffman† ∗http://www.cs.allegheny.edu/∼gkapfham/ †https://sites.google.com/a/allegheny.edu/jonathan-kauffman/ Thank you for your attention! Contact us with questions and/or comments!