A Framework to Support Research in and Encourage the Industrial Adoption of Regression Testing Techniques

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Kauffman2012a/

Transcript

1. A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques Jonathan Miller Kauffman and Gregory M. Kapfhammer Department of Computer Science Allegheny College Testing: Academic & Industrial Conference – Practice and Research Techniques Montr´ eal, Qu´ ebec, Canada April 21, 2012
2. Regression Testing T1 T2 T3 T4 . . . Tn Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
3. Regression Testing T1 T2 T3 T4 . . . Tn Developers may introduce faults when adding new functionality Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
4. Regression Testing T1 T2 T3 T4 . . . Tn Run a regression test suite to detect these faults Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
5. Regression Testing T1 T2 T3 T4 . . . Tn Need ways to manage regression test suites as they grow in size Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
6. Regression Testing T1 T2 T3 T4 . . . Tn Regression Testing Technique Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
7. Regression Testing T1 T2 T3 T4 . . . Tn Regression Testing Technique What if Some Test Cases are More Effective? T3 Tn Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
8. Regression Testing T1 T2 T3 T4 . . . Tn Regression Testing Technique What if Some Test Cases are More Effective? T3 Tn Prioritization T3 Tn T1 T4 . . . T2 Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
9. Regression Testing T1 T2 T3 T4 . . . Tn Regression Testing Technique What if Some Test Cases are More Effective? T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
10. Regression Testing T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn What if Some Test Cases are Redundant? T1 T2 T3 T4 . . . Tn Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
11. Regression Testing T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn What if Some Test Cases are Redundant? T1 T2 T3 T4 . . . Tn Reduction T4 Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
12. Regression Testing Challenges Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
13. Regression Testing Challenges Research Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
14. Regression Testing Challenges Research Replication difficult due to unavailability of artifacts Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
15. Regression Testing Challenges Research Replication difficult due to unavailability of artifacts Insufficient number of trials and inappropriate statistical analyses Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
16. Regression Testing Challenges Research Replication difficult due to unavailability of artifacts Insufficient number of trials and inappropriate statistical analyses Practice Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
17. Regression Testing Challenges Research Replication difficult due to unavailability of artifacts Insufficient number of trials and inappropriate statistical analyses Practice Unwilling to adopt techniques due to lack of empirical studies Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
18. Regression Testing Challenges Research Replication difficult due to unavailability of artifacts Insufficient number of trials and inappropriate statistical analyses Practice Unwilling to adopt techniques due to lack of empirical studies Techniques must be usable with minimal configuration Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
19. Regression Testing Challenges Research Replication difficult due to unavailability of artifacts Insufficient number of trials and inappropriate statistical analyses Practice Unwilling to adopt techniques due to lack of empirical studies Techniques must be usable with minimal configuration Address challenges by releasing two open-source tools Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
20. Proteja: Test Suite Executor & Coverage Monitor Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
21. Proteja: Test Suite Executor & Coverage Monitor P Program written in the Java programming language Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
22. Proteja: Test Suite Executor & Coverage Monitor P T JUnit test suite that exercizes program P Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
23. Proteja: Test Suite Executor & Coverage Monitor P T Test Coverage Monitoring Controls test case execution and coverage monitoring Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
24. Proteja: Test Suite Executor & Coverage Monitor P T Test Coverage Monitoring Coverage Report Identifies the program entities executed by each test case Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
25. Proteja: Test Suite Executor & Coverage Monitor P T Test Coverage Monitoring Coverage Report Coverage criteria: Statement, method, and class Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
26. Proteja: Test Suite Executor & Coverage Monitor P T Test Coverage Monitoring Coverage Report T Run test suite according to a reduction or prioritization Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
27. Modificare: Test Suite Management & Experimentation Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
28. Modificare: Test Suite Management & Experimentation Algorithms: Greedy, Hill Climbing, Random, Adap- tive Random, Simulated Annealing, Genetic Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
29. Modificare: Test Suite Management & Experimentation Coverage Report Per-test coverage information produced by Proteja Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
30. Modificare: Test Suite Management & Experimentation Coverage Report Technique Configurations Controls the execution of the reduction and prioritization algorithms Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
31. Modificare: Test Suite Management & Experimentation Coverage Report Technique Configurations Data Sets Visualized and statistically analyzed to identify trends Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
32. Modificare: Test Suite Management & Experimentation Coverage Report Technique Configurations Data Sets T Description of the reduced or prioritized test suite Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
33. Integrating Proteja and Modificare Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
34. Integrating Proteja and Modificare P, T Test Suite Executor JUnit Cobertura Coverage Report Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
35. Integrating Proteja and Modificare P, T Test Suite Executor JUnit Cobertura Coverage Report Reduction Prioritization T Data Set Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
36. Integrating Proteja and Modificare P, T Test Suite Executor JUnit Cobertura Coverage Report Reduction Prioritization T Data Set Experimentation Visualization Analysis Visualization and analysis used to identify trends Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
37. Integrating Proteja and Modificare P, T Test Suite Executor JUnit Cobertura Coverage Report Reduction Prioritization T Data Set Experimentation Visualization Analysis T executed using Proteja Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
38. Integrating Proteja and Modificare P, T Test Suite Executor JUnit Cobertura Coverage Report Reduction Prioritization T Data Set Experimentation Visualization Analysis Use any tool that produces data in the correct format Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
39. Integrating Proteja and Modificare P, T Test Suite Executor JUnit Cobertura Coverage Report Reduction Prioritization T Data Set Experimentation Visualization Analysis However, greatest benefits realized when tools used together Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
40. Empirical Results Population Size NAPSC 0.3240 0.3245 0.3250 0.3255 0.3260 0.3265 10 20 30 JodaTime (JT) Statement Original NAPSC Score: 0.2784 Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
41. Empirical Results Population Size NAPSC 0.3240 0.3245 0.3250 0.3255 0.3260 0.3265 10 20 30 JodaTime (JT) Statement Negligible NAPSC increase as population size increases Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
42. Empirical Results Population Size Runtime (seconds) 10 15 20 10 20 30 JodaTime (JT) Statement Increases in runtime are more marked Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
43. Empirical Results Similarity Metric NAPSC 0.3194 0.3195 0.3196 0.3197 Euclidean Jaccard Manhattan JodaTime (JT) Statement Original NAPSC Score: 0.2784 Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
44. Empirical Results Similarity Metric NAPSC 0.3194 0.3195 0.3196 0.3197 Euclidean Jaccard Manhattan JodaTime (JT) Statement NAPSC changes little as similarity metric is varied Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
45. Empirical Results Similarity Metric NAPSC 0.3194 0.3195 0.3196 0.3197 Euclidean Jaccard Manhattan JodaTime (JT) Statement Scores are comparable to those produced by random (0.3240 - 0.3265) Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
46. Empirical Results Similarity Metric Runtime (seconds) 80 90 100 110 120 Euclidean Jaccard Manhattan JodaTime (JT) Statement Adaptive random executes more slowly than random Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
47. Empirical Results Similarity Metric Runtime (seconds) 80 90 100 110 120 Euclidean Jaccard Manhattan JodaTime (JT) Statement Choose random because it produces comparable NAPSC scores in less time Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
48. Getting Involved Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
49. Getting Involved How can you “battle” research and practice challenges? Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
50. Getting Involved How can you “battle” research and practice challenges? Download Proteja and Modificare Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
51. Getting Involved How can you “battle” research and practice challenges? Download Proteja and Modificare Proteja: http://proteja.googlecode.com Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
52. Getting Involved How can you “battle” research and practice challenges? Download Proteja and Modificare Modificare: http://modificare.googlecode.com Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
53. Getting Involved How can you “battle” research and practice challenges? Download Proteja and Modificare Contribute new features and bug fixes Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
54. Getting Involved How can you “battle” research and practice challenges? Download Proteja and Modificare Contribute new features and bug fixes All contributions will be recognized Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
55. Getting Involved How can you “battle” research and practice challenges? Download Proteja and Modificare Contribute new features and bug fixes Perform empirical studies Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
56. Getting Involved How can you “battle” research and practice challenges? Download Proteja and Modificare Contribute new features and bug fixes Perform empirical studies Study more algorithms or configurations of individual algorithms Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
57. Getting Involved How can you “battle” research and practice challenges? Download Proteja and Modificare Contribute new features and bug fixes Perform empirical studies Share indus- try experi- ence reports Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
58. Getting Involved How can you “battle” research and practice challenges? Download Proteja and Modificare Contribute new features and bug fixes Perform empirical studies Share indus- try experi- ence reports How well do these tools work in practice? Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
59. Conclusions and Future Work Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
60. Conclusions and Future Work Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
61. Conclusions and Future Work Test Suite Executor & Coverage Monitor Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
62. Conclusions and Future Work Test Suite Executor & Coverage Monitor http://proteja.googlecode.com Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
63. Conclusions and Future Work Test Suite Executor & Coverage Monitor http://proteja.googlecode.com Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
64. Conclusions and Future Work Test Suite Executor & Coverage Monitor http://proteja.googlecode.com Test Suite Manage- ment & Experimentation Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
65. Conclusions and Future Work Test Suite Executor & Coverage Monitor http://proteja.googlecode.com Test Suite Manage- ment & Experimentation http://modificare.googlecode.com Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
66. Conclusions and Future Work Test Suite Executor & Coverage Monitor http://proteja.googlecode.com Test Suite Manage- ment & Experimentation http://modificare.googlecode.com Choose algorithm based on efficiency instead of effectiveness Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
67. Conclusions and Future Work Test Suite Executor & Coverage Monitor http://proteja.googlecode.com Test Suite Manage- ment & Experimentation http://modificare.googlecode.com Choose algorithm based on efficiency instead of effectiveness Extend and enhance tools and perform additional empirical studies Jonathan Miller Kauffman and Gregory M. Kapfhammer Allegheny College A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques
68. A Framework to Support Research in and Encourage Industrial Adoption of Regression Testing Techniques Jonathan Miller Kauffman and Gregory M. Kapfhammer Thank you for your attention! Questions?