Towards regression testing for database applications

Interested in learning more about this topic? Visit this web site to see related presentations: https://www.gregorykapfhammer.com/research/presentations/

Transcript

1. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Towards Regression Testing for Database Applications Gregory M. Kapfhammer† Department of Computer Science Allegheny College, Pennsylvania, USA http://cs.allegheny.edu/~gkapfham/ ASTReNet and SOSoRNet, King’s College London, 2007 †In Conjunction with Mary Lou Soffa (UVa/CS), Panos Chrysanthis (Pitt/CS), Bruce Childers (Pitt/CS) Gregory M. Kapfhammer Towards Regression Testing for Database Applications
2. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Testing Database Applications Research Contribution A regression testing framework for traditional database applications. Future research includes service-oriented applications that use Grid-enabled databases. Gregory M. Kapfhammer Towards Regression Testing for Database Applications
3. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions An Interesting Defect Report Database Server Crashes When you run a complex query against Microsoft SQL Server 2000, the SQL Server scheduler may stop responding. Additionally, you receive an error message that resembles the following: Date Time server Error: 17883 Severity: 1, State: 0 Date Time server Process 52:0 (94c) ... An Input-Dependent Defect This problem occurs when one or more of the following conditions are true: The query contains a U N I O N clause or a U N I O N A L L clause that affects many columns. The query contains several J O I N statements. The query has a large estimated cost. BUG 473858 (SQL Server 8.0) Gregory M. Kapfhammer Towards Regression Testing for Database Applications
4. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions An Interesting Defect Report Database Server Crashes When you run a complex query against Microsoft SQL Server 2000, the SQL Server scheduler may stop responding. Additionally, you receive an error message that resembles the following: Date Time server Error: 17883 Severity: 1, State: 0 Date Time server Process 52:0 (94c) ... An Input-Dependent Defect This problem occurs when one or more of the following conditions are true: The query contains a U N I O N clause or a U N I O N A L L clause that affects many columns. The query contains several J O I N statements. The query has a large estimated cost. BUG 473858 (SQL Server 8.0) Gregory M. Kapfhammer Towards Regression Testing for Database Applications
5. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Real World Example A Severe Defect The Risks Digest, Volume 22, Issue 64, 2003 Jeppesen reports airspace boundary problems About 350 airspace boundaries contained in Jeppesen NavData are incorrect, the FAA has warned. The error occurred at Jeppesen after a software upgrade when information was pulled from a database containing 20,000 airspace boundaries worldwide for the March NavData update, which takes effect March 20. An Important Point Practically all use of databases occurs from within application programs [Silberschatz et al., 2006, pg. 311] Gregory M. Kapfhammer Towards Regression Testing for Database Applications
6. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Real World Example A Severe Defect The Risks Digest, Volume 22, Issue 64, 2003 Jeppesen reports airspace boundary problems About 350 airspace boundaries contained in Jeppesen NavData are incorrect, the FAA has warned. The error occurred at Jeppesen after a software upgrade when information was pulled from a database containing 20,000 airspace boundaries worldwide for the March NavData update, which takes effect March 20. An Important Point Practically all use of databases occurs from within application programs [Silberschatz et al., 2006, pg. 311] Gregory M. Kapfhammer Towards Regression Testing for Database Applications
7. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Program and Database Interactions 1 D e D P m update select insert delete Basic Operation Program P creates SQL statements in order to view and/or modify the state of the relational database SQL Construction Static analysis does not reveal the exact SQL command since the program constructs the full SQL statement at run-time Gregory M. Kapfhammer Towards Regression Testing for Database Applications
8. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Database Interaction Granularity P m i m j Dl Dk R R2 1 E F G H A B C D I R3 J K L Database Interactions Program P interacts with two relational databases Dk and Dl at different levels of granularity (relation, record, attribute, ...) Gregory M. Kapfhammer Towards Regression Testing for Database Applications
9. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Overview of the Coverage Monitoring Process Program Instrumentation Test Suite Adequacy Criterion Instrumented Program Test Coverage Monitoring Instrumented Test Suite Coverage Results Adequacy Calculation Test Requirements Adequacy Measurements Database Calculating Coverage Use instrumentation probes to capture and analyze a program’s interaction with the databases Regression Testing The adequacy measurements can be used to support both test suite reduction and prioritization Gregory M. Kapfhammer Towards Regression Testing for Database Applications
10. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Database-Aware Coverage Trees Test Case Method Invocation Database Interaction Point Database Relation Attribute Record Attribute Value Instrumentation Probes Use static and dynamic (load-time) instrumentation techniques to insert coverage monitoring probes Coverage Trees Store the coverage results in a tree in order to support the calculation of many types of coverage (e.g., data flow or call tree) Gregory M. Kapfhammer Towards Regression Testing for Database Applications
11. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Comparing the Coverage Trees Tree Characteristics Tree DB? Context Probe Time Tree Space CCT × Partial Low - Moderate Low DCT × Full Low Moderate - High DI-CCT Partial Moderate Moderate DI-DCT Full Moderate High Table Legend Database? ∈ {×, } Context ∈ {Partial, Full} Probe Time Overhead ∈ {Low, Moderate, High} Tree Space Overhead ∈ {Low, Moderate, High} Gregory M. Kapfhammer Towards Regression Testing for Database Applications
12. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Database-Aware Regression Testing Begin Coverage Report End VSRT Repeat Program and Database Reduction or Prioritization Original Test Suite Modified Test Suite Test Suite Execution Testing Results GRT Repeat Regression Testing Overview Reduction aims to find a smaller test suite that covers the same requirements as the original suite. Prioritization re-orders the tests so that they cover the requirements more effectively. Gregory M. Kapfhammer Towards Regression Testing for Database Applications
13. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Finding the Overlap in Coverage T2 T3 T6 T9 R R1 R2 T4 R3 T8 R3 T11 R3 T12 R4 T1 R4 R4 R4 T5 R4 R4 T10 R4 T11 R5 T5 R5 R6 R6 T10 R7 R7 T7 R7 T10 Test Suite Reduction Rj → Ti means that requirement Rj is covered by test Ti T = T2 , T3 , T6 , T9 cover all of the test requirements Gregory M. Kapfhammer Towards Regression Testing for Database Applications
14. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Measuring Coverage Effectiveness Testing Time . . . Covered Test Reqs acements T1 Done Tn−1 Done Tn Done Cover R(T1 ) Cover n−1 i=1 R(Ti ) Cover R(T) Area t(n) 0 C(T, t) C(T , t) (t) Prioritize to increase the CE of a test suite CE = Actual Ideal Gregory M. Kapfhammer Towards Regression Testing for Database Applications
15. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Configuring the Regression Testing Framework Type Data Flow Coverage Tree Path Configuration of the Regression Tester Technique Test Cost Requirements Reduction Prioritization Type Greedy HGS Delayed Greedy Reverse Random Overlap-Aware Not Overlap-Aware Cost Coverage Ratio Unit Actual Traditional Database-Aware Unique Dominant Type of Tree Super Path Containing Path DCT CCT Regression tester uses several algorithms and test requirements Gregory M. Kapfhammer Towards Regression Testing for Database Applications
16. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Characterizing the Case Study Applications Test Suites Application # Tests Test NCSS / Total NCSS R M 13 227/548 = 50.5% F F 16 330/558 = 59.1% P I 15 203/579 = 35.1% S T 25 365/620 = 58.9% T M 27 355/748 = 47.5% G B 51 769/1455 = 52.8% Gregory M. Kapfhammer Towards Regression Testing for Database Applications
17. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Details About the Database Interactions Static Interaction Counts Application executeUpdate executeQuery Total R M 3 4 7 F F 3 4 7 P I 3 2 5 S T 4 3 7 T M 36 9 45 G B 11 23 34 Dynamic Interaction Counts Database interactions that occur in iterative or recursive computations are executed more frequently Gregory M. Kapfhammer Towards Regression Testing for Database Applications
18. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Reducing the Size of the Test Suite (Size of Reduced Test Suite, Reduction Factor) App Rel Attr Rec Attr Value RM (13) (7, .462) (7, .462) (10, .300) (9, .308) FF (16) (7, .563) (7, .563) (11, .313) (11, .313) PI (15) (6, .600) (6, .600) (8, .700) (7, .533) ST (25) (5, .800) (5, .760) (11, .560) (10, .600) TM (27) (14, .481) (14, .481) (15, .449) (14, .481) GB (51) (33, .352) (33, .352) (33, .352) (32, .373) All (24.5) (12, .510) (12.17, .503) (14.667, .401) (13.83, .435) Reduction factor for test suite size varies from .352 to .8 Gregory M. Kapfhammer Towards Regression Testing for Database Applications
19. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Reducing the Testing Time Rc Av Database Interaction GB 0.2 0.4 0.6 0.8 1 Reduction Factor Time Rc Av 0.78 0.78 0.94 0.94 0.06 0.06 0.81 0.81 0.79 0.79 0.36 0.39 GRO GRC GRV GRR RVR RAR GRO reduces test execution time even though it removes few tests Gregory M. Kapfhammer Towards Regression Testing for Database Applications
20. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Preserving Requirement Coverage Rc Av Database Interaction GB 0.2 0.4 0.6 0.8 1 Preservation Factor Coverage Rc Av 1. 1. 0.3 0.09 0.98 0.99 0.96 0.98 0.91 0.94 0.71 0.72 GRO GRC GRV GRR RVR RAR GRO guarantees coverage preservation while the others do not Gregory M. Kapfhammer Towards Regression Testing for Database Applications
21. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Improving Coverage Effectiveness Rc Av Database Interaction GB 0.2 0.4 0.6 0.8 1 Coverage Effectiveness Rc Av 0.94 0.94 0.88 0.87 0.87 0.9 0.93 0.93 0.84 0.86 0.22 0.22 0.55 0.56 GPO GPC GPV GPR RVP ORP RAP GRO is the best choice and the original ordering is poor Gregory M. Kapfhammer Towards Regression Testing for Database Applications
22. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Future Work: Avoid Database Restarts T0 T1 T2 T9 T13 T15 T5 T8 T12 T3 T7 T11 T14 T16 T10 T4 T6 Use prioritization to reduce testing time by avoiding database restarts Gregory M. Kapfhammer Towards Regression Testing for Database Applications
23. Introduction to Database Applications Database-Aware Test Coverage Monitoring Regression Testing Experimental Study Future Work and Conclusions Conclusions and Future Work Concluding Remarks A new perspective on software testing and an efficient and effective method for database-aware regression testing Future Work Challenges associated with grid-enabled databases Conduct experiments with larger database applications Resources http://cs.allegheny.edu/~gkapfham/research/diatoms/ Gregory M. Kapfhammer Towards Regression Testing for Database Applications