A Test Adequacy Infrastructure with Database Interaction Awareness Gregory M. Kapfhammer Department of Computer Science Allegheny College (in collaboration with Mary Lou Soffa) A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 1/35
Motivation The Risks Digest, Volume 22, Issue 64, 2003 Jeppesen reports airspace boundary problems About 350 airspace boundaries contained in Jeppesen NavData are incorrect, the FAA has warned. The error occurred at Jeppe- sen after a software upgrade when information was pulled from a database containing 20,000 airspace boundaries worldwide for the March NavData update, which takes effect March 20. A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 2/35
Looking Ahead Test adequacy infrastructure that can find faults and establish confidence in the correctness of a database-centric application Model of database interaction faults Unifed application representation Family of test adequacy criteria Experiments with real applications that measure the number of test requirements and the time and space overheads incurred by enumeration Foundation for a comprehensive methodology for testing database-centric applications A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 3/35
Testing Challenges Must consider the environment in which software applications actually execute Should test a program and its interaction with a database Database-centric application’s state space is well-structured, but essentially infinite (Chays et al.) Need to show program does not violate database integrity, where integrity = consistency + validity (Motro) Must locate program and database coupling points that vary in granularity A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 4/35
Database-Centric Applications P m i m j Dl Dk R R2 1 E F G H A B C D I R3 J K L Program P interacts with two relational databases Dk and Dl A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 5/35
Database Interactions 1 D n D P m update select insert delete Program P can view and/or modify the state of the database A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 6/35
Database Interaction Faults: (1-v) P m actual before after expected insert update P uses update or insert to incorrectly modify items within database Commission fault that violates database validity Structural adequacy criteria can support fault isolation A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 7/35
Database Interaction Faults: (1-c) P m actual before after expected delete P uses delete to remove incorrect items from database Commission fault that violates database completeness Structural adequacy criteria can support fault isolation A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 8/35
Database Interaction Faults: (2-v) P m actual before after expected delete P does not submit delete to remove items from database Commission or omission fault that violates database validity Structural adequacy criteria cannot easily support omission fault isolation A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 9/35
Database Interaction Faults: (2-c) P m actual before after expected insert update P does not submit update or insert to database Commission or omission fault that violates database completeness Structural adequacy criteria cannot easily support omission fault isolation A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 10/35
Database Interaction Levels Database Level D1 P Dn A program can interact with a relational database at different levels of granularity A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 11/35
Database Interaction Levels UserInfo 4 0 0 0 0 32142 41601 45322 56471 Relation Level P D1 Dn Brian Zorman Robert Roos Marcus Bittman Geoffrey Arnold 3 card_number pin_number user_name acct_lock 2 1 A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 11/35
Database Interaction Levels UserInfo 4 3 0 0 0 0 32142 41601 45322 56471 Record Level P n D1 D user_name acct_lock Brian Zorman Robert Roos Marcus Bittman Geoffrey Arnold 1 2 card_number pin_number A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 11/35
Database Interaction Levels UserInfo 4 1 2 3 0 0 0 0 32142 41601 45322 56471 Attribute Level P D1 Dn user_name acct_lock Brian Zorman Robert Roos Geoffrey Arnold card_number pin_number Marcus Bittman A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 11/35
Database Interaction Levels UserInfo 4 1 2 3 0 0 0 0 32142 41601 45322 56471 Attribute Value Level P D1 n D user_name acct_lock Brian Zorman Robert Roos Geoffrey Arnold Marcus Bittman card_number pin_number A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 11/35
Database Interaction Points: DML select A1 , A2 , . . . , Aq from r1 , r2 , . . . , rm where Q delete from r where Q insert into r(A1 , A2 , . . . , Aq ) values(v1 , v2 , . . . , vq ) update r set Al = F(Al ) where Q A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 12/35
Analyzing Database Interaction Points Database interaction point Ir ∈ I corresponds to the execution of a SQL DML statement Consider the relevant portions of SQL that are parsed by HSQLDB (http : //hsqldb.sf.net) Interaction points are normally encoded within Java programs as dynamically constructed Strings select uses Dk , delete defines Dk , insert defines Dk , update defines and/or uses Dk A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 13/35
Refined Database-Centric Application P m i m j R R2 1 E F G H A B C D l D k D where set J = 500 update L < 1000 R3 select 1 * from R R2 from ) select where A < ( avg(G) I R3 J K L A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 14/35
Test Adequacy Concepts P violates a database Dk ’s validity when it: (1-v) inserts entities into Dk that do not reflect real world P violates a database Dk ’s completeness when it: (1-c) deletes entities from Dk that still reflect real world In order to verify (1-v) and (1-c), T must cause P to define and then use entities within D1 , . . . , Dn ! A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 15/35
Data Flow Information Interaction point: “UPDATE UserInfo SET acct_lock = 1 WHERE card_number =” + card_number + “;”; Database Level: define(BankDB) Attribute Level: define(acct_lock) and use(card_number) Data flow information varies with respect to the granularity of the database interaction A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 16/35
Database Entities UserInfo 4 1 Brian Zorman 2 Robert Roos 3 Marcus Bittman Geoffrey Arnold 41601 45322 56471 32142 0 0 0 0 user_name acct_lock pin_number card_number v r A (I ) = { 32142 } 1 Geoffrey Arnold 0 , , . . . , , Enumerate database entities at the attribute value level A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 17/35
Application Types Interaction Approach Program Location Database−Centric Applications Embedded Inside DBMS Interface Outside DBMS Testing methodology relevant to all types of applications Current tool support focuses on Interface-Outside applications Example: Java application that submits SQL Strings to HSQLDB relational database using JDBC drivers A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 18/35
The DICFG: A Unified Representation entry lockAccount temp1 = parameter0:c_n temp2 = LocalDatabaseEntity0:Bank temp3 = LocalDatabaseEntity1:acct_lock temp4 = LocalDatabaseEntity2:card_number “Database-enhanced” CFG for lockAccount Automatically constructed with tool support Define temporaries to represent the program’s interaction at the levels of database and attribute A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 19/35
The DICFG: A Unified Representation exit G G G G r r2 r 2 r1 1 entry entry exit lockAccount update_lock = m_connect.createStatement() if( result_lock == 1) completed = true exit D qu_lck = "UPDATE UserInfo ..." + temp1 + ";" use(temp4) result_lock = update_lock.executeUpdate(qu_lck) define(temp2) A Ir define(temp3) Database interaction graphs (DIGs) are placed before interaction point Ir Multiple DIGs can be integrated into a single CFG String at Ir is determined in a control-flow sensitive fashion using enhanced BRICS JSA A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 19/35
DICFG Construction Algorithms 1 5 CreateDIG 4 CreateDICFG CreateRepresentation CreateDIGNode GetDatabaseEntities 2 3 Iteratively construct a database aware CFG to support data flow analysis and enumerate test requirements A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 20/35
Test Adequacy Criteria all−attribute−value−DUs all−record−DUs all−attribute−DUs all−relation−DUs all−database−DUs Database interaction association (DIA) involves the def and use of a database entity DIAs can be located in the DICFG with data flow analysis all-database-DUs requires tests to exercise all DIAs for all of the accessed databases A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 21/35
Generating Test Requirements Database Seeder Database (P) (C) Relational Schema Requirements Test System Under Test Test Adequacy Criterion Test Case Specification Measured time and space overhead when computing family of test adequacy criteria Modified ATM and mp3cd to contain appropriate database interaction points Soot 1.2.5 to calculate intraprocedural associations GNU/Linux workstation with kernel 2.4.18-smp and dual 1 GHz Pentium III Xeon processors A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 22/35
Experiment Goals and Design Reseach Question One: Does the incorporation of database interactions yield more test requirements? Reseach Question Two: Can test requirement enumeration be performed efficiently if database interactions are included? Experiment Metrics: Number of test requirements (T R), time overhead (T ), and space overhead (S) Applications: ATM (1732 NCSS and 136 methods) and mp3cd (2913 NCSS and 452 methods) A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 23/35
Number of Test Requirements: ATM D R Rc A Av P Database Granularity 500 1000 1500 2000 Number of Associations D R Rc A Av P 39 41 81 107 203 1910 80.7% increase in number of test requirements from D to Av A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 24/35
Number of Test Requirements: mp3cd D R Rc A Av P Database Granularity 2000 4000 6000 8000 10000 Number of Associations D R Rc A Av P 132 259 510 940 1768 8718 92.5% increase in number of test requirements from D to Av A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 25/35
Time Overhead: ATM P P D P R P Rc P A P Av Database Granularity 5 10 15 20 25 Time sec P P D P R P Rc P A P Av 20.50 20.74 20.77 20.78 20.94 21.06 2.7% increase in time overhead from P to P + Av A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 26/35
Time Overhead: mp3cd P P D P R P Rc P A P Av Database Granularity 10 20 30 40 Time sec P P D P R P Rc P A P Av 32.72 34.66 34.82 35.13 36.37 38.23 14.4% increase in time overhead from P to P + Av A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 27/35
Space Overhead: ATM 5 10 15 20 Node and Edge Count P Av P A P Rc P R P D P Database Granularity P Av P A P Rc P R P D P 17.5 16.5 16.2 15.8 15.8 15.5 17.2 16.2 15.9 15.5 15.4 15.1 Edges Nodes Average number of {DI}CFG nodes and edges is stable A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 28/35
Space Overhead: mp3cd 5 10 15 20 25 30 35 Node and Edge Count P Av P A P Rc P R P D P Database Granularity P Av P A P Rc P R P D P 29.8 26.5 24.8 23.7 23.2 22.7 24.5 22.0 20.7 19.9 19.5 19.2 Edges Nodes mp3cd has more database interactions and larger database A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 29/35
Average Increase in CFG Nodes SN% I (R, D) SN % I (Rc, R) SN % I (A, R) ATM .6 2.5 4.3 mp3cd 2.0 3.8 9.5 SN% I (Av, Rc ) SN% I (Av, A) SN % I (Av, D) SN % I (Av, P) ATM 7.5 5.8 10.4 12.2 mp3cd 15.5 10.2 20.4 21.6 A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 30/35
Average Increase in CFG Edges SE% I (R, D) SE% I (Rc, R) SE% I (A, R) ATM 0.0 2.4 4.2 mp3cd 2.1 4.4 10.5 SE% I (Av, Rc ) SE% I (Av, A) SE% I (Av, D) SE% I (Av, P) ATM 7.4 5.7 9.7 11.4 mp3cd 16.7 11.0 22.1 23.8 A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 31/35
Related Work Jin and Offutt and Whittaker and Voas have suggested that the environment of a software system is important Chan and Cheung transform SQL statements into C code segments Chays et al. and Chays and Deng have created the category-partition inspired AGENDA tool suite Neufeld et al. and Zhang et al. have proposed techniques for database state generation Dauo et al. focused on the regression testing of database-driven applications A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 32/35
Ongoing Research Test suite execution that minimizes number of costly database restarts and initializations Test coverage monitoring through a database interaction calling context tree (DICCT) Regression test suite reduction and prioritization that incorporates database aware adequacy and test case cost Detailed empirical studies with ten case study applications of varying code and database size Comprehensive tool support to assist the testing of database-centric applications A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 33/35
Conclusions Must test the program’s interaction with the database Test adequacy infrastructure provides : (i) database interaction fault model, (ii) unified application representation, (iii) family of test adequacy criteria Unique family of test adequacy criteria to detect all type (1) and some type (2) violations of database validity and completeness Intraprocedural database interactions can be computed from a DICFG with minimal time and space overhead Foundation for a complete testing methodology A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 34/35
Further Resources Gregory M. Kapfhammer and Mary Lou Soffa. A Family of Test Adequacy Criteria for Database-Driven Applications. In ESEC/FSE 2003. Gregory M. Kapfhammer. Software Testing. CRC Press Computer Science Handbook. June, 2004. http : //cs.allegheny.edu/˜gkapfham/research/diatoms/ A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 35/35
gkapfham
yukizenimoto
naokiuc
akiratosei
yellowshippo
hidenorimatsubayashi
peterdesmet
konakalab
s1ok69oo
mprove
dasaptaerwin
jcasabona
chriscoyier
ufuk
jonyablonski
michaelherold
jonrohan
jakevdp
matthewcrist
holman
paulrobertlloyd
lara