Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Towards regression testing for database applications

Towards regression testing for database applications

Interested in learning more about this topic? Visit this web site to see related presentations: https://www.gregorykapfhammer.com/research/presentations/

Gregory Kapfhammer

October 12, 2007
Tweet

More Decks by Gregory Kapfhammer

Other Decks in Technology

Transcript

  1. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Towards Regression Testing for
    Database Applications
    Gregory M. Kapfhammer†
    Department of Computer Science
    Allegheny College, Pennsylvania, USA
    http://cs.allegheny.edu/~gkapfham/
    ASTReNet and SOSoRNet, King’s College London, 2007
    †In Conjunction with Mary Lou Soffa (UVa/CS), Panos Chrysanthis (Pitt/CS), Bruce Childers (Pitt/CS)
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  2. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Testing Database Applications
    Research Contribution
    A regression testing framework for traditional database
    applications. Future research includes service-oriented
    applications that use Grid-enabled databases.
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  3. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    An Interesting Defect Report
    Database Server Crashes
    When you run a complex query against Microsoft SQL Server
    2000, the SQL Server scheduler may stop responding.
    Additionally, you receive an error message that resembles the
    following: Date Time server Error: 17883 Severity: 1, State:
    0 Date Time server Process 52:0 (94c) ...
    An Input-Dependent Defect
    This problem occurs when one or more of the following
    conditions are true: The query contains a U N I O N clause or a
    U N I O N A L L clause that affects many columns. The query
    contains several J O I N statements. The query has a large
    estimated cost. BUG 473858 (SQL Server 8.0)
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  4. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    An Interesting Defect Report
    Database Server Crashes
    When you run a complex query against Microsoft SQL Server
    2000, the SQL Server scheduler may stop responding.
    Additionally, you receive an error message that resembles the
    following: Date Time server Error: 17883 Severity: 1, State:
    0 Date Time server Process 52:0 (94c) ...
    An Input-Dependent Defect
    This problem occurs when one or more of the following
    conditions are true: The query contains a U N I O N clause or a
    U N I O N A L L clause that affects many columns. The query
    contains several J O I N statements. The query has a large
    estimated cost. BUG 473858 (SQL Server 8.0)
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  5. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Real World Example
    A Severe Defect
    The Risks Digest, Volume 22, Issue 64, 2003
    Jeppesen reports airspace boundary problems
    About 350 airspace boundaries contained in Jeppesen NavData
    are incorrect, the FAA has warned. The error occurred at
    Jeppesen after a software upgrade when information was pulled
    from a database containing 20,000 airspace boundaries worldwide
    for the March NavData update, which takes effect March 20.
    An Important Point
    Practically all use of databases occurs from within application
    programs [Silberschatz et al., 2006, pg. 311]
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  6. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Real World Example
    A Severe Defect
    The Risks Digest, Volume 22, Issue 64, 2003
    Jeppesen reports airspace boundary problems
    About 350 airspace boundaries contained in Jeppesen NavData
    are incorrect, the FAA has warned. The error occurred at
    Jeppesen after a software upgrade when information was pulled
    from a database containing 20,000 airspace boundaries worldwide
    for the March NavData update, which takes effect March 20.
    An Important Point
    Practically all use of databases occurs from within application
    programs [Silberschatz et al., 2006, pg. 311]
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  7. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Program and Database Interactions
    1
    D
    e
    D
    P m
    update
    select insert
    delete
    Basic Operation
    Program P creates SQL
    statements in order to view
    and/or modify the state of the
    relational database
    SQL Construction
    Static analysis does not reveal
    the exact SQL command since
    the program constructs the full
    SQL statement at run-time
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  8. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Database Interaction Granularity
    P
    m i
    m
    j
    Dl
    Dk
    R
    R2
    1
    E F G H
    A B C D
    I
    R3
    J K L
    Database Interactions
    Program P interacts with two relational databases Dk
    and Dl
    at
    different levels of granularity (relation, record, attribute, ...)
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  9. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Overview of the Coverage Monitoring Process
    Program
    Instrumentation
    Test Suite
    Adequacy
    Criterion
    Instrumented
    Program
    Test Coverage
    Monitoring
    Instrumented
    Test Suite
    Coverage
    Results
    Adequacy
    Calculation
    Test
    Requirements
    Adequacy
    Measurements
    Database
    Calculating Coverage
    Use instrumentation probes
    to capture and analyze a
    program’s interaction with the
    databases
    Regression Testing
    The adequacy
    measurements can be used
    to support both test suite
    reduction and prioritization
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  10. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Database-Aware Coverage Trees
    Test Case
    Method Invocation
    Database Interaction Point
    Database
    Relation
    Attribute Record
    Attribute Value
    Instrumentation Probes
    Use static and dynamic
    (load-time) instrumentation
    techniques to insert coverage
    monitoring probes
    Coverage Trees
    Store the coverage results in
    a tree in order to support the
    calculation of many types of
    coverage (e.g., data flow or
    call tree)
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  11. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Comparing the Coverage Trees
    Tree Characteristics
    Tree DB? Context Probe Time Tree Space
    CCT × Partial Low - Moderate Low
    DCT × Full Low Moderate - High
    DI-CCT Partial Moderate Moderate
    DI-DCT Full Moderate High
    Table Legend
    Database? ∈ {×, }
    Context ∈ {Partial, Full}
    Probe Time Overhead ∈ {Low, Moderate, High}
    Tree Space Overhead ∈ {Low, Moderate, High}
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  12. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Database-Aware Regression Testing
    Begin Coverage Report End
    VSRT Repeat
    Program and
    Database
    Reduction
    or Prioritization
    Original
    Test Suite
    Modified
    Test Suite
    Test Suite
    Execution
    Testing Results
    GRT Repeat
    Regression Testing Overview
    Reduction aims to find a smaller test suite that covers the same
    requirements as the original suite. Prioritization re-orders the
    tests so that they cover the requirements more effectively.
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  13. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Finding the Overlap in Coverage
    T2 T3
    T6
    T9
    R
    R1
    R2
    T4
    R3
    T8
    R3
    T11
    R3
    T12
    R4
    T1
    R4
    R4
    R4
    T5
    R4
    R4
    T10
    R4
    T11
    R5
    T5
    R5
    R6
    R6
    T10
    R7
    R7
    T7
    R7
    T10
    Test Suite Reduction
    Rj
    → Ti
    means that requirement Rj
    is covered by test Ti
    T = T2
    , T3
    , T6
    , T9
    cover all of the test requirements
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  14. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Measuring Coverage Effectiveness
    Testing Time
    . . .
    Covered Test Reqs
    acements
    T1
    Done Tn−1
    Done
    Tn Done
    Cover R(T1
    ) Cover n−1
    i=1
    R(Ti
    )
    Cover R(T)
    Area t(n)
    0
    C(T, t)
    C(T
    ,
    t)
    (t)
    Prioritize to increase the CE of a test suite CE = Actual
    Ideal
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  15. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Configuring the Regression Testing Framework
    Type
    Data Flow Coverage Tree Path
    Configuration of the Regression Tester
    Technique Test Cost Requirements
    Reduction Prioritization Type
    Greedy HGS Delayed Greedy Reverse Random
    Overlap-Aware Not Overlap-Aware
    Cost Coverage Ratio
    Unit Actual Traditional Database-Aware
    Unique Dominant Type of Tree
    Super Path Containing Path DCT CCT
    Regression tester uses several algorithms and test requirements
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  16. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Characterizing the Case Study Applications
    Test Suites
    Application # Tests Test NCSS / Total NCSS
    R M 13 227/548 = 50.5%
    F F 16 330/558 = 59.1%
    P I 15 203/579 = 35.1%
    S T 25 365/620 = 58.9%
    T M 27 355/748 = 47.5%
    G B 51 769/1455 = 52.8%
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  17. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Details About the Database Interactions
    Static Interaction Counts
    Application executeUpdate executeQuery Total
    R M 3 4 7
    F F 3 4 7
    P I 3 2 5
    S T 4 3 7
    T M 36 9 45
    G B 11 23 34
    Dynamic Interaction Counts
    Database interactions that occur in iterative or recursive
    computations are executed more frequently
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  18. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Reducing the Size of the Test Suite
    (Size of Reduced Test Suite, Reduction Factor)
    App Rel Attr Rec Attr Value
    RM (13) (7, .462) (7, .462) (10, .300) (9, .308)
    FF (16) (7, .563) (7, .563) (11, .313) (11, .313)
    PI (15) (6, .600) (6, .600) (8, .700) (7, .533)
    ST (25) (5, .800) (5, .760) (11, .560) (10, .600)
    TM (27) (14, .481) (14, .481) (15, .449) (14, .481)
    GB (51) (33, .352) (33, .352) (33, .352) (32, .373)
    All (24.5) (12, .510) (12.17, .503) (14.667, .401) (13.83, .435)
    Reduction factor for test suite size varies from .352 to .8
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  19. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Reducing the Testing Time
    Rc Av
    Database Interaction GB
    0.2
    0.4
    0.6
    0.8
    1
    Reduction Factor Time
    Rc Av
    0.78 0.78
    0.94 0.94
    0.06 0.06
    0.81 0.81
    0.79 0.79
    0.36
    0.39
    GRO GRC GRV GRR RVR RAR
    GRO reduces test execution time even though it removes few tests
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  20. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Preserving Requirement Coverage
    Rc Av
    Database Interaction GB
    0.2
    0.4
    0.6
    0.8
    1
    Preservation Factor Coverage
    Rc Av
    1. 1.
    0.3
    0.09
    0.98 0.99
    0.96
    0.98
    0.91
    0.94
    0.71 0.72
    GRO GRC GRV GRR RVR RAR
    GRO guarantees coverage preservation while the others do not
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  21. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Improving Coverage Effectiveness
    Rc Av
    Database Interaction GB
    0.2
    0.4
    0.6
    0.8
    1
    Coverage Effectiveness
    Rc Av
    0.94 0.94
    0.88 0.87
    0.87
    0.9
    0.93 0.93
    0.84
    0.86
    0.22 0.22
    0.55 0.56
    GPO GPC GPV GPR RVP ORP RAP
    GRO is the best choice and the original ordering is poor
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  22. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Future Work: Avoid Database Restarts
    T0
    T1
    T2
    T9
    T13
    T15
    T5
    T8
    T12
    T3
    T7
    T11
    T14
    T16
    T10
    T4
    T6
    Use prioritization to reduce testing time by avoiding database restarts
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide

  23. Introduction to Database Applications
    Database-Aware Test Coverage Monitoring
    Regression Testing
    Experimental Study
    Future Work and Conclusions
    Conclusions and Future Work
    Concluding Remarks
    A new perspective on software testing and an efficient and
    effective method for database-aware regression testing
    Future Work
    Challenges associated with grid-enabled databases
    Conduct experiments with larger database applications
    Resources
    http://cs.allegheny.edu/~gkapfham/research/diatoms/
    Gregory M. Kapfhammer Towards Regression Testing for Database Applications

    View Slide