Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Test Adequacy Infrastructure with Database Interaction Awareness

A Test Adequacy Infrastructure with Database Interaction Awareness

Interested in learning more about this topic? Read the overview of my research to learn more: https://www.gregorykapfhammer.com/research/

Gregory Kapfhammer

November 07, 2006
Tweet

More Decks by Gregory Kapfhammer

Other Decks in Science

Transcript

  1. A Test Adequacy Infrastructure with
    Database Interaction Awareness
    Gregory M. Kapfhammer
    Department of Computer Science
    Allegheny College
    (in collaboration with Mary Lou Soffa)
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 1/35

    View Slide

  2. Motivation
    The Risks Digest, Volume 22, Issue 64, 2003
    Jeppesen reports airspace boundary problems
    About 350 airspace boundaries contained in Jeppesen NavData
    are incorrect, the FAA has warned. The error occurred at Jeppe-
    sen after a software upgrade when information was pulled from a
    database containing 20,000 airspace boundaries worldwide for
    the March NavData update, which takes effect March 20.
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 2/35

    View Slide

  3. Looking Ahead
    Test adequacy infrastructure that can find faults and establish
    confidence in the correctness of a database-centric application
    Model of database interaction faults
    Unifed application representation
    Family of test adequacy criteria
    Experiments with real applications that measure the number of
    test requirements and the time and space overheads incurred
    by enumeration
    Foundation for a comprehensive methodology for testing
    database-centric applications
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 3/35

    View Slide

  4. Testing Challenges
    Must consider the environment in which software applications
    actually execute
    Should test a program and its interaction with a database
    Database-centric application’s state space is well-structured,
    but essentially infinite (Chays et al.)
    Need to show program does not violate database integrity,
    where integrity = consistency + validity (Motro)
    Must locate program and database coupling points that vary
    in granularity
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 4/35

    View Slide

  5. Database-Centric Applications
    P
    m i
    m
    j
    Dl
    Dk
    R
    R2
    1
    E F G H
    A B C D
    I
    R3
    J K L
    Program P interacts with two relational
    databases Dk
    and Dl
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 5/35

    View Slide

  6. Database Interactions
    1
    D
    n
    D
    P m
    update
    select insert
    delete
    Program P can view and/or modify the
    state of the database
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 6/35

    View Slide

  7. Database Interaction Faults: (1-v)
    P m
    actual
    before after
    expected
    insert
    update
    P uses update or
    insert to incorrectly
    modify items within
    database
    Commission fault
    that violates
    database validity
    Structural adequacy
    criteria can support
    fault isolation
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 7/35

    View Slide

  8. Database Interaction Faults: (1-c)
    P m
    actual
    before after
    expected
    delete
    P uses delete to
    remove incorrect
    items from database
    Commission fault that
    violates database
    completeness
    Structural adequacy
    criteria can support
    fault isolation
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 8/35

    View Slide

  9. Database Interaction Faults: (2-v)
    P m
    actual
    before after
    expected
    delete
    P does not submit
    delete to remove items
    from database
    Commission or
    omission fault that
    violates database
    validity
    Structural adequacy
    criteria cannot easily
    support omission fault
    isolation
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 9/35

    View Slide

  10. Database Interaction Faults: (2-c)
    P m
    actual
    before after
    expected
    insert
    update
    P does not submit
    update or insert to
    database
    Commission or
    omission fault that
    violates database
    completeness
    Structural adequacy
    criteria cannot easily
    support omission fault
    isolation
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 10/35

    View Slide

  11. Database Interaction Levels
    Database Level
    D1
    P
    Dn
    A program can interact with a
    relational database at different
    levels of granularity
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 11/35

    View Slide

  12. Database Interaction Levels
    UserInfo
    4
    0
    0
    0
    0
    32142
    41601
    45322
    56471
    Relation Level
    P
    D1
    Dn
    Brian Zorman
    Robert Roos
    Marcus Bittman
    Geoffrey Arnold
    3
    card_number pin_number user_name acct_lock
    2
    1
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 11/35

    View Slide

  13. Database Interaction Levels
    UserInfo
    4
    3
    0
    0
    0
    0
    32142
    41601
    45322
    56471
    Record Level
    P
    n
    D1
    D user_name acct_lock
    Brian Zorman
    Robert Roos
    Marcus Bittman
    Geoffrey Arnold
    1
    2
    card_number pin_number
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 11/35

    View Slide

  14. Database Interaction Levels
    UserInfo
    4
    1
    2
    3
    0
    0
    0
    0
    32142
    41601
    45322
    56471
    Attribute Level
    P
    D1
    Dn
    user_name acct_lock
    Brian Zorman
    Robert Roos
    Geoffrey Arnold
    card_number pin_number
    Marcus Bittman
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 11/35

    View Slide

  15. Database Interaction Levels
    UserInfo
    4
    1
    2
    3
    0
    0
    0
    0
    32142
    41601
    45322
    56471
    Attribute Value Level
    P
    D1
    n
    D user_name acct_lock
    Brian Zorman
    Robert Roos
    Geoffrey Arnold
    Marcus Bittman
    card_number pin_number
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 11/35

    View Slide

  16. Database Interaction Points: DML
    select A1
    , A2
    , . . . , Aq
    from r1
    , r2
    , . . . , rm
    where Q
    delete from r
    where Q
    insert into r(A1
    , A2
    , . . . , Aq
    )
    values(v1
    , v2
    , . . . , vq
    )
    update r
    set Al
    = F(Al
    )
    where Q
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 12/35

    View Slide

  17. Analyzing Database Interaction Points
    Database interaction point Ir
    ∈ I corresponds to the
    execution of a SQL DML statement
    Consider the relevant portions of SQL that are parsed
    by HSQLDB (http : //hsqldb.sf.net)
    Interaction points are normally encoded within Java
    programs as dynamically constructed Strings
    select uses Dk
    , delete defines Dk
    , insert defines Dk
    ,
    update defines and/or uses Dk
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 13/35

    View Slide

  18. Refined Database-Centric Application
    P
    m i
    m
    j
    R
    R2
    1
    E F G H
    A B C D
    l
    D
    k
    D
    where
    set J = 500
    update
    L < 1000
    R3
    select 1
    * from R
    R2
    from )
    select
    where A < ( avg(G)
    I
    R3
    J K L
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 14/35

    View Slide

  19. Test Adequacy Concepts
    P violates a database Dk
    ’s validity when it:
    (1-v) inserts entities into Dk
    that do not reflect real
    world
    P violates a database Dk
    ’s completeness when it:
    (1-c) deletes entities from Dk
    that still reflect real
    world
    In order to verify (1-v) and (1-c), T must cause P to
    define and then use entities within D1
    , . . . , Dn
    !
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 15/35

    View Slide

  20. Data Flow Information
    Interaction point:
    “UPDATE UserInfo SET acct_lock = 1 WHERE
    card_number =” + card_number + “;”;
    Database Level: define(BankDB)
    Attribute Level: define(acct_lock) and
    use(card_number)
    Data flow information varies with respect to
    the granularity of the database interaction
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 16/35

    View Slide

  21. Database Entities
    UserInfo
    4
    1 Brian Zorman
    2 Robert Roos
    3 Marcus Bittman
    Geoffrey Arnold
    41601
    45322
    56471
    32142
    0
    0
    0
    0
    user_name acct_lock
    pin_number
    card_number
    v r
    A (I ) = { 32142 }
    1 Geoffrey Arnold 0
    , , . . . , ,
    Enumerate database entities at the attribute value level
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 17/35

    View Slide

  22. Application Types
    Interaction Approach Program Location
    Database−Centric Applications
    Embedded Inside DBMS
    Interface Outside DBMS
    Testing methodology relevant to all types of applications
    Current tool support focuses on Interface-Outside
    applications
    Example: Java application that submits SQL Strings to
    HSQLDB relational database using JDBC drivers
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 18/35

    View Slide

  23. The DICFG: A Unified Representation
    entry lockAccount
    temp1 = parameter0:c_n
    temp2 = LocalDatabaseEntity0:Bank
    temp3 = LocalDatabaseEntity1:acct_lock
    temp4 = LocalDatabaseEntity2:card_number
    “Database-enhanced”
    CFG for lockAccount
    Automatically
    constructed with tool
    support
    Define temporaries to
    represent the
    program’s interaction
    at the levels of
    database and attribute
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 19/35

    View Slide

  24. The DICFG: A Unified Representation
    exit
    G
    G G
    G
    r
    r2
    r 2
    r1
    1
    entry entry
    exit
    lockAccount
    update_lock = m_connect.createStatement()
    if( result_lock == 1)
    completed = true
    exit
    D
    qu_lck = "UPDATE UserInfo ..." + temp1 + ";"
    use(temp4)
    result_lock = update_lock.executeUpdate(qu_lck)
    define(temp2)
    A
    Ir
    define(temp3)
    Database interaction
    graphs (DIGs) are
    placed before interaction
    point Ir
    Multiple DIGs can be
    integrated into a single
    CFG
    String at Ir is
    determined in a
    control-flow sensitive
    fashion using enhanced
    BRICS JSA
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 19/35

    View Slide

  25. DICFG Construction Algorithms
    1
    5
    CreateDIG
    4
    CreateDICFG
    CreateRepresentation
    CreateDIGNode
    GetDatabaseEntities
    2
    3
    Iteratively construct a database aware CFG to support
    data flow analysis and enumerate test requirements
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 20/35

    View Slide

  26. Test Adequacy Criteria
    all−attribute−value−DUs
    all−record−DUs all−attribute−DUs
    all−relation−DUs
    all−database−DUs
    Database interaction
    association (DIA) involves the
    def and use of a database
    entity
    DIAs can be located in the
    DICFG with data flow analysis
    all-database-DUs requires
    tests to exercise all DIAs for all
    of the accessed databases
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 21/35

    View Slide

  27. Generating Test Requirements
    Database Seeder
    Database
    (P) (C)
    Relational Schema
    Requirements
    Test
    System Under Test Test Adequacy Criterion
    Test Case Specification
    Measured time and space overhead
    when computing family of test
    adequacy criteria
    Modified ATM and mp3cd to contain
    appropriate database interaction
    points
    Soot 1.2.5 to calculate
    intraprocedural associations
    GNU/Linux workstation with kernel
    2.4.18-smp and dual 1 GHz Pentium
    III Xeon processors
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 22/35

    View Slide

  28. Experiment Goals and Design
    Reseach Question One: Does the incorporation of
    database interactions yield more test requirements?
    Reseach Question Two: Can test requirement
    enumeration be performed efficiently if database
    interactions are included?
    Experiment Metrics: Number of test requirements (T R),
    time overhead (T ), and space overhead (S)
    Applications: ATM (1732 NCSS and 136 methods) and
    mp3cd (2913 NCSS and 452 methods)
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 23/35

    View Slide

  29. Number of Test Requirements: ATM
    D R Rc A Av P
    Database Granularity
    500
    1000
    1500
    2000
    Number of Associations
    D R Rc A Av P
    39 41 81 107
    203
    1910
    80.7% increase in number of test requirements from D to Av
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 24/35

    View Slide

  30. Number of Test Requirements: mp3cd
    D R Rc A Av P
    Database Granularity
    2000
    4000
    6000
    8000
    10000
    Number of Associations
    D R Rc A Av P
    132 259
    510
    940
    1768
    8718
    92.5% increase in number of test requirements from D to Av
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 25/35

    View Slide

  31. Time Overhead: ATM
    P P D P R P Rc P A P Av
    Database Granularity
    5
    10
    15
    20
    25
    Time sec
    P P D P R P Rc P A P Av
    20.50 20.74 20.77 20.78 20.94 21.06
    2.7% increase in time overhead from P to P + Av
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 26/35

    View Slide

  32. Time Overhead: mp3cd
    P P D P R P Rc P A P Av
    Database Granularity
    10
    20
    30
    40
    Time sec
    P P D P R P Rc P A P Av
    32.72
    34.66 34.82 35.13
    36.37
    38.23
    14.4% increase in time overhead from P to P + Av
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 27/35

    View Slide

  33. Space Overhead: ATM
    5 10 15 20
    Node and Edge Count
    P Av
    P A
    P Rc
    P R
    P D
    P
    Database Granularity
    P Av
    P A
    P Rc
    P R
    P D
    P
    17.5
    16.5
    16.2
    15.8
    15.8
    15.5
    17.2
    16.2
    15.9
    15.5
    15.4
    15.1
    Edges
    Nodes
    Average number of {DI}CFG nodes and edges is stable
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 28/35

    View Slide

  34. Space Overhead: mp3cd
    5 10 15 20 25 30 35
    Node and Edge Count
    P Av
    P A
    P Rc
    P R
    P D
    P
    Database Granularity
    P Av
    P A
    P Rc
    P R
    P D
    P
    29.8
    26.5
    24.8
    23.7
    23.2
    22.7
    24.5
    22.0
    20.7
    19.9
    19.5
    19.2
    Edges
    Nodes
    mp3cd has more database interactions and larger database
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 29/35

    View Slide

  35. Average Increase in CFG Nodes
    SN%
    I
    (R, D) SN %
    I
    (Rc, R) SN %
    I
    (A, R)
    ATM .6 2.5 4.3
    mp3cd 2.0 3.8 9.5
    SN%
    I
    (Av, Rc
    ) SN%
    I
    (Av, A) SN %
    I
    (Av, D) SN %
    I
    (Av, P)
    ATM 7.5 5.8 10.4 12.2
    mp3cd 15.5 10.2 20.4 21.6
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 30/35

    View Slide

  36. Average Increase in CFG Edges
    SE%
    I
    (R, D) SE%
    I
    (Rc, R) SE%
    I
    (A, R)
    ATM 0.0 2.4 4.2
    mp3cd 2.1 4.4 10.5
    SE%
    I
    (Av, Rc
    ) SE%
    I
    (Av, A) SE%
    I
    (Av, D) SE%
    I
    (Av, P)
    ATM 7.4 5.7 9.7 11.4
    mp3cd 16.7 11.0 22.1 23.8
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 31/35

    View Slide

  37. Related Work
    Jin and Offutt and Whittaker and Voas have suggested that
    the environment of a software system is important
    Chan and Cheung transform SQL statements into C code
    segments
    Chays et al. and Chays and Deng have created the
    category-partition inspired AGENDA tool suite
    Neufeld et al. and Zhang et al. have proposed techniques
    for database state generation
    Dauo et al. focused on the regression testing of
    database-driven applications
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 32/35

    View Slide

  38. Ongoing Research
    Test suite execution that minimizes number of costly
    database restarts and initializations
    Test coverage monitoring through a database interaction
    calling context tree (DICCT)
    Regression test suite reduction and prioritization that
    incorporates database aware adequacy and test case cost
    Detailed empirical studies with ten case study applications
    of varying code and database size
    Comprehensive tool support to assist the testing of
    database-centric applications
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 33/35

    View Slide

  39. Conclusions
    Must test the program’s interaction with the database
    Test adequacy infrastructure provides : (i) database
    interaction fault model, (ii) unified application representation,
    (iii) family of test adequacy criteria
    Unique family of test adequacy criteria to detect all type (1)
    and some type (2) violations of database validity and
    completeness
    Intraprocedural database interactions can be computed from
    a DICFG with minimal time and space overhead
    Foundation for a complete testing methodology
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 34/35

    View Slide

  40. Further Resources
    Gregory M. Kapfhammer and Mary Lou Soffa. A Family of
    Test Adequacy Criteria for Database-Driven Applications.
    In ESEC/FSE 2003.
    Gregory M. Kapfhammer. Software Testing. CRC Press
    Computer Science Handbook. June, 2004.
    http : //cs.allegheny.edu/˜gkapfham/research/diatoms/
    A Test Adequacy Infrastrcture with Database Interaction Awareness, UCSB, November 7, 2005 – p. 35/35

    View Slide