Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Set covers, knapsacks, and regression testing techniques

Set covers, knapsacks, and regression testing techniques

Interested in learning more about this topic? Visit this web site to see related presentations: https://www.gregorykapfhammer.com/research/presentations/

Gregory Kapfhammer

February 08, 2008
Tweet

More Decks by Gregory Kapfhammer

Other Decks in Programming

Transcript

  1. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Set Covers, Knapsacks, and
    Regression Testing Techniques
    Gregory M. Kapfhammer†
    Department of Computer Science
    Allegheny College, Pennsylvania, USA
    http://www.cs.allegheny.edu/~gkapfham/
    Madras Christian College, February 2008
    †In Conjunction with Mary Lou Soffa, Kristen Walcott (UVa/CS)
    Suvarshi Bhadra, Joshua Geiger, Adam Smith, Gavilan Steinman, Yuting Zhang (Allegheny/CS)
    Featuring images from the WikiMedia Commons
    1 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  2. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Presentation Outline
    1 Testing Challenges
    2 Reduction and Prioritization
    3 Time-Aware Prioritization
    4 Future Work
    5 Conclusions
    2 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  3. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    The Challenge of Software Testing
    I shall not deny that the construction of these testing
    programs has been a major intellectual effort: to
    convince oneself that one has not overlooked “a
    relevant state” and to convince oneself that the testing
    programs generate them all is no simple matter. The
    encouraging thing is that (as far as we know!) it could
    be done.
    Edsger W. Dijkstra, Communications of the ACM, 1968
    Additional Challenge: understanding the fundamental
    difficulties associated with practical testing techniques
    3 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  4. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    NP-Complete Problems
    Minimal Set Cover 0/1 Knapsack
    Question: what are the connections between the theory and
    practice of computer science?
    4 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  5. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Halting Problem
    Program
    Halting
    Detector
    Inputs
    Yes!
    No!
    Question: What approach can we take in order to completely
    implement the halting detector?
    You can assume any existing hardware platform (e.g., fast
    multi-core processor) or software application (e.g., compiler).
    5 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  6. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Minimal Set Cover Problem
    S1
    E1 E2
    E3
    S2
    E4
    E5
    E6
    S3
    E7
    E8
    E9
    S4
    Sets contain elements (e.g., S2
    → E4
    means that set S2
    contains the element E4
    )
    Question: Can you find a subset of the sets that will
    contain all of the elements?
    This problem is NP-complete (see Garey and Johnson) and
    yet it has many practical applications in software testing
    6 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  7. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    0/1 Knapsack Problem
    Knapsack
    Capacity = 12
    Cost = 4
    Benefit = 6
    Cost = 10
    Benefit = 9
    Cost = 7
    Benefit = 4
    Cost = 9
    Benefit = 8
    Question: Can you select items so that you maximize the benefit
    while ensuring that the cost does not exceed the capacity?
    This problem is NP-complete (see Garey and Johnson) and yet it
    also has many practical applications in both software and finance
    7 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  8. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    0/1 Knapsack Problem
    Knapsack
    Capacity = 12
    Cost = 4
    Benefit = 6
    Cost = 10
    Benefit = 9
    Cost = 7
    Benefit = 4
    Cost = 9
    Benefit = 8
    Question: Can you select items so that you maximize the benefit
    while ensuring that the cost does not exceed the capacity?
    This problem is NP-complete (see Garey and Johnson) and yet it
    also has many practical applications in both software and finance
    8 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  9. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Presentation Outline
    1 Testing Challenges
    2 Reduction and Prioritization
    3 Time-Aware Prioritization
    4 Future Work
    5 Conclusions
    9 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  10. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Approaches to Software Testing
    Specification
    Testing
    Testing
    Random
    MTTF(P) = k
    Reliability
    Software
    Input
    Freq
    Testing
    Structural
    Input Space
    Testing isolates defects and establishes a confidence in the
    correctness of a software application
    10 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  11. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Where are Defects Located?
    System
    Operating
    File
    System
    Virtual
    P
    Machine
    Graphical
    Interface
    Database
    Byte Code
    Input
    a 5 print ... exit Final Result: 45
    Output
    Defects may exist in the individual components or the interactions
    11 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  12. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    What is a Test Case?
    Input
    Method
    Under Test
    Output
    Test
    Oracle
    Expected
    Output
    Verdict
    Tear Down
    Set Up
    Test suite executor runs each test case independently
    Each test invokes a method within the program and then compares
    the actual and expected output values
    12 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  13. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Using Tests to Find Faults
    Test Case Faults
    f1
    f2
    f3
    f4
    f5
    T1
    × ×
    T2
    × ×
    T3
    × × ×
    T4
    × × ×
    T5
    × ×
    The Importance of Test Ordering: T3
    , T4
    , T1
    , T2
    , T5
    detects
    faults more rapidly than T1
    , T2
    , T3
    , T4
    , T5
    Since we do not have a priori knowledge of the faults that exist
    within a program, we must use a proxy like coverage
    13 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  14. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Test Coverage Monitoring
    Program
    Instrumentation
    Adequacy
    Criterion
    Residual Test
    Requirements
    Cummulative
    Adequacy Calculator
    Instrumented
    Program
    Test Suite
    Execution
    Covered
    Requirements
    Structural adequacy criteria focus on the coverage of nodes,
    edges, paths, and definition-use associations
    Instrumentation probes track the coverage of test requirements
    14 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  15. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Finding the Overlap in Coverage
    T2 T3
    T6
    T9
    R
    R1
    R2
    T4
    T8
    T11
    R3
    T12
    R4
    T1
    R4
    R4
    R4
    T5
    R4
    R4
    T10
    R4
    T11
    R5
    T5
    R5
    R6
    R6
    T10
    R7
    R7
    T7
    R7
    T10
    Rj
    → Ti
    means that requirement Rj
    is covered by test Ti
    T = T2
    , T3
    , T6
    , T9
    covers all of the test requirements
    Test suite reduction discards the test cases that
    redundantly cover the test requirements
    15 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  16. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Regression Test Suite Prioritization
    Begin Coverage Report End
    VSRT Repeat
    Program
    Test Suite
    Prioritization
    Original
    Test Suite
    Modified
    Test Suite
    Test Suite
    Execution
    Testing Results
    GRT Repeat
    Overview
    Prioritization re-orders the tests so that they cover the
    requirements more effectively
    GRT uses the same prioritization across multiple runs of the test
    suite whereas VSRT creates a new prioritization for each test run
    16 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  17. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Comparing Prioritization Techniques
    GPO
    GPC
    GPV
    GPR
    RVP
    ORP
    RAP
    0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
    Coverage Effectiveness
    Prioritization Technique
    Comparing Test Suite Prioritizers
    Which prioritization technique is the best?
    17 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  18. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Presentation Outline
    1 Testing Challenges
    2 Reduction and Prioritization
    3 Time-Aware Prioritization
    4 Future Work
    5 Conclusions
    18 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  19. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Time-Aware Orderings (Fault Data)
    f1
    f2
    f3
    f4
    f5 f6
    f7 f8
    T1
    X X X X X X X
    T2
    X
    T3
    X X
    T4
    X X X
    T5 X X X
    T6
    X X X
    It is very common to confront a testing time budget
    Question: If fault information is known and there is a testing time
    limit, then what is the best ordering?
    19 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  20. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Time-Aware Orderings (Faults and Costs)
    # Faults Time Cost Avg. Faults per Min.
    T1
    7 9 0.778
    T2
    1 1 1.000
    T3
    2 3 0.667
    T4
    3 4 0.750
    T5 3 4 0.750
    T6
    3 4 0.750
    When test case cost varies, then some tests are able to detect
    fault more rapidly than the others
    Question: What is the best ordering for this test suite?
    20 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  21. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Time-Aware Orderings (Comparison)
    Time Limit: 12 minutes
    Fault Time APFD Intelligent
    T1
    T2
    T2
    T5
    T3
    T1
    T4
    T4
    T3
    T5
    Tot. Faults 7 8 7 8
    Tot. Time 9 12 10 11
    The existence of a time limit prevents the use of traditional minimal
    set cover solvers that only look at overlap
    When fault information is not available, we can use coverage
    21 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  22. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Comparing Time-Aware Prioritizers
    Traditional 0/1 Knapsack Solvers Genetic Algorithm Techniques
    Empirical Results: prioritizers that consider coverage overlap
    take longer to re-order, but they arrive at good orderings
    22 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  23. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Presentation Outline
    1 Testing Challenges
    2 Reduction and Prioritization
    3 Time-Aware Prioritization
    4 Future Work
    5 Conclusions
    23 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  24. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Search-Based Test Suite Prioritization
    0 20 40 60 80 100 120
    0.50 0.55 0.60 0.65 0.70 0.75
    Test Prioritization Index
    Coverage Effectiveness Value
    Coverage Effectiveness Values
    Use heuristic search (HC, SANN, GA) to prioritize the test suite
    24 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  25. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Detailed Empirical Evaluations
    Synthetic Test Suites Real World Programs
    Conduct experiments to systematically study the trade-offs
    associated with testing techniques by using both synthetic
    coverage and real world applications
    25 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  26. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Presentation Outline
    1 Testing Challenges
    2 Reduction and Prioritization
    3 Time-Aware Prioritization
    4 Future Work
    5 Conclusions
    26 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  27. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Concluding Remarks
    Minimal Set Cover 0/1 Knapsack
    Establishes a connection between practical regression testing
    challenges and NP-complete problems
    Many approaches to testing are now ready for integration into
    frameworks such as JUnit
    27 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide

  28. Testing Challenges Reduction and Prioritization Time-Aware Prioritization Future Work Conclusions
    Personal Reflections
    So now, come back to your God! Act
    on the principles of love and justice,
    and always live in confident
    dependence on your God.
    Hosea 12:6 (New Living Translation)
    Please keep in touch!
    http://www.cs.allegheny.edu/~gkapfham/
    28 / 28
    Set Covers, Knapsacks, and , Regression Testing Techniques

    View Slide