Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

Interested in learning more about this topic? Visit this web site to read a related post: https://www.gregorykapfhammer.com/testing/research/advice/2017/07/03/Regression-Testing-Costs/

Gregory Kapfhammer

May 09, 2012
Tweet

More Decks by Gregory Kapfhammer

Other Decks in Research

Transcript

  1. Software Quality Improvement through
    Repeated Test Execution: An Exploration of
    the Present and Future of Regression Testing
    Gregory M. Kapfhammer†
    Department of Computer Science
    Allegheny College
    http://www.cs.allegheny.edu/∼gkapfham/
    University of Delhi – May 9, 2012
    †Joint with Jonathan Miller Kauffman (Allegheny College)

    View full-size slide

  2. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Important Points
    Presenter Introduction: Gregory M. Kapfhammer
    test
    testing
    software
    suites
    prioritization
    components
    coverage
    empirically
    regression
    suite
    applications
    evaluating
    algorithm
    analysis
    data
    effectiveness
    empirical
    genetic
    performance
    understanding
    using
    approach
    commercialofftheshelf
    comparison
    comprehensive
    constrained
    creation
    databasecentric
    environments
    execution
    finding
    framework
    identifying
    interactive
    java
    methods
    multiplots
    mutation
    party
    prioritized
    reduction
    relational
    study
    third
    timeaware
    towards
    105
    adequacy
    approaches
    array
    automatically
    building
    call
    challenges
    chapter
    communication
    compare
    compressing
    computer
    conditional
    cost
    cots
    covering
    creating
    criteria
    database
    databaseaware
    databasedriven
    databases
    declarative
    dependable
    detection
    devices
    distributed
    distributing
    duringsearchbased
    dynamic
    efficiency
    efficient
    engineering
    environment
    evaluate
    examination
    executing
    experimental
    family
    flow
    forward
    frameworks
    free
    generation
    greedy
    gui
    hamiltonian
    handbook
    heaps
    implementation
    improve
    incorporating
    increase
    information
    initial
    intranode
    invariant
    javaspace
    javaspacebased
    kernel
    knapsack
    linux
    measurement
    memory
    method
    monitoring
    operators
    paths
    poster
    potential
    preliminary
    primitives
    prioritizations
    prioritizers
    prioritizing
    problematic
    receive
    remote
    reports
    resource
    resourceconstrained
    results
    role
    runtime
    science
    searchbased
    selection
    solutions
    solvers
    space
    studies
    studying
    supported
    synthetic
    techniques
    transmission
    transparently
    trees
    tuple
    unstructured
    wrappers
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  3. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Everywhere
    Computer
    Server
    Program
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  4. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Everywhere
    Computer
    Server
    Program
    Computer
    Server
    Program
    Desktop
    Computer
    Program
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  5. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Everywhere
    Computer
    Server
    Program
    Computer
    Server
    Program
    Desktop
    Computer
    Program
    Desktop
    Computer
    Program
    Mobile
    Computer
    Program
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  6. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Everywhere
    Computer
    Server
    Program
    Computer
    Server
    Program
    Desktop
    Computer
    Program
    Desktop
    Computer
    Program
    Mobile
    Computer
    Program
    Mobile
    Computer
    Program
    Household
    Appliance
    Program
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  7. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Everywhere
    Computer
    Server
    Program
    Computer
    Server
    Program
    Desktop
    Computer
    Program
    Desktop
    Computer
    Program
    Mobile
    Computer
    Program
    Mobile
    Computer
    Program
    Household
    Appliance
    Program
    Scientific
    Device
    Program
    Household
    Appliance
    Program
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  8. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Everywhere
    Computer
    Server
    Program
    Computer
    Server
    Program
    Desktop
    Computer
    Program
    Desktop
    Computer
    Program
    Mobile
    Computer
    Program
    Mobile
    Computer
    Program
    Household
    Appliance
    Program
    Scientific
    Device
    Program
    Household
    Appliance
    Program
    Network
    Router
    Program
    Scientific
    Device
    Program
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  9. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Complex
    Computer
    Software
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  10. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Complex
    Computer
    Software
    Lines of
    Code
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  11. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Complex
    Computer
    Software
    Lines of
    Code
    Numerous
    Features
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  12. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Complex
    Computer
    Software
    Lines of
    Code
    Numerous
    Features
    Feature
    Interactions
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  13. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Complex
    Computer
    Software
    Lines of
    Code
    Numerous
    Features
    Feature
    Interactions
    Execution
    Environments
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  14. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Complex
    Computer
    Software
    Lines of
    Code
    Numerous
    Features
    Feature
    Interactions
    Execution
    Environments
    Software entities are more complex for their size than per-
    haps any other human construct - Frederick P. Brooks, Jr.
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  15. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Evolving
    Execution
    Environment
    Program
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  16. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Evolving
    Execution
    Environment
    Program
    Execution
    Environment
    Program
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  17. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Evolving
    Execution
    Environment
    Program
    Execution
    Environment
    Program
    Program Changed because of the addition
    of a new feature or the correction of a defect
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  18. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Evolving
    Execution
    Environment
    Program
    Execution
    Environment
    Program
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  19. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Evolving
    Execution
    Environment
    Program
    Execution
    Environment
    Program
    Execution
    Environment
    Program
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  20. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Software is Evolving
    Execution
    Environment
    Program
    Execution
    Environment
    Program
    Execution
    Environment
    Program
    Execution Environment Changed due to an up-
    grade in a kernel, device driver, or virtual machine
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  21. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Regression Testing to the Rescue
    Computer
    Software
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  22. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Regression Testing to the Rescue
    Computer
    Software
    Pervasive
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  23. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Regression Testing to the Rescue
    Computer
    Software
    Pervasive
    Complex
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  24. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Regression Testing to the Rescue
    Computer
    Software
    Pervasive
    Complex
    Evolving
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  25. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Software Challenges
    Regression Testing to the Rescue
    Computer
    Software
    Pervasive
    Complex
    Evolving
    Regression Testing supports the efficient construction
    of pervasive software that is complex and rapidly evolving
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  26. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Case?
    Method
    Under Test
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  27. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Case?
    Method
    Under Test
    Input
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  28. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Case?
    Method
    Under Test
    Input Output
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  29. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Case?
    Method
    Under Test
    Test
    Set Up
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  30. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Case?
    Method
    Under Test
    Test
    Set Up
    Input
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  31. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Case?
    Method
    Under Test
    Test
    Set Up
    Input Output
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  32. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Case?
    Method
    Under Test
    Test
    Set Up
    Input Output
    Test
    Clean Up
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  33. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Case?
    Method
    Under Test
    Test
    Set Up
    Input Output
    Test
    Clean Up
    Test
    Oracle
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  34. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Case?
    Method
    Under Test
    Test
    Set Up
    Input Output
    Test
    Clean Up
    Test
    Oracle
    Expected
    Output
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  35. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Case?
    Method
    Under Test
    Test
    Set Up
    Input Output
    Test
    Clean Up
    Test
    Oracle
    Expected
    Output
    Test
    Verdict
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  36. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Case?
    Method
    Under Test
    Test
    Set Up
    Input Output
    Test
    Clean Up
    Test
    Oracle
    Expected
    Output
    Test
    Verdict
    Expected
    Output
    Output
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  37. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Case?
    Method
    Under Test
    Test
    Set Up
    Input Output
    Test
    Clean Up
    Test
    Oracle
    Expected
    Output
    Test
    Verdict
    Expected
    Output
    Output
    Test
    Verdict
    The test case passes and the code is correct!
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  38. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Case?
    Method
    Under Test
    Test
    Set Up
    Input Output
    Test
    Clean Up
    Test
    Oracle
    Expected
    Output
    Test
    Verdict
    Expected
    Output
    Output
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  39. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Case?
    Method
    Under Test
    Test
    Set Up
    Input Output
    Test
    Clean Up
    Test
    Oracle
    Expected
    Output
    Test
    Verdict
    Expected
    Output
    Output
    Test
    Verdict
    The test case fails and a defect is found!
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  40. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Suite?
    T1
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  41. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Suite?
    T1 T2
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  42. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Suite?
    T1 T2
    T3
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  43. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Suite?
    T1 T2
    T3 T4
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  44. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Suite?
    T1 T2
    T3 T4
    . . .
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  45. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Suite?
    T1 T2
    T3 T4
    . . . Tn
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  46. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Suite?
    T1 T2
    T3 T4
    . . . Tn
    Organize the Test Cases into a Test Suite
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  47. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Suite?
    T1 T2
    T3 T4
    . . . Tn
    Organize the Test Cases into a Test Suite
    Tool Support for Software Testing?
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  48. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Suite?
    T1 T2
    T3 T4
    . . . Tn
    Organize the Test Cases into a Test Suite
    Tool Support for Software Testing?
    JUnit
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  49. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Suite?
    T1 T2
    T3 T4
    . . . Tn
    Organize the Test Cases into a Test Suite
    Tool Support for Software Testing?
    JUnit Apache Ant
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  50. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    What is a Test Suite?
    T1 T2
    T3 T4
    . . . Tn
    Organize the Test Cases into a Test Suite
    Tool Support for Software Testing?
    JUnit Apache Ant Eclipse
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  51. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    Test Suite Management
    T1 T2
    T3 T4
    . . . Tn
    Organize the Test Cases into a Test Suite
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  52. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    Test Suite Management
    T1 T2
    T3 T4
    . . . Tn
    Regression Testing Technique
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  53. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    Test Suite Management
    T1 T2
    T3 T4
    . . . Tn
    Regression Testing Technique
    What if Some Test Cases are More Effective?
    T3 Tn
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  54. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    Test Suite Management
    T1 T2
    T3 T4
    . . . Tn
    Regression Testing Technique
    What if Some Test Cases are More Effective?
    T3 Tn
    Prioritization
    T3 Tn T1 T4
    . . . T2
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  55. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    Test Suite Management
    T1 T2
    T3 T4
    . . . Tn
    Regression Testing Technique
    What if Some Test Cases are More Effective?
    T3 Tn
    Prioritization
    T3 Tn T1 T4
    . . . T2
    T1 T2
    T3 T4
    . . . Tn
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  56. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    Test Suite Management
    T1 T2
    T3 T4
    . . . Tn
    Regression Testing Technique
    T3 Tn
    Prioritization
    T3 Tn T1 T4
    . . . T2
    T1 T2
    T3 T4
    . . . Tn
    What if Some Test Cases are Redundant?
    T1 T2
    T3 T4
    . . . Tn
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  57. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    Test Suite Management
    T1 T2
    T3 T4
    . . . Tn
    Regression Testing Technique
    T3 Tn
    Prioritization
    T3 Tn T1 T4
    . . . T2
    T1 T2
    T3 T4
    . . . Tn
    What if Some Test Cases are Redundant?
    T1 T2
    T3 T4
    . . . Tn
    Reduction
    T4
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  58. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    Test Suite Management
    T1 T2
    T3 T4
    . . . Tn
    Regression Testing Technique
    T3 Tn
    Prioritization
    T3 Tn T1 T4
    . . . T2
    T1 T2
    T3 T4
    . . . Tn
    What if Some Test Cases are Redundant?
    T1 T2
    T3 T4
    . . . Tn
    Reduction
    T4
    T1 T2
    T3 T4
    . . . Tn
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  59. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    Test Suite Management
    T1 T2
    T3 T4
    . . . Tn
    Regression Testing Technique
    T3 Tn
    Prioritization
    T3 Tn T1 T4
    . . . T2
    T1 T2
    T3 T4
    . . . Tn
    T1 T2
    T3 T4
    . . . Tn
    Reduction
    T4
    T1 T2
    T3 T4
    . . . Tn
    What if Only Certain Tests are Needed?
    T2 Tn
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  60. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Testing Opportunities
    Test Suite Management
    T1 T2
    T3 T4
    . . . Tn
    Regression Testing Technique
    T3 Tn
    Prioritization
    T3 Tn T1 T4
    . . . T2
    T1 T2
    T3 T4
    . . . Tn
    T1 T2
    T3 T4
    . . . Tn
    Reduction
    T4
    T1 T2
    T3 T4
    . . . Tn
    What if Only Certain Tests are Needed?
    T2 Tn
    Selection
    T1 T2
    T3 T4
    . . . Tn
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  61. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Model of Regression Testing
    Start
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  62. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Model of Regression Testing
    Start
    Coverage
    Report
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  63. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Model of Regression Testing
    Start
    Coverage
    Report
    Selection
    Reduction
    Prioritization
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  64. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Model of Regression Testing
    Start
    Coverage
    Report
    Selection
    Reduction
    Prioritization
    Original Test Suite
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  65. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Model of Regression Testing
    Start
    Coverage
    Report
    Selection
    Reduction
    Prioritization
    Original Test Suite
    Modified
    Test
    Suite
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  66. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Model of Regression Testing
    Start
    Coverage
    Report
    Selection
    Reduction
    Prioritization
    Original Test Suite
    Modified
    Test
    Suite
    Test Suite
    Execution
    Test Coverage
    Monitoring
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  67. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Model of Regression Testing
    Start
    Coverage
    Report
    Selection
    Reduction
    Prioritization
    Original Test Suite
    Modified
    Test
    Suite
    Test Suite
    Execution
    Test Coverage
    Monitoring
    Program
    Adequacy
    Criterion
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  68. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Model of Regression Testing
    Start
    Coverage
    Report
    Selection
    Reduction
    Prioritization
    Original Test Suite
    Modified
    Test
    Suite
    Test Suite
    Execution
    Test Coverage
    Monitoring
    Program
    Adequacy
    Criterion
    Test Results
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  69. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Model of Regression Testing
    Start
    Coverage
    Report
    Selection
    Reduction
    Prioritization
    Original Test Suite
    Modified
    Test
    Suite
    Test Suite
    Execution
    Test Coverage
    Monitoring
    Program
    Adequacy
    Criterion
    Test Results End
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  70. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Model of Regression Testing
    Start
    Coverage
    Report
    Selection
    Reduction
    Prioritization
    Original Test Suite
    Modified
    Test
    Suite
    Test Suite
    Execution
    Test Coverage
    Monitoring
    Program
    Adequacy
    Criterion
    Test Results End
    Use the Coverage Report During the Next Round of Regression Testing
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  71. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Model of Regression Testing
    Start
    Coverage
    Report
    Selection
    Reduction
    Prioritization
    Original Test Suite
    Modified
    Test
    Suite
    Test Suite
    Execution
    Test Coverage
    Monitoring
    Program
    Adequacy
    Criterion
    Test Results End
    Use the Same Test Suite for the Next Round of Regression Testing
    Standard Repeat
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  72. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Model of Regression Testing
    Start
    Coverage
    Report
    Selection
    Reduction
    Prioritization
    Original Test Suite
    Modified
    Test
    Suite
    Test Suite
    Execution
    Test Coverage
    Monitoring
    Program
    Adequacy
    Criterion
    Test Results End
    Standard Repeat
    Make a New Test Suite for the Next Round of Regression Testing
    Version Specific Repeat
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  73. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Model of Regression Testing
    Start
    Coverage
    Report
    Selection
    Reduction
    Prioritization
    Original Test Suite
    Modified
    Test
    Suite
    Test Suite
    Execution
    Test Coverage
    Monitoring
    Program
    Adequacy
    Criterion
    Test Results End
    Standard Repeat
    Make a New Test Suite for the Next Round of Regression Testing
    Version Specific Repeat
    Our Tools Support All of the Phases in this Model!
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  74. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Adequacy
    T1 T2
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  75. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Adequacy
    T1 T2
    T3 T4
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  76. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Adequacy
    T1 T2
    T3 T4 T5 T6
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  77. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Adequacy
    T1 T2
    T3 T4 T5 T6 T7 T8
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  78. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Adequacy
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  79. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Adequacy
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  80. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Adequacy
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  81. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Adequacy
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  82. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Adequacy
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  83. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Adequacy
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  84. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Adequacy
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  85. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Adequacy
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  86. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Adequacy
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  87. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Adequacy
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    R12
    R12
    R12
    R12
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  88. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Adequacy
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    R12
    R12
    R12
    R12
    R12
    R12
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  89. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Adequacy
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  90. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Execution
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  91. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Execution
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  92. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Execution
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    JUnit Test Automation Framework
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  93. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Execution
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    JUnit Test Automation Framework
    T1
    Run Test Case
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  94. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Execution
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    JUnit Test Automation Framework
    T1
    T1
    Passing Test Case: OA = OE
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  95. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Execution
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    JUnit Test Automation Framework
    T1
    T1
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  96. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Execution
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    JUnit Test Automation Framework
    T1
    T1 T2
    T3 T4 T5 T6
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  97. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Execution
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    JUnit Test Automation Framework
    T1
    T1 T2
    T3 T4 T5 T6 T7
    Run Test Case
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  98. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Execution
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    JUnit Test Automation Framework
    T1
    T1 T2
    T3 T4 T5 T6 T7
    T7
    Failing Test Case: OA = OE
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  99. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Execution
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    JUnit Test Automation Framework
    T1
    T1 T2
    T3 T4 T5 T6 T7
    T7
    Failing Test Case: OA = OE
    Stop Running T
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  100. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Execution
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    JUnit Test Automation Framework
    T1
    T1 T2
    T3 T4 T5 T6 T7
    T7
    Failing Test Case: OA = OE
    Stop Running T
    T8 T9 T10
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  101. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Execution
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    JUnit Test Automation Framework
    T1
    T1 T2
    T3 T4 T5 T6 T7
    T7
    Failing Test Case: OA = OE
    Stop Running T
    T8 T9 T10
    T8 T9 T10
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  102. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Execution
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    JUnit Test Automation Framework
    T1
    T1 T2
    T3 T4 T5 T6 T7
    T7
    Failing Test Case: OA = OE
    Stop Running T
    T8 T9 T10
    T8 T9 T10
    Continue Running T
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  103. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Suite Execution
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    JUnit Test Automation Framework
    T1
    T1 T2
    T3 T4 T5 T6 T7
    T7
    Failing Test Case: OA = OE
    Stop Running T
    T8 T9 T10
    T8 T9 T10
    Continue Running T
    T8 T9 T10
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  104. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Coverage Monitoring
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  105. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Coverage Monitoring
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  106. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Coverage Monitoring
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    JUnit Test Automation Framework
    Cobertura Test Coverage Monitor
    Proteja Test Suite Manager
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  107. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Coverage Monitoring
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    JUnit Test Automation Framework
    Cobertura Test Coverage Monitor
    Proteja Test Suite Manager
    T1
    Run Test Case
    Collect Per-Test Case Coverage
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  108. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Coverage Monitoring
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  109. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Coverage Monitoring
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  110. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Coverage Monitoring
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  111. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Coverage Monitoring
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    Requirements Set R for ... Statement Coverage
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  112. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Coverage Monitoring
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    Requirements Set R for ... Mutation Coverage
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  113. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Supporting Methods
    Test Coverage Monitoring
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    Requirements Set R for ... Definition-Use Coverage
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  114. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Importance of Test Suite Prioritization
    0 5 10 15
    0 1 2 3 4 5
    Testing Time (l)
    Covered Test Requirements (C(T, l))
    1, 2, 3 CE = 0.3789
    Prioritize to increase the CE of a test suite CE = Actual
    Ideal
    ∈ [0, 1]
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  115. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Importance of Test Suite Prioritization
    0 5 10 15
    0 1 2 3 4 5
    Testing Time (l)
    Covered Test Requirements (C(T, l))
    1, 2, 3 CE = 0.3789
    1,2,3
    Test Orderings
    Original ordering exhibits poor effectiveness score - CE = 0.3789
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  116. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Importance of Test Suite Prioritization
    0 5 10 15
    0 1 2 3 4 5
    Testing Time (l)
    Covered Test Requirements (C(T, l))
    1, 3, 2 CE = 0.5053
    1,2,3
    Test Orderings
    1,2,3
    1,3,2
    Different ordering improves the effectiveness score - CE = 0.5053
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  117. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Importance of Test Suite Prioritization
    0 5 10 15
    0 1 2 3 4 5
    Testing Time (l)
    Covered Test Requirements (C(T, l))
    2, 3, 1 CE = 0.4316
    1,2,3
    Test Orderings
    1,2,3
    1,3,2
    1,2,3
    1,3,2
    2,3,1
    Some orderings have less improved scores - CE = 0.4316
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  118. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Importance of Test Suite Prioritization
    0 5 10 15
    0 1 2 3 4 5
    Testing Time (l)
    Covered Test Requirements (C(T, l))
    3, 1, 2 CE = 0.5789
    1,2,3
    Test Orderings
    1,2,3
    1,3,2
    1,2,3
    1,3,2
    2,3,1
    1,2,3
    1,3,2
    2,3,1
    3,1,2
    Best ordering shows a higher effectiveness scores - CE = 0.5789
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  119. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Importance of Test Suite Prioritization
    0 5 10 15
    0 1 2 3 4 5
    Testing Time (l)
    Covered Test Requirements (C(T, l))
    3, 1, 2 CE = 0.5789
    1,2,3
    Test Orderings
    1,2,3
    1,3,2
    1,2,3
    1,3,2
    2,3,1
    1,2,3
    1,3,2
    2,3,1
    3,1,2
    Greedy methods often produce high-effectiveness orderings
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  120. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Importance of Test Suite Prioritization
    0 5 10 15
    0 1 2 3 4 5
    Testing Time (l)
    Covered Test Requirements (C(T, l))
    3, 1, 2 CE = 0.5789
    1,2,3
    Test Orderings
    1,2,3
    1,3,2
    1,2,3
    1,3,2
    2,3,1
    1,2,3
    1,3,2
    2,3,1
    3,1,2
    Search-based techniques may have some desirable characteristics
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  121. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  122. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  123. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  124. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  125. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  126. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    R12
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  127. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    R12
    ratio(T1) = cost(T1)
    coverage(T1)
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  128. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    R12
    ratio(T1) = 5
    3
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  129. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    R12
    ratio(T1) = 1.66
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  130. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    1.66
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  131. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    1.66
    R12
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  132. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    1.66
    R12
    ratio(T5) = cost(T5)
    coverage(T5)
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  133. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    1.66
    R12
    ratio(T5) = 8
    3
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  134. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    1.66
    R12
    ratio(T5) = 2.66
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  135. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    1.66 2.66
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  136. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    1.66 2.66 ratio(T1) < ratio(T5)
    Prefer T1 over T5
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  137. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    Test Suite T = T1, T2, . . . , T9, T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    1.66 2.66
    Proceed incrementally,
    picking the test case with the
    lowest ratio value for the
    uncovered requirements
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  138. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    T8 T4
    T9 T1
    T10 T3 T7 T2
    T6 T5
    Test Suite T = T8, T4, T9, T1, T10, T3, T7, T2, T6, T5
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  139. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    T8 T4
    T9 T1
    T10 T3 T7 T2
    T6 T5
    R12
    1 2 3 5 2 4 4 6 10 8
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  140. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    T8 T4
    T9 T1
    T10 T3 T7 T2
    T6 T5
    R12
    1 2 3 5 2 4 4 6 10 8
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  141. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    T8 T4
    T9 T1
    T10 T3 T7 T2
    T6 T5
    R12
    1 2 3 5 2 4 4 6 10 8
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  142. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    T8 T4
    T9 T1
    T10 T3 T7 T2
    T6 T5
    R12
    1 2 3 5 2 4 4 6 10 8
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  143. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    T8 T4
    T9 T1
    T10 T3 T7 T2
    T6 T5
    R12
    1 2 3 5 2 4 4 6 10 8
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  144. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    T8 T4
    T9 T1
    T10 T3 T7 T2
    T6 T5
    R12
    1 2 3 5 2 4 4 6 10 8
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  145. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    T8 T4
    T9 T1
    T10 T3 T7 T2
    T6 T5
    R12
    1 2 3 5 2 4 4 6 10 8
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  146. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    T8 T4
    T9 T1
    T10 T3 T7 T2
    T6 T5
    R12
    1 2 3 5 2 4 4 6 10 8
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  147. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    T8 T4
    T9 T1
    T10 T3 T7 T2
    T6 T5
    R12
    1 2 3 5 2 4 4 6 10 8
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  148. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    T8 T4
    T9 T1
    T10 T3 T7 T2
    T6 T5
    R12
    1 2 3 5 2 4 4 6 10 8
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  149. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    T8 T4
    T9 T1
    T10 T3 T7 T2
    T6 T5
    T2
    T6 T5
    Test Suite T = T8, T4, T9, T1, T10, T3, T7
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  150. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Greedy Algorithms
    T1 T2
    T3 T4 T5 T6 T7 T8 T9 T10
    R1 R2
    R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
    Requirements Set R = {R1, R2, . . . , R11, R12}
    T8 T4
    T9 T1
    T10 T3 T7 T2
    T6 T5
    T2
    T6 T5
    Test Suite T = T8, T4, T9, T1, T10, T3, T7
    R12
    R12
    R12
    R12
    R12
    R12
    R12
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  151. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Limitations of Greedy Algorithms
    T1
    T3 T2 T4
    Possible configuration of the coverage report
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  152. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Limitations of Greedy Algorithms
    T1
    T3 T2 T4
    R1 R2
    R3
    Possible configuration of the coverage report
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  153. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Limitations of Greedy Algorithms
    T1
    T3 T2 T4
    R1 R2
    R3
    T1
    T3 T2 T4
    R1 R2
    R3
    Possible configuration of the coverage report
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  154. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Limitations of Greedy Algorithms
    T1
    T3 T2 T4
    R1 R2
    R3
    T1
    T3 T2 T4
    R1 R2
    R3
    time(T1) = 1
    time(T2) = 1
    time(T4) = 1
    Execution time of the test cases may mislead greedy
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  155. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Limitations of Greedy Algorithms
    T1
    T3 T2 T4
    R1 R2
    R3
    T1
    T3 T2 T4
    R1 R2
    R3
    time(T1) = 1
    time(T2) = 1
    time(T4) = 1
    time(T3) = 2.45
    Execution time of the test cases may mislead greedy
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  156. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Limitations of Greedy Algorithms
    T1
    T3 T2 T4
    R1 R2
    R3
    T1
    T3 T2 T4
    R1 R2
    R3
    time(T1) = 1
    time(T2) = 1
    time(T4) = 1
    time(T3) = 2.45
    T1 T2
    T3 T4
    Original ordering has low effectiveness score
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  157. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Limitations of Greedy Algorithms
    T1
    T3 T2 T4
    R1 R2
    R3
    T1
    T3 T2 T4
    R1 R2
    R3
    time(T1) = 1
    time(T2) = 1
    time(T4) = 1
    time(T3) = 2.45
    T1 T2
    T3 T4
    CE(T) = 0.54
    Original ordering has low effectiveness score
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  158. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Limitations of Greedy Algorithms
    T1
    T3 T2 T4
    R1 R2
    R3
    T1
    T3 T2 T4
    R1 R2
    R3
    time(T1) = 1
    time(T2) = 1
    time(T4) = 1
    time(T3) = 2.45
    T1 T2
    T3 T4
    T3 T1 T2 T4
    Greedy method constructs suite with marginal improvement
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  159. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Limitations of Greedy Algorithms
    T1
    T3 T2 T4
    R1 R2
    R3
    T1
    T3 T2 T4
    R1 R2
    R3
    time(T1) = 1
    time(T2) = 1
    time(T4) = 1
    time(T3) = 2.45
    T1 T2
    T3 T4
    T3 T1 T2 T4
    CE(T ) = 0.55
    Greedy method constructs suite with marginal improvement
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  160. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Limitations of Greedy Algorithms
    T1
    T3 T2 T4
    R1 R2
    R3
    T1
    T3 T2 T4
    R1 R2
    R3
    time(T1) = 1
    time(T2) = 1
    time(T4) = 1
    time(T3) = 2.45
    T1 T2
    T3 T4
    T3 T1 T2 T4
    CE(T ) = 0.55
    Greedy can exhibit high run-times (Jiang et al. ASE 2009)
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  161. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Limitations of Greedy Algorithms
    T1
    T3 T2 T4
    R1 R2
    R3
    T1
    T3 T2 T4
    R1 R2
    R3
    time(T1) = 1
    time(T2) = 1
    time(T4) = 1
    time(T3) = 2.45
    T1 T2
    T3 T4
    T3 T1 T2 T4
    T1 T2 T4
    T3
    Genetic may find better orderings (Conrad et al. GECCO 2010)
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  162. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Limitations of Greedy Algorithms
    T1
    T3 T2 T4
    R1 R2
    R3
    T1
    T3 T2 T4
    R1 R2
    R3
    time(T1) = 1
    time(T2) = 1
    time(T4) = 1
    time(T3) = 2.45
    T1 T2
    T3 T4
    T3 T1 T2 T4
    T1 T2 T4
    T3
    CE(T ) = 0.63
    Genetic may find better orderings (Conrad et al. GECCO 2010)
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  163. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Limitations of Greedy Algorithms
    T1
    T3 T2 T4
    R1 R2
    R3
    T1
    T3 T2 T4
    R1 R2
    R3
    time(T1) = 1
    time(T2) = 1
    time(T4) = 1
    time(T3) = 2.45
    T1 T2
    T3 T4
    T3 T1 T2 T4
    T1 T2 T4
    T3
    CE(T ) = 0.63
    Search-based algorithms are amenable to parallelization
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  164. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Limitations of Greedy Algorithms
    T1
    T3 T2 T4
    R1 R2
    R3
    T1
    T3 T2 T4
    R1 R2
    R3
    time(T1) = 1
    time(T2) = 1
    time(T4) = 1
    time(T3) = 2.45
    T1 T2
    T3 T4
    T3 T1 T2 T4
    T1 T2 T4
    T3
    CE(T ) = 0.63
    Search-based algorithms support “human in the loop”
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  165. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Limitations of Greedy Algorithms
    T1
    T3 T2 T4
    R1 R2
    R3
    T1
    T3 T2 T4
    R1 R2
    R3
    time(T1) = 1
    time(T2) = 1
    time(T4) = 1
    time(T3) = 2.45
    T1 T2
    T3 T4
    T3 T1 T2 T4
    T1 T2 T4
    T3
    CE(T ) = 0.63
    Search-based algorithms construct diverse test orderings
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  166. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Hill Climbing Algorithm
    Explore the “neighborhood” of test suites from a starting point
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  167. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Hill Climbing Algorithm
    Explore the “neighborhood” of test suites from a starting point
    T1, T2, T3, T4, T5
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  168. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Hill Climbing Algorithm
    Explore the “neighborhood” of test suites from a starting point
    T1, T2, T3, T4, T5
    T2, T1, T3, T4, T5
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  169. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Hill Climbing Algorithm
    Explore the “neighborhood” of test suites from a starting point
    T1, T2, T3, T4, T5
    T2, T1, T3, T4, T5
    T3, T2, T1, T4, T5
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  170. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Hill Climbing Algorithm
    Explore the “neighborhood” of test suites from a starting point
    T1, T2, T3, T4, T5
    T2, T1, T3, T4, T5
    T3, T2, T1, T4, T5
    T4, T2, T3, T1, T5
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  171. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Hill Climbing Algorithm
    Explore the “neighborhood” of test suites from a starting point
    T1, T2, T3, T4, T5
    T2, T1, T3, T4, T5
    T3, T2, T1, T4, T5
    T4, T2, T3, T1, T5
    T5, T2, T3, T4, T1
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  172. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Hill Climbing Algorithm
    fitness
    time
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  173. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Hill Climbing Algorithm
    fitness
    time
    T
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  174. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Hill Climbing Algorithm
    fitness
    time
    T
    T
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  175. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Hill Climbing Algorithm
    fitness
    time
    T
    T
    T
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  176. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Hill Climbing Algorithm
    fitness
    time
    T
    T
    T
    T
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  177. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Key Algorithms
    Hill Climbing Algorithm
    fitness
    time
    T
    T
    T
    T
    T
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  178. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Model for Experimentation
    How Do I Evaluate Regression Testing Methods?
    Start
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  179. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Model for Experimentation
    How Do I Evaluate Regression Testing Methods?
    Start
    Programs
    Test Suites
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  180. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Model for Experimentation
    How Do I Evaluate Regression Testing Methods?
    Start
    Programs
    Test Suites
    Conduct
    Experiments
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  181. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Model for Experimentation
    How Do I Evaluate Regression Testing Methods?
    Start
    Programs
    Test Suites
    Conduct
    Experiments
    Regression Testing Techniques
    Technique Configurations
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  182. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Model for Experimentation
    How Do I Evaluate Regression Testing Methods?
    Start
    Programs
    Test Suites
    Conduct
    Experiments
    Regression Testing Techniques
    Technique Configurations
    Data Sets
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  183. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Model for Experimentation
    How Do I Evaluate Regression Testing Methods?
    Start
    Programs
    Test Suites
    Conduct
    Experiments
    Regression Testing Techniques
    Technique Configurations
    Data Sets
    Visualization
    Statistical Analysis
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  184. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Model for Experimentation
    How Do I Evaluate Regression Testing Methods?
    Start
    Programs
    Test Suites
    Conduct
    Experiments
    Regression Testing Techniques
    Technique Configurations
    Data Sets
    Visualization
    Statistical Analysis
    Graphs
    Diagrams
    Summarized
    Data Sets
    Models
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  185. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Model for Experimentation
    How Do I Evaluate Regression Testing Methods?
    Start
    Programs
    Test Suites
    Conduct
    Experiments
    Regression Testing Techniques
    Technique Configurations
    Data Sets
    Visualization
    Statistical Analysis
    Graphs
    Diagrams
    Summarized
    Data Sets
    Models
    Iteratively Perform Visualization and Statistical Analysis
    Repeat
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  186. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Model for Experimentation
    How Do I Evaluate Regression Testing Methods?
    Start
    Programs
    Test Suites
    Conduct
    Experiments
    Regression Testing Techniques
    Technique Configurations
    Data Sets
    Visualization
    Statistical Analysis
    Graphs
    Diagrams
    Summarized
    Data Sets
    Models
    Iteratively Perform Visualization and Statistical Analysis
    Repeat
    End
    Conduct Experiments with Additional Programs, Test Suites, and Techniques
    Repeat
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  187. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Model for Experimentation
    How Do I Evaluate Regression Testing Methods?
    Start
    Programs
    Test Suites
    Conduct
    Experiments
    Regression Testing Techniques
    Technique Configurations
    Data Sets
    Visualization
    Statistical Analysis
    Graphs
    Diagrams
    Summarized
    Data Sets
    Models
    Repeat
    End
    Conduct Experiments with Additional Programs, Test Suites, and Techniques
    Repeat
    Our tools support all of these tasks!
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  188. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Model for Experimentation
    How Do I Evaluate Regression Testing Methods?
    Start
    Programs
    Test Suites
    Conduct
    Experiments
    Regression Testing Techniques
    Technique Configurations
    Data Sets
    Visualization
    Statistical Analysis
    Graphs
    Diagrams
    Summarized
    Data Sets
    Models
    Repeat
    End
    Conduct Experiments with Additional Programs, Test Suites, and Techniques
    Repeat
    Greedy, Hill Climbing, Random, Adaptive Random, Simulated Annealing, Genetic
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  189. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Case Study Applications
    Application Program Size
    Lines Methods Classes Faults Test Cases
    CommissionEmployee 34 18 2 95 15
    Point 125 26 3 44 13
    DataStructures 189 57 8 324 106
    Employee 192 52 7 84 14
    LoopFinder 193 26 4 34 13
    Sudoku 231 58 4 414 25
    JDepend 1,462 282 35 2,659 39
    Reduction and Prioritization 2,050 211 19 1,412 38
    Barbecue 2,501 422 59 18,312 140
    JodaTime 12,687 3,644 223 20,894 206
    CommonsMath 20,763 4,185 556 6,077 268
    Total 40,427 8,981 920 50,349 877
    Average 3,675 816 84 4,577 80
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  190. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Greedy and Search-Based
    Prioritizer NAPSC Runtime (sec) Application
    GRD 0.98 0.22 Sudoku
    GRD 0.36 0.38 ReductionAndPrioritization
    GRD 0.71 33.88 JodaTime
    HC SA FS 0.98 0.01 Sudoku
    HC SA FS 0.36 0.04 ReductionAndPrioritization
    HC SA FS 0.69 7.88 JodaTime
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  191. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Greedy and Search-Based
    Prioritizer NAPSC Runtime (sec) Application
    GRD 0.98 0.22 Sudoku
    GRD 0.36 0.38 ReductionAndPrioritization
    GRD 0.71 33.88 JodaTime
    HC SA FS 0.98 0.01 Sudoku
    HC SA FS 0.36 0.04 ReductionAndPrioritization
    HC SA FS 0.69 7.88 JodaTime
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  192. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Greedy and Search-Based
    Prioritizer NAPSC Runtime (sec) Application
    GRD 0.98 0.22 Sudoku
    GRD 0.36 0.38 ReductionAndPrioritization
    GRD 0.71 33.88 JodaTime
    HC SA FS 0.98 0.01 Sudoku
    HC SA FS 0.36 0.04 ReductionAndPrioritization
    HC SA FS 0.69 7.88 JodaTime
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  193. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Greedy and Search-Based
    Prioritizer NAPSC Runtime (sec) Application
    GRD 0.98 0.22 Sudoku
    GRD 0.36 0.38 ReductionAndPrioritization
    GRD 0.71 33.88 JodaTime
    HC SA FS 0.98 0.01 Sudoku
    HC SA FS 0.36 0.04 ReductionAndPrioritization
    HC SA FS 0.69 7.88 JodaTime
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  194. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Greedy and Search-Based
    Prioritizer NAPSC Runtime (sec) Application
    GRD 0.98 0.22 Sudoku
    GRD 0.36 0.38 ReductionAndPrioritization
    GRD 0.71 33.88 JodaTime
    HC SA FS 0.98 0.01 Sudoku
    HC SA FS 0.36 0.04 ReductionAndPrioritization
    HC SA FS 0.69 7.88 JodaTime
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  195. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Greedy and Search-Based
    Prioritizer NAPSC Runtime (sec) Application
    GRD 0.98 0.22 Sudoku
    GRD 0.36 0.38 ReductionAndPrioritization
    GRD 0.71 33.88 JodaTime
    HC SA FS 0.98 0.01 Sudoku
    HC SA FS 0.36 0.04 ReductionAndPrioritization
    HC SA FS 0.69 7.88 JodaTime
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  196. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Greedy and Search-Based
    Prioritizer NAPSC Runtime (sec) Application
    GRD 0.98 0.22 Sudoku
    GRD 0.36 0.38 ReductionAndPrioritization
    GRD 0.71 33.88 JodaTime
    HC SA FS 0.98 0.01 Sudoku
    HC SA FS 0.36 0.04 ReductionAndPrioritization
    HC SA FS 0.69 7.88 JodaTime
    Greedy and hill climbing produce comparable orderings
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  197. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Greedy and Search-Based
    Prioritizer NAPSC Runtime (sec) Application
    GRD 0.98 0.22 Sudoku
    GRD 0.36 0.38 ReductionAndPrioritization
    GRD 0.71 33.88 JodaTime
    HC SA FS 0.98 0.01 Sudoku
    HC SA FS 0.36 0.04 ReductionAndPrioritization
    HC SA FS 0.69 7.88 JodaTime
    However, hill climbing is slightly more efficient than greedy
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  198. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Greedy and Search-Based
    Prioritizer NAPSC Runtime (sec) Application
    GRD 0.98 0.22 Sudoku
    GRD 0.36 0.38 ReductionAndPrioritization
    GRD 0.71 33.88 JodaTime
    HC SA FS 0.98 0.01 Sudoku
    HC SA FS 0.36 0.04 ReductionAndPrioritization
    HC SA FS 0.69 7.88 JodaTime
    Greedy produces a slightly better ordering than hill climbing
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  199. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Greedy and Search-Based
    Prioritizer NAPSC Runtime (sec) Application
    GRD 0.98 0.22 Sudoku
    GRD 0.36 0.38 ReductionAndPrioritization
    GRD 0.71 33.88 JodaTime
    HC SA FS 0.98 0.01 Sudoku
    HC SA FS 0.36 0.04 ReductionAndPrioritization
    HC SA FS 0.69 7.88 JodaTime
    But the hill climbing algorithm executes over four times faster!
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  200. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Greedy and Search-Based
    Prioritizer NAPSC Runtime (sec) Application
    GRD 0.98 0.22 Sudoku
    GRD 0.36 0.38 ReductionAndPrioritization
    GRD 0.71 33.88 JodaTime
    HC SA FS 0.98 0.01 Sudoku
    HC SA FS 0.36 0.04 ReductionAndPrioritization
    HC SA FS 0.69 7.88 JodaTime
    A small NAPSC increase may result in a large runtime increase
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  201. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Random and Adaptive Random
    Population Size
    NAPSC
    0.3240
    0.3245
    0.3250
    0.3255
    0.3260
    0.3265
    10 20 30
    JodaTime (JT)
    Statement
    Original NAPSC Score: 0.2784
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  202. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Random and Adaptive Random
    Population Size
    NAPSC
    0.3240
    0.3245
    0.3250
    0.3255
    0.3260
    0.3265
    10 20 30
    JodaTime (JT)
    Statement
    Negligible NAPSC increase as population size increases
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  203. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Random and Adaptive Random
    Population Size
    Runtime (seconds)
    10
    15
    20
    10 20 30
    JodaTime (JT)
    Statement
    Increases in runtime are more marked
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  204. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Random and Adaptive Random
    Similarity Metric
    NAPSC
    0.3194
    0.3195
    0.3196
    0.3197
    Euclidean Jaccard Manhattan
    JodaTime (JT)
    Statement
    Original NAPSC Score: 0.2784
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  205. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Random and Adaptive Random
    Similarity Metric
    NAPSC
    0.3194
    0.3195
    0.3196
    0.3197
    Euclidean Jaccard Manhattan
    JodaTime (JT)
    Statement
    NAPSC changes little as similarity metric is varied
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  206. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Random and Adaptive Random
    Similarity Metric
    NAPSC
    0.3194
    0.3195
    0.3196
    0.3197
    Euclidean Jaccard Manhattan
    JodaTime (JT)
    Statement
    Scores are comparable to those produced by random (0.3240 - 0.3265)
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  207. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Random and Adaptive Random
    Similarity Metric
    Runtime (seconds)
    80
    90
    100
    110
    120
    Euclidean Jaccard Manhattan
    JodaTime (JT)
    Statement
    Adaptive random executes more slowly than random
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  208. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Concrete Examples
    Empirical Results: Random and Adaptive Random
    Similarity Metric
    Runtime (seconds)
    80
    90
    100
    110
    120
    Euclidean Jaccard Manhattan
    JodaTime (JT)
    Statement
    Choose random because it produces comparable NAPSC scores in less time
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  209. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Future
    The Future of Regression Testing
    Research
    • Reproducible research by releasing software tools and data
    • Integrate many existing algorithms into a single framework
    • Develop new forums for publishing important results
    • Software Quality Journal special issue
    • International Workshop on Regression Testing
    Practice
    • Encourage the use of coarse-grained information
    • Try to apply existing tools to industrial programs
    • Participate in community events; publish experience reports
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  210. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Future
    The Future of Regression Testing
    Research
    • Reproducible research by releasing software tools and data
    • Integrate many existing algorithms into a single framework
    • Develop new forums for publishing important results
    • Software Quality Journal special issue
    • International Workshop on Regression Testing
    Practice
    • Encourage the use of coarse-grained information
    • Try to apply existing tools to industrial programs
    • Participate in community events; publish experience reports
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  211. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Conclusions
    Conclusions and Future Work
    Concluding Remarks
    • Comprehensive framework for regression testing
    • Interesting empirical results demonstrate trade-offs
    • Free/open source tools are available for download
    • http://proteja.googlecode.com
    • http://modificare.googlecode.com
    Future Work
    • Add new algorithms for regression testing
    • Conduct experiments with more case study applications
    • Further develop statistically meaningful empirical results
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  212. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion
    Conclusions
    Conclusions and Future Work
    Concluding Remarks
    • Comprehensive framework for regression testing
    • Interesting empirical results demonstrate trade-offs
    • Free/open source tools are available for download
    • http://proteja.googlecode.com
    • http://modificare.googlecode.com
    Future Work
    • Add new algorithms for regression testing
    • Conduct experiments with more case study applications
    • Further develop statistically meaningful empirical results
    Kapfhammer Allegheny College
    Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

    View full-size slide

  213. Software Quality Improvement through
    Repeated Test Execution: An Exploration of
    the Present and Future of Regression Testing
    Gregory M. Kapfhammer
    http://www.cs.allegheny.edu/∼gkapfham/
    Thank you for your attention!
    Contact me with questions and/or comments!

    View full-size slide