Upgrade to Pro — share decks privately, control downloads, hide ads and more …

STICCER: Fast and Effective Database Test Suite Reduction Through Merging of Similar Test Cases

STICCER: Fast and Effective Database Test Suite Reduction Through Merging of Similar Test Cases

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Alsharif2020/

Gregory Kapfhammer

October 26, 2020
Tweet

More Decks by Gregory Kapfhammer

Other Decks in Programming

Transcript

  1. STICCER: Fast and Effective Database
    Test Suite Reduction Through
    Merging of Similar Test Cases
    by Abdullah Alsharif([email protected]), Gregory M. Kapfhammer,
    and Phil McMinn
    1

    View full-size slide

  2. RELATIONAL DATABASES ARE EVERYWHERE AND THE
    BACKBONE OF MOST SOFTWARE SYSTEMS
    2

    View full-size slide

  3. Testing Relational Database Schemas
    Database Schema
    "A good [relational] database schema should have many
    constraints. [Therefore], you should test them“
    Szymon Guz, 2011
    3

    View full-size slide

  4. The Process
    Create a Schema
    Create Test Cases
    (Auto or Manual)
    Reduce Tests
    Run Tests and
    Mutation Analysis
    Evaluate Tests and
    Results
    4

    View full-size slide

  5. Too Many Test Cases
    MANY CHANGES CAN
    INCREASES THE
    NUMBER OF TESTS
    RUNNING TESTS MIGHT
    CONSUME TIME
    INCREASE INSPECTION
    EFFORT (HUMAN
    ORACLE COST)
    5

    View full-size slide

  6. Schema
    Data Types
    Integrity
    Constraints
    6

    View full-size slide

  7. An Example of Faults in a Database Schema
    Fault 1:
    Adding a PK onto the
    name column
    Fault 2:
    expiry = 0 and forgetting to
    add expriy > last_access
    8

    View full-size slide

  8. Testing Integrity Constraints
    • Test 1 - Violating the PK constraint:
    1. Prepare an empty database
    9

    View full-size slide

  9. Testing Integrity Constraints
    1. Prepare an empty database
    2. Create Test INSERTs to exercise the PK
    • Test 1 - Violating the PK constraint:
    10

    View full-size slide

  10. Testing Integrity Constraints
    1. Prepare an empty database
    2. Create Test INSERTs to exercise the PK
    • Test 1 - Violating the PK constraint:
    11

    View full-size slide

  11. Testing Integrity Constraints
    • Test 1 - Violating the PK constraint:
    • Test 2 - Violating the UNIQUE constraint:
    • Test 1 - Violating this constraint:
    1. Prepare an empty database
    2. Create Test INSERTs to exercise the PK
    12

    View full-size slide

  12. We Can Generate Tests Automatically
    13

    View full-size slide

  13. We Can Generate Tests Automatically
    SchemaAnalyst
    14

    View full-size slide

  14. Generating Tests Automatically
    Generates 20 – 71
    Test Cases
    SchemaAnalyst
    Test data wrapped into INSERTs and
    into JUnit test cases 15

    View full-size slide

  15. The Solution
    TO USE TRADITIONAL TEST SUITE REDUCTION TECHNIQUES
    16

    View full-size slide

  16. Test Suite Reduction Background
    r1 r2 r3 r4 r5 r6
    t1 X X X
    t2 X X
    t3 X X
    t4 X X
    t5 X
    • We can use the following approaches:
    • Random Reduction – randomly select test case until all the requirements covered
    • Additional Greedy (or called greedy in TSR literature)
    • HGS (an approach by Harrold, Gupta, and Soffa)
    17

    View full-size slide

  17. Greedy Test Suite Reduction
    r1 r2 r3 r4 r5 r6
    t1 X X X
    t2 X X
    t3 X X
    t4 X X
    t5 X
    r1 r2 r3 r4 r5 r6
    18

    View full-size slide

  18. Greedy Test Suite Reduction
    r1 r2 r3 r4 r5 r6
    t1 X X X
    t2 X X
    t3 X X
    t4 X X
    t5 X
    r1 r2 r3 r4 r5 r6
    t1 X X X
    19

    View full-size slide

  19. Greedy Test Suite Reduction
    r1 r2 r3 r4 r5 r6
    t1 X X X
    t2 X X
    t3 X X
    t4 X X
    t5 X
    r1 r2 r3 r4 r5 r6
    t1 X X X
    t3 X X
    20

    View full-size slide

  20. Greedy Test Suite Reduction
    r1 r2 r3 r4 r5 r6
    t1 X X X
    t2 X X
    t3 X X
    t4 X X
    t5 X
    r1 r2 r3 r4 r5 r6
    t1 X X X
    t3 X X
    t2 X X
    21

    View full-size slide

  21. Greedy Test Suite Reduction
    r1 r2 r3 r4 r5 r6
    t1 X X X
    t2 X X
    t3 X X
    t4 X X
    t5 X
    r1 r2 r3 r4 r5 r6
    t1 X X X
    t3 X X
    t2 X X
    t4 X X
    22

    View full-size slide

  22. HGS Test Suite Reduction
    r1 r2 r3 r4 r5 r6
    t1 X X X
    t2 X X
    t3 X X
    t4 X X
    t5 X
    T1 T2 T3 T4 T5 T6
    T R tn Cardinality
    T1 r1 {t1, t2} 2
    T2 r2 {t1, t3} 2
    T3 r3 {t1, t4} 2
    T4 r4 {t2} 1
    T5 r5 {t3, t5} 2
    T6 r6 {t4} 1
    23

    View full-size slide

  23. HGS Test Suite Reduction
    r1 r2 r3 r4 r5 r6
    t2 X X
    t4 X X
    T R tn Cardinality
    T1 r1 {t1, t2} 2
    T2 r2 {t1, t3} 2
    T3 r3 {t1, t4} 2
    T4 r4 {t2} 1
    T5 r5 {t3, t5} 2
    T6 r6 {t4} 1
    24

    View full-size slide

  24. HGS Test Suite Reduction
    r1 r2 r3 r4 r5 r6
    t2 X X
    t4 X X
    T R tn Cardinality
    T1 r1 {t1, t2} 2
    T2 r2 {t1, t3} 2
    T3 r3 {t1, t4} 2
    T4 r4 {t2} 1
    T5 r5 {t3, t5} 2
    T6 r6 {t4} 1
    25

    View full-size slide

  25. HGS Test Suite Reduction
    r1 r2 r3 r4 r5 r6
    t2 X X
    t4 X X
    t3 X X
    T R tn Cardinality
    T1 r1 {t1, t2} 2
    T2 r2 {t1, t3} 2
    T3 r3 {t1, t4} 2
    T4 r4 {t2} 1
    T5 r5 {t3, t5} 2
    T6 r6 {t4} 1
    26

    View full-size slide

  26. What is missing?
    Can we merge similar test cases (decreasing the data restarts)?
    Can we decrease the number of INSERTs (decreasing database interactions)?
    Can we remove any extra redundancy?
    r1 r2 r3 r4 r5 r6
    t2 X X
    t4 X X
    t3 X X
    27

    View full-size slide

  27. Analyzing Automatically Generated Tests
    28

    View full-size slide

  28. Analyzing Automatically Generated Tests
    Equal
    29

    View full-size slide

  29. Analyzing Automatically Generated Tests
    Remove
    30

    View full-size slide

  30. Analyzing Automatically Generated Tests
    Unnecessary
    31

    View full-size slide

  31. Analyzing Automatically Generated Tests
    Remove
    32

    View full-size slide

  32. Analyzing Automatically Generated Tests
    Merge
    33

    View full-size slide

  33. Schema Test Integrity Constraints Combination
    for Efficient Reduction (STICCER)
    Full Test Suite 34

    View full-size slide

  34. Schema Test Integrity Constraints Combination
    for Efficient Reduction (STICCER)
    Reduce Tests
    Full Test Suite
    Reduced Test Suite
    35

    View full-size slide

  35. Merge Tests
    Greedy Reduction
    Full Test Suite
    Reduced Test Suite
    Merged Tests
    Schema Test Integrity Constraints Combination
    for Efficient Reduction (STICCER)
    36

    View full-size slide

  36. Research
    Questions
    RQ1: Reduction Effectiveness - How
    effective is STICCER at reducing the
    number of test cases and INSERTs?
    RQ2: Impact on Fault Finding Capability -
    How is the fault-finding capability of the
    test suites affected?
    RQ3: Impact on Test Suite and
    Mutation Analysis Runtime - How are the
    running times of the reduced test suites
    on mutation analysis affected?
    37

    View full-size slide

  37. Methodology
    34 schemas
    1 – 42 tables
    3 – 309 columns
    1 – 134 integrity constraints
    Two test data generators
    30 runs
    Four reduction techniques
    Mutation analysis
    38

    View full-size slide

  38. RQ1: Reduction Effectiveness Results
    • iTrust schema includes 42 tables, 309 columns, 134 Integrity
    Constraints
    • Highest merge count = 539 merges.
    39

    View full-size slide

  39. RQ1: Reduction Effectiveness Results
    • iTrust schema includes 42 tables, 309 columns, 134 Integrity
    Constraints
    • Highest merge count = 539 merges.
    Metric OTS STICCER Random Greedy HGS
    Test Cases 1517 85% (235) 44% (849) 49% (776) 50% (754)
    INSERTs 2204 57% (940) 45% (1212) 50% (1101) 52% (1064)
    40

    View full-size slide

  40. RQ1: Reduction Effectiveness Results
    • iTrust schema includes 42 tables, 309 columns, 134 Integrity
    Constraints
    • Highest merge count = 539 merges.
    • On Average:
    • No loss of coverage
    Metric OTS STICCER Random Greedy HGS
    Test Cases 1517 85% (235) 44% (849) 49% (776) 50% (754)
    INSERTs 2204 57% (940) 45% (1212) 50% (1101) 52% (1064)
    Metric STICCER Random Greedy HGS
    Test Cases 74% 42% 48% 50%
    INSERTs 59% 43% 49% 51%
    41

    View full-size slide

  41. RQ1: Reduction Effectiveness Results
    • iTrust schema includes 42 tables, 309 columns, 134 Integrity
    Constraints
    • Highest merge count = 539 merges.
    • On Average:
    • No loss of coverage
    Metric OTS STICCER Random Greedy HGS
    Test Cases 1517 85% (235) 44% (849) 49% (776) 50% (754)
    INSERTs 2204 57% (940) 45% (1212) 50% (1101) 52% (1064)
    Metric STICCER Random Greedy HGS
    Test Cases 74% 42% 48% 50%
    INSERTs 59% 43% 49% 51%
    STICCER is the most effective at
    reducing the number of test
    cases and the overall number of
    INSERT statements in a test
    suite
    42

    View full-size slide

  42. RQ2: Impact on Fault Finding
    Capability Results
    • AVM-D generated and reduced test case impacted the Fault-Finding Capability
    • 5 test suites were impacted by STICCER reduction
    • Maximum impact was only 3.2% compared to OTS
    43

    View full-size slide

  43. RQ2: Impact on Fault Finding
    Capability Results
    • AVM-D generated and reduced test case impacted the Fault-Finding Capability
    • 5 test suites were impacted by STICCER reduction
    • Maximum impact was only 3.2% compared to OTS
    Mutation scores of the test
    suites were preserved following
    reduction. While some test
    suites experienced a non-
    substantial drop in mutation
    score (3.2% maximum)
    44

    View full-size slide

  44. RQ3: Impact on Test Suite and Mutation
    Analysis Runtime Results
    • Mutation analysis runtime:
    • STICCER test suites were 5X faster than the original test suite
    • 2.5X faster than other traditional reduction techniques
    • Example: iTrust test suites and running mutation analysis
    Unit OTS STICCER Random Greedy HGS
    Minutes 38 7 (+2 reduction) 21 19 18.5
    45

    View full-size slide

  45. RQ3: Impact on Test Suite and Mutation
    Analysis Runtime Results
    • Mutation analysis runtime:
    • STICCER test suites were 5X faster than the original test suite
    • 2.5X faster than other traditional reduction techniques
    • Example: iTrust test suites and running mutation analysis
    Unit OTS STICCER Random Greedy HGS
    Minutes 38 7 (~2 for reduction) 21 19 18.5
    In general, STICCER reduced
    test suites ran faster compared
    to the OTS and those reduced
    by other techniques.
    46

    View full-size slide

  46. Conclusions
    and Future
    Work
    • STICCER = Reduce + Merge
    • Outperforms other reduction techniques and maintains coverage
    • A maximum of 3.2% loss of fault-finding capabilities (mutation)
    • Mutation analysis execution:
    • 5X faster than the original test suite
    • 2.5X faster than other traditional reduction techniques
    • Future Work:
    • Integrate STICCER within the test data generator
    • Enhance STICCER with multi-objective test data generators
    • Adapt STICCER into traditional programs that manipulate
    complex state in other formats
    github.com/schemaanalyst/schemaanalyst
    47

    View full-size slide