a test suite TS and a coverage target t, the fraction of test cases in TS that cover t is called hitting fraction. • If TS has no test to cover t, HF(TS, t) = 0 • If all tests in TS cover t, HF(TS, t) = 1 Hitting Fraction of statements of YAFFS2 using yaffs2tester
parts of code • Regression Testing Initial Test Suite Focused Test Suites Generating new test suites to increase hitting fraction (HF) of less-frequently covered targets.
Test Features F1,F2,F3,F4.... (e.g. name of APIs) Save test case ( output) Pick a random subset of test features Alex Groce, Chaoqiang Zhang, Eric Eide, Yang Chen, John Regehr: Swarm testing. ISSTA 2012: 78-88 OK NOT OK
test cases • Example- faults, coverage entities, mutant • Relationship between feature f and target t • Trigger • Suppressor • Irrelevant • Wilson Score of Confidence [ISSRE’13] Alex Groce, Chaoqiang Zhang, Mohammad Amin Alipour, Eric Eide, Yang Chen, John Regehr: Help, help, I'm being suppressed! The significance of suppressors in software testing. ISSRE 2013: 390-399 F1 F2 F3 F4 ….. T1 S T I I I T2 I I S I I T3 S S I T I …… I I S I T
target can be a suppressor for another one, or vice versa. Merging Strategies: • Round-robin (No merging) • Subsumption Merge a configuration to a stricter one • Aggressive Merging Tries to merge as many as non-conflicting configuration • Optimum Generates minimum set of configurations NP-Complete [RW] [RW] Edward L. Robertson and Catharine M. Wyss: Optimal Tuple Merge in NP-Complete, Technical Report TR599, Indiana University
Flash file system GCC 4.4.7 860K Csmith C and C++ compiler Spidermonkey 118K Jsfunfuzz Javascript engine for Mozilla SUT # Features Seed time(min) Run time(min) YAFFS 43 15 5 GCC 25 60 10 Spidermonkey 171 30 10 Subjects Experimental parameters
swarm testing, it is possible to produce focused random tests • The method is able to increase the frequency with which tests cover targeted code by a factor often more than 2x • This approach is readily applicable to existing, industrial-strength random testing tools for critical systems software.
- api calls, grammar terminal & production rules, c semantics Configuration of a test generator Pure Random testing: configuration = set of all features Swarm testing: configuration = each feature chosen with a coin toss Increases feature count per test and increases interaction