The Theory of Composite Faults

ICST 2017 Tokyo

Transcript

1. The Theory of Composite Faults Rahul Gopinath Carlos Jensen Alex Groce
2. The Coupling Effect March 1, 2017 2 Complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect a high percentage of complex faults (Jia et al. 2011) (Mutation Testing)
3. The Coupling Effect March 1, 2017 3 Complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect a high percentage of complex faults (Jia et al. 2011) (Mutation Testing) Coupling effect was verified for combined faults with up to 3 simple faults. (Offutt 92)
4. Theory of Fault Coupling March 1, 2017 4 • Model programs as composed of q independent functions with same domain and range n • A complex fault can be split into independent faults • All faults have equal probability of occurring (Wah 1995) The program composed of q independent functions A single function with same domain and range q b A fault a
5. Theory of Fault Coupling March 1, 2017 5 • Fault masking probability = 1/n
 (where n is the size of the domain) • The coupling ratio = 1 - Fault Masking = (n - 1)/n • Fault masking becomes stronger as q approaches n
 (where q is the number of independent functions that compose the program) • i.e Coupling effect is unreliable in very large programs (Wah 1995)
6. Fault Interactions March 1, 2017 6 def sum(lst): i, result=1, 0 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[0,1] = 1 assert sum[1,0] = 1 Original
7. Fault Interactions March 1, 2017 7 def sum(lst): i, result=2, 0 while i <= len(lst): result+=lst[i] i += 1 return result assert sum[1,0] = 1 0 def sum(lst): i, result=1, 0 while i <= len(lst)-1: result+=lst[i] i += 1 return result assert sum[0,1] = 1 0 def sum(lst): i, result=1, 0 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[0,1] = 1 assert sum[1,0] = 1 Original
8. Fault Interactions March 1, 2017 8 def sum(lst): i, result=2, 0 while i <= len(lst): result+=lst[i] i += 1 return result assert sum[1,0] = 1 0 def sum(lst): i, result=1, 0 while i <= len(lst)-1: result+=lst[i] i += 1 return result assert sum[0,1] = 1 0 def sum(lst): i, result=1, 0 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[1,0] = 1 assert sum[0,1] = 1 def sum(lst): i, result=2, 0 while i <= len(lst)-1: result+=lst[i-1] i += 1 return result assert sum[1,0] = 1 0 assert sum[0,1] = 1 0 Original
9. What is Fault Masking? March 1, 2017 9 def sum(lst): i, result=1, 1 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[1,1] = 2 3 def sum(lst): i, result=2, 0 while i <len(lst): result+=lst[i] i += 1 return result assert sum[1,1] = 2 1 def sum(lst): i, result=1, 0 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[1,1] = 2 Original
10. What is Fault Masking? March 1, 2017 10 def sum(lst): i, result=1, 1 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[1,1] = 2 3 def sum(lst): i, result=2, 0 while i <len(lst): result+=lst[i] i += 1 return result assert sum[1,1] = 2 1 def sum(lst): i, result=1, 0 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[1,1] = 2 def sum(lst): i, result=2, 1 while i <=len(lst): result+=lst[i-1] i += 1 return result assert sum[1,1] = 2 Original
11. What is Fault Masking? March 1, 2017 11 def sum(lst): i, result=1, 1 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[1,1] = 2 3 def sum(lst): i, result=2, 0 while i <len(lst): result+=lst[i] i += 1 return result assert sum[1,1] = 2 1 def sum(lst): i, result=1, 0 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[1,1] = 2 def sum(lst): i, result=2, 1 while i <=len(lst): result+=lst[i-1] i += 1 return result assert sum[1,1] = 2 assert sum[0,1] = 1 2 Original
12. Equivalent programs March 1, 2017 12 def sum(lst): i, result=0, 0 while i < len(lst): i += 1 result+=lst[i-1] return result def sum(lst): i, result=0, 0 while i < len(lst): result+=lst[i] i += 1 return result Equivalents def sum(lst): i, result=0, 0 while i < len(lst): result+=lst[i-1] i += 1 return result Original
13. Strong Interaction March 1, 2017 13 def sum(lst): i, result=0, 0 while i < len(lst): i += 1 result+=lst[i-1] return result def sum(lst): i, result=0, 0 while i < len(lst): result+=lst[i] i += 1 return result Equivalents def sum(lst): i, result=0, 0 while i < len(lst): result+=lst[i-1] i += 1 return result Original def sum(lst): i, result=0, 0 while i < len(lst): i += 1 result+=lst[i] return result assert sum[0,1] = 1 error Faulty These faults can't be separated to independent faults
14. Theory of Fault Coupling: Drawbacks March 1, 2017 14 • Model programs as composed of q independent functions • Caveat: Ignores recursion and iteration • A complex fault can be split into independent faults • Caveat: Ignores strongly interacting faults • The functions have same domain and range n • Caveat: Real world is often more complex • All faults have equal probability of occurring • Caveat: Ignores the syntactic neighborhood
15. The Theory of Composite Faults March 1, 2017 15 • Model based on fault propagation on function pairs • Only model composite faults (where faults can be split) • Functions have different domain and co-domain • All faults have equal probability of occurring • But also considers syntactic neighborhood A program with a composite fault Faults in independent functions with different domain and co-domain a b
16. The Theory of Composite Faults March 1, 2017 16 function: h function: g a b c d a' b' c' d' a' b' c' d' a'' b'' c'' d'' Domain Co-domain Domain Co-domain
17. The Theory of Composite Faults March 1, 2017 17 function: f = g ⨾ h a b c d a' b' c' d' a'' b'' c'' d'' Domain Co-domain
18. The Theory of Composite Faults March 1, 2017 18 h' g' a b c d a' b' c' d' a' b' c' d' a'' b'' c'' d''
19. The Theory of Composite Faults March 1, 2017 19 function: f'' = g' ⨾ h' a b c d a' b' c' d' a'' b'' c'' d'' not masked masked Fault from input a in g' is masked by function h'
20. The Theory of Composite Faults March 1, 2017 20 g : L x M h: M x N a b c d a' b' c' d' a'' b'' c'' d'' function: f = g ⨾ h nm unique alternatives for h
21. The Theory of Composite Faults March 1, 2017 21 g' : L x M h': M x N function: f'' = g' ⨾ h' nm-1 alternatives of h' can correct g' a b c d a' b' c' d' a'' b'' c'' d''
22. The Theory of Composite Faults March 1, 2017 22 m is the domain and n is the co-domain nm unique alternatives for h nm-1 alternatives of h' can correct g' Composite coupling ratio = 1 - (nm-1/nm) = (n -1)/n (i.e fault masking probability = 1/n)
23. Recursion and Iteration March 1, 2017 23 First iteration Masked values Non masked values Faulty input
24. Recursion and Iteration March 1, 2017 24 First iteration Masked values Non masked values Faulty input Next iteration
25. Premature loop exits March 1, 2017 25 If there are premature exits (e.g. crashes), there is a smaller probability of masking than 1/n Premature exits for x% inputs x + (ny - 1) ≥ (n-1) _______ _____ ny n x% where y = x - 1
26. The impact of Syntactical Neighborhood March 1, 2017 26 g h i0 j0 k0 g’ h i0 j1 k1 Both g and h are non-faulty g’ is a faulty version of g, but h is non-faulty
27. Summary March 1, 2017 27 • Probability of fault masking is 1/n where n is the size of the co-domain for the program considered • The probability of fault masking does not increase with increase in program size or length of execution for the common patterns we studied. • The probability of fault masking does not increase even when the effects of syntactical neighborhood are considered.
28. Empirical Evaluation March 1, 2017 28
29. Questions for Empirical Evaluation March 1, 2017 29 The composite coupling ratio: Given two faults, and and their combined fault, what percentage of test cases detecting them in isolation will detect the combined fault? The general coupling ratio: Given two faults, and their combined fault, what is the ratio between the number of test cases detecting them in isolation vs tests detecting the combined fault?
30. Methodology March 1, 2017 30 • Subjects: 25 Apache commons-libraries • Size 84 KLOC - 1.4 KLOC • Collected commits • Generated reverse patches that can be independently applied from bug-fixes without compilation errors • Collected test cases failing each patch • Combined pairs of patches together • Collected test cases failing combined patches
31. Results: Paired Patches for All Projects 31 • Composite Coupling Ratio = 0.99916 • General Coupling Raio = 0.99931
32. Larger Fault Combinations March 1, 2017 32 • Subject: Apache commons-math • Size 84 KLOC • Combined 2,4,8,16,32,64 patches • Collected test cases failing combined patches
33. Results: Apache commons-math 33 For Apache commons-math: • Composite Coupling Ratio = 0.98956 • General Coupling Raio = 0.9944
34. Summary of Empirical Analysis March 1, 2017 34 The Composite fault hypothesis Tests detecting a fault in isolation will (with probability ~ 99%) continue to detect the fault even when in combination with other faults.
35. Summary March 1, 2017 35 • The theory of composite faults • Probability of fault masking is 1/n where n is the size of the co-domain for the program considered • The probability of fault masking does not increase with increase in program size or length of execution for the common patterns we studied. • Empirical evaluation of the composite fault ratio • Between 98.8% and 99.1% of test cases detecting a fault in isolation can be expected to detect a composite fault including that fault.
 (95% confidence level, p < 0.0001)