# The Theory of Composite Faults

ICST 2017 Tokyo

March 14, 2017

## Transcript

Groce
2. ### The Coupling Effect March 1, 2017 2 Complex faults are

coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect a high percentage of complex faults (Jia et al. 2011) (Mutation Testing)
3. ### The Coupling Effect March 1, 2017 3 Complex faults are

coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect a high percentage of complex faults (Jia et al. 2011) (Mutation Testing) Coupling effect was verified for combined faults with up to 3 simple faults. (Offutt 92)
4. ### Theory of Fault Coupling March 1, 2017 4 • Model

programs as composed of q independent functions with same domain and range n • A complex fault can be split into independent faults • All faults have equal probability of occurring (Wah 1995) The program composed of q independent functions A single function with same domain and range q b A fault a
5. ### Theory of Fault Coupling March 1, 2017 5 • Fault

masking probability = 1/n  (where n is the size of the domain) • The coupling ratio = 1 - Fault Masking = (n - 1)/n • Fault masking becomes stronger as q approaches n  (where q is the number of independent functions that compose the program) • i.e Coupling effect is unreliable in very large programs (Wah 1995)
6. ### Fault Interactions March 1, 2017 6 def sum(lst): i, result=1,

0 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[0,1] = 1 assert sum[1,0] = 1 Original
7. ### Fault Interactions March 1, 2017 7 def sum(lst): i, result=2,

0 while i <= len(lst): result+=lst[i] i += 1 return result assert sum[1,0] = 1 0 def sum(lst): i, result=1, 0 while i <= len(lst)-1: result+=lst[i] i += 1 return result assert sum[0,1] = 1 0 def sum(lst): i, result=1, 0 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[0,1] = 1 assert sum[1,0] = 1 Original
8. ### Fault Interactions March 1, 2017 8 def sum(lst): i, result=2,

0 while i <= len(lst): result+=lst[i] i += 1 return result assert sum[1,0] = 1 0 def sum(lst): i, result=1, 0 while i <= len(lst)-1: result+=lst[i] i += 1 return result assert sum[0,1] = 1 0 def sum(lst): i, result=1, 0 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[1,0] = 1 assert sum[0,1] = 1 def sum(lst): i, result=2, 0 while i <= len(lst)-1: result+=lst[i-1] i += 1 return result assert sum[1,0] = 1 0 assert sum[0,1] = 1 0 Original
9. ### What is Fault Masking? March 1, 2017 9 def sum(lst):

i, result=1, 1 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[1,1] = 2 3 def sum(lst): i, result=2, 0 while i <len(lst): result+=lst[i] i += 1 return result assert sum[1,1] = 2 1 def sum(lst): i, result=1, 0 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[1,1] = 2 Original
10. ### What is Fault Masking? March 1, 2017 10 def sum(lst):

i, result=1, 1 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[1,1] = 2 3 def sum(lst): i, result=2, 0 while i <len(lst): result+=lst[i] i += 1 return result assert sum[1,1] = 2 1 def sum(lst): i, result=1, 0 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[1,1] = 2 def sum(lst): i, result=2, 1 while i <=len(lst): result+=lst[i-1] i += 1 return result assert sum[1,1] = 2 Original
11. ### What is Fault Masking? March 1, 2017 11 def sum(lst):

i, result=1, 1 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[1,1] = 2 3 def sum(lst): i, result=2, 0 while i <len(lst): result+=lst[i] i += 1 return result assert sum[1,1] = 2 1 def sum(lst): i, result=1, 0 while i <= len(lst): result+=lst[i-1] i += 1 return result assert sum[1,1] = 2 def sum(lst): i, result=2, 1 while i <=len(lst): result+=lst[i-1] i += 1 return result assert sum[1,1] = 2 assert sum[0,1] = 1 2 Original
12. ### Equivalent programs March 1, 2017 12 def sum(lst): i, result=0,

0 while i < len(lst): i += 1 result+=lst[i-1] return result def sum(lst): i, result=0, 0 while i < len(lst): result+=lst[i] i += 1 return result Equivalents def sum(lst): i, result=0, 0 while i < len(lst): result+=lst[i-1] i += 1 return result Original
13. ### Strong Interaction March 1, 2017 13 def sum(lst): i, result=0,

0 while i < len(lst): i += 1 result+=lst[i-1] return result def sum(lst): i, result=0, 0 while i < len(lst): result+=lst[i] i += 1 return result Equivalents def sum(lst): i, result=0, 0 while i < len(lst): result+=lst[i-1] i += 1 return result Original def sum(lst): i, result=0, 0 while i < len(lst): i += 1 result+=lst[i] return result assert sum[0,1] = 1 error Faulty These faults can't be separated to independent faults
14. ### Theory of Fault Coupling: Drawbacks March 1, 2017 14 •

Model programs as composed of q independent functions • Caveat: Ignores recursion and iteration • A complex fault can be split into independent faults • Caveat: Ignores strongly interacting faults • The functions have same domain and range n • Caveat: Real world is often more complex • All faults have equal probability of occurring • Caveat: Ignores the syntactic neighborhood
15. ### The Theory of Composite Faults March 1, 2017 15 •

Model based on fault propagation on function pairs • Only model composite faults (where faults can be split) • Functions have different domain and co-domain • All faults have equal probability of occurring • But also considers syntactic neighborhood A program with a composite fault Faults in independent functions with different domain and co-domain a b
16. ### The Theory of Composite Faults March 1, 2017 16 function:

h function: g a b c d a' b' c' d' a' b' c' d' a'' b'' c'' d'' Domain Co-domain Domain Co-domain
17. ### The Theory of Composite Faults March 1, 2017 17 function:

f = g ⨾ h a b c d a' b' c' d' a'' b'' c'' d'' Domain Co-domain
18. ### The Theory of Composite Faults March 1, 2017 18 h'

g' a b c d a' b' c' d' a' b' c' d' a'' b'' c'' d''
19. ### The Theory of Composite Faults March 1, 2017 19 function:

f'' = g' ⨾ h' a b c d a' b' c' d' a'' b'' c'' d'' not masked masked Fault from input a in g' is masked by function h'
20. ### The Theory of Composite Faults March 1, 2017 20 g

: L x M h: M x N a b c d a' b' c' d' a'' b'' c'' d'' function: f = g ⨾ h nm unique alternatives for h
21. ### The Theory of Composite Faults March 1, 2017 21 g'

: L x M h': M x N function: f'' = g' ⨾ h' nm-1 alternatives of h' can correct g' a b c d a' b' c' d' a'' b'' c'' d''
22. ### The Theory of Composite Faults March 1, 2017 22 m

is the domain and n is the co-domain nm unique alternatives for h nm-1 alternatives of h' can correct g' Composite coupling ratio = 1 - (nm-1/nm) = (n -1)/n (i.e fault masking probability = 1/n)
23. ### Recursion and Iteration March 1, 2017 23 First iteration Masked

values Non masked values Faulty input
24. ### Recursion and Iteration March 1, 2017 24 First iteration Masked

values Non masked values Faulty input Next iteration
25. ### Premature loop exits March 1, 2017 25 If there are

premature exits (e.g. crashes), there is a smaller probability of masking than 1/n Premature exits for x% inputs x + (ny - 1) ≥ (n-1) _______ _____ ny n x% where y = x - 1
26. ### The impact of Syntactical Neighborhood March 1, 2017 26 g

h i0 j0 k0 g’ h i0 j1 k1 Both g and h are non-faulty g’ is a faulty version of g, but h is non-faulty
27. ### Summary March 1, 2017 27 • Probability of fault masking

is 1/n where n is the size of the co-domain for the program considered • The probability of fault masking does not increase with increase in program size or length of execution for the common patterns we studied. • The probability of fault masking does not increase even when the effects of syntactical neighborhood are considered.

29. ### Questions for Empirical Evaluation March 1, 2017 29 The composite

coupling ratio: Given two faults, and and their combined fault, what percentage of test cases detecting them in isolation will detect the combined fault? The general coupling ratio: Given two faults, and their combined fault, what is the ratio between the number of test cases detecting them in isolation vs tests detecting the combined fault?
30. ### Methodology March 1, 2017 30 • Subjects: 25 Apache commons-libraries

• Size 84 KLOC - 1.4 KLOC • Collected commits • Generated reverse patches that can be independently applied from bug-fixes without compilation errors • Collected test cases failing each patch • Combined pairs of patches together • Collected test cases failing combined patches
31. ### Results: Paired Patches for All Projects 31 • Composite Coupling

Ratio = 0.99916 • General Coupling Raio = 0.99931
32. ### Larger Fault Combinations March 1, 2017 32 • Subject: Apache

commons-math • Size 84 KLOC • Combined 2,4,8,16,32,64 patches • Collected test cases failing combined patches
33. ### Results: Apache commons-math 33 For Apache commons-math: • Composite Coupling

Ratio = 0.98956 • General Coupling Raio = 0.9944
34. ### Summary of Empirical Analysis March 1, 2017 34 The Composite

fault hypothesis Tests detecting a fault in isolation will (with probability ~ 99%) continue to detect the fault even when in combination with other faults.
35. ### Summary March 1, 2017 35 • The theory of composite

faults • Probability of fault masking is 1/n where n is the size of the co-domain for the program considered • The probability of fault masking does not increase with increase in program size or length of execution for the common patterns we studied. • Empirical evaluation of the composite fault ratio • Between 98.8% and 99.1% of test cases detecting a fault in isolation can be expected to detect a composite fault including that fault.  (95% confidence level, p < 0.0001)