120

# The Theory of Composite Faults

ICST 2017 Tokyo March 14, 2017

## Transcript

1. The Theory of
Composite Faults
Rahul Gopinath
Carlos Jensen
Alex Groce

2. The Coupling Effect
March 1, 2017
2
Complex faults are coupled to simple faults in such a way
that a test data set that detects all simple faults in a
program will detect a high percentage of complex faults
(Jia et al. 2011)
(Mutation Testing)

3. The Coupling Effect
March 1, 2017
3
Complex faults are coupled to simple faults in such a way
that a test data set that detects all simple faults in a
program will detect a high percentage of complex faults
(Jia et al. 2011)
(Mutation Testing)
Coupling effect was verified for combined faults with up to 3
simple faults. (Offutt 92)

4. Theory of Fault Coupling
March 1, 2017
4
• Model programs as composed of q independent functions
with same domain and range n
• A complex fault can be split into independent faults
• All faults have equal probability of occurring
(Wah 1995)
The program composed of q
independent functions
A single function with
same domain and range
q
b
A fault
a

5. Theory of Fault Coupling
March 1, 2017
5
• Fault masking probability = 1/n
(where n is the size of the domain)
• The coupling ratio = 1 - Fault Masking = (n - 1)/n
• Fault masking becomes stronger as q approaches n
(where q is the number of independent functions that
compose the program)
• i.e Coupling effect is unreliable in very large programs
(Wah 1995)

6. Fault Interactions
March 1, 2017
6
def sum(lst):
i, result=1, 0
while i <= len(lst):
result+=lst[i-1]
i += 1
return result
assert sum[0,1] = 1
assert sum[1,0] = 1
Original

7. Fault Interactions
March 1, 2017
7
def sum(lst):
i, result=2, 0
while i <= len(lst):
result+=lst[i]
i += 1
return result
assert sum[1,0] = 1 0
def sum(lst):
i, result=1, 0
while i <= len(lst)-1:
result+=lst[i]
i += 1
return result
assert sum[0,1] = 1 0
def sum(lst):
i, result=1, 0
while i <= len(lst):
result+=lst[i-1]
i += 1
return result
assert sum[0,1] = 1
assert sum[1,0] = 1
Original

8. Fault Interactions
March 1, 2017
8
def sum(lst):
i, result=2, 0
while i <= len(lst):
result+=lst[i]
i += 1
return result
assert sum[1,0] = 1 0
def sum(lst):
i, result=1, 0
while i <= len(lst)-1:
result+=lst[i]
i += 1
return result
assert sum[0,1] = 1 0
def sum(lst):
i, result=1, 0
while i <= len(lst):
result+=lst[i-1]
i += 1
return result
assert sum[1,0] = 1
assert sum[0,1] = 1
def sum(lst):
i, result=2, 0
while i <= len(lst)-1:
result+=lst[i-1]
i += 1
return result
assert sum[1,0] = 1 0
assert sum[0,1] = 1 0
Original

March 1, 2017
9
def sum(lst):
i, result=1, 1
while i <= len(lst):
result+=lst[i-1]
i += 1
return result
assert sum[1,1] = 2 3
def sum(lst):
i, result=2, 0
while i result+=lst[i]
i += 1
return result
assert sum[1,1] = 2 1
def sum(lst):
i, result=1, 0
while i <= len(lst):
result+=lst[i-1]
i += 1
return result
assert sum[1,1] = 2
Original

March 1, 2017
10
def sum(lst):
i, result=1, 1
while i <= len(lst):
result+=lst[i-1]
i += 1
return result
assert sum[1,1] = 2 3
def sum(lst):
i, result=2, 0
while i result+=lst[i]
i += 1
return result
assert sum[1,1] = 2 1
def sum(lst):
i, result=1, 0
while i <= len(lst):
result+=lst[i-1]
i += 1
return result
assert sum[1,1] = 2
def sum(lst):
i, result=2, 1
while i <=len(lst):
result+=lst[i-1]
i += 1
return result
assert sum[1,1] = 2
Original

March 1, 2017
11
def sum(lst):
i, result=1, 1
while i <= len(lst):
result+=lst[i-1]
i += 1
return result
assert sum[1,1] = 2 3
def sum(lst):
i, result=2, 0
while i result+=lst[i]
i += 1
return result
assert sum[1,1] = 2 1
def sum(lst):
i, result=1, 0
while i <= len(lst):
result+=lst[i-1]
i += 1
return result
assert sum[1,1] = 2
def sum(lst):
i, result=2, 1
while i <=len(lst):
result+=lst[i-1]
i += 1
return result
assert sum[1,1] = 2
assert sum[0,1] = 1 2
Original

12. Equivalent programs
March 1, 2017
12
def sum(lst):
i, result=0, 0
while i < len(lst):
i += 1
result+=lst[i-1]
return result
def sum(lst):
i, result=0, 0
while i < len(lst):
result+=lst[i]
i += 1
return result
Equivalents
def sum(lst):
i, result=0, 0
while i < len(lst):
result+=lst[i-1]
i += 1
return result
Original

13. Strong Interaction
March 1, 2017
13
def sum(lst):
i, result=0, 0
while i < len(lst):
i += 1
result+=lst[i-1]
return result
def sum(lst):
i, result=0, 0
while i < len(lst):
result+=lst[i]
i += 1
return result
Equivalents
def sum(lst):
i, result=0, 0
while i < len(lst):
result+=lst[i-1]
i += 1
return result
Original
def sum(lst):
i, result=0, 0
while i < len(lst):
i += 1
result+=lst[i]
return result
assert sum[0,1] = 1 error
Faulty
These faults can't be separated to independent faults

14. Theory of Fault Coupling: Drawbacks
March 1, 2017
14
• Model programs as composed of q independent functions
• Caveat: Ignores recursion and iteration
• A complex fault can be split into independent faults
• Caveat: Ignores strongly interacting faults
• The functions have same domain and range n
• Caveat: Real world is often more complex
• All faults have equal probability of occurring
• Caveat: Ignores the syntactic neighborhood

15. The Theory of Composite Faults
March 1, 2017
15
• Model based on fault propagation on function pairs
• Only model composite faults (where faults can be split)
• Functions have different domain and co-domain
• All faults have equal probability of occurring
• But also considers syntactic neighborhood
A program with a composite fault
Faults in independent functions
with different domain and co-domain
a b

16. The Theory of Composite Faults
March 1, 2017
16
function: h
function: g
a
b
c
d
a'
b'
c'
d'
a'
b'
c'
d'
a''
b''
c''
d''
Domain Co-domain Domain Co-domain

17. The Theory of Composite Faults
March 1, 2017
17
function: f = g ⨾ h
a
b
c
d
a'
b'
c'
d'
a''
b''
c''
d''
Domain Co-domain

18. The Theory of Composite Faults
March 1, 2017
18
h'
g'
a
b
c
d
a'
b'
c'
d'
a'
b'
c'
d'
a''
b''
c''
d''

19. The Theory of Composite Faults
March 1, 2017
19
function: f'' = g' ⨾ h'
a
b
c
d
a'
b'
c'
d'
a''
b''
c''
d''
Fault from input a in g' is masked by function h'

20. The Theory of Composite Faults
March 1, 2017
20
g : L x M h: M x N
a
b
c
d
a'
b'
c'
d'
a''
b''
c''
d''
function: f = g ⨾ h
nm unique alternatives for h

21. The Theory of Composite Faults
March 1, 2017
21
g' : L x M h': M x N
function: f'' = g' ⨾ h'
nm-1 alternatives of h' can correct g'
a
b
c
d
a'
b'
c'
d'
a''
b''
c''
d''

22. The Theory of Composite Faults
March 1, 2017
22
m is the domain and n is the co-domain
nm unique alternatives for h
nm-1 alternatives of h' can correct g'
Composite coupling ratio = 1 - (nm-1/nm) = (n -1)/n
(i.e fault masking probability = 1/n)

23. Recursion and Iteration
March 1, 2017
23
First iteration
Faulty input

24. Recursion and Iteration
March 1, 2017
24
First iteration
Faulty input
Next iteration

25. Premature loop exits
March 1, 2017
25
If there are premature exits (e.g. crashes), there is
a smaller probability of masking than 1/n
Premature exits for x% inputs
x + (ny - 1) ≥ (n-1)
_______ _____
ny n
x%
where y = x - 1

26. The impact of Syntactical Neighborhood
March 1, 2017
26
g h
i0 j0 k0
g’
h
i0
j1 k1
Both g and h are non-faulty
g’ is a faulty version of g, but h is non-faulty

27. Summary
March 1, 2017
27
• Probability of fault masking is 1/n where n is the size of
the co-domain for the program considered
• The probability of fault masking does not increase with
increase in program size or length of execution for the
common patterns we studied.
• The probability of fault masking does not increase even
when the effects of syntactical neighborhood are
considered.

28. Empirical Evaluation
March 1, 2017
28

29. Questions for Empirical Evaluation
March 1, 2017
29
The composite coupling ratio:
Given two faults, and and their combined fault, what
percentage of test cases detecting them in isolation will
detect the combined fault?
The general coupling ratio:
Given two faults, and their combined fault, what is the ratio
between the number of test cases detecting them in
isolation vs tests detecting the combined fault?

30. Methodology
March 1, 2017
30
• Subjects: 25 Apache commons-libraries
• Size 84 KLOC - 1.4 KLOC
• Collected commits
• Generated reverse patches that can be independently
applied from bug-fixes without compilation errors
• Collected test cases failing each patch
• Combined pairs of patches together
• Collected test cases failing combined patches

31. Results: Paired Patches for All Projects
31
• Composite Coupling Ratio = 0.99916
• General Coupling Raio = 0.99931

32. Larger Fault Combinations
March 1, 2017
32
• Subject: Apache commons-math
• Size 84 KLOC
• Combined 2,4,8,16,32,64 patches
• Collected test cases failing combined patches

33. Results: Apache commons-math
33
For Apache commons-math:
• Composite Coupling Ratio = 0.98956
• General Coupling Raio = 0.9944

34. Summary of Empirical Analysis
March 1, 2017
34
The Composite fault hypothesis
Tests detecting a fault in isolation will (with probability ~ 99%) continue
to detect the fault even when in combination with other faults.

35. Summary
March 1, 2017
35
• The theory of composite faults
• Probability of fault masking is 1/n where n is the size of
the co-domain for the program considered
• The probability of fault masking does not increase with
increase in program size or length of execution for the
common patterns we studied.
• Empirical evaluation of the composite fault ratio
• Between 98.8% and 99.1% of test cases detecting a fault
in isolation can be expected to detect a composite fault
including that fault.
(95% confidence level, p < 0.0001)