The Theory of Composite Faults

ICST 2017 Tokyo

Transcript

1. The Theory of
   Composite Faults
   Rahul Gopinath
   Carlos Jensen
   Alex Groce
   

   View Slide

2. The Coupling Effect
   March 1, 2017
   2
   Complex faults are coupled to simple faults in such a way
   that a test data set that detects all simple faults in a
   program will detect a high percentage of complex faults
   (Jia et al. 2011)
   (Mutation Testing)
   

   View Slide

3. The Coupling Effect
   March 1, 2017
   3
   Complex faults are coupled to simple faults in such a way
   that a test data set that detects all simple faults in a
   program will detect a high percentage of complex faults
   (Jia et al. 2011)
   (Mutation Testing)
   Coupling effect was verified for combined faults with up to 3
   simple faults. (Offutt 92)
   

   View Slide

4. Theory of Fault Coupling
   March 1, 2017
   4
   • Model programs as composed of q independent functions
   with same domain and range n
   • A complex fault can be split into independent faults
   • All faults have equal probability of occurring
   (Wah 1995)
   The program composed of q
   independent functions
   A single function with
   same domain and range
   q
   b
   A fault
   a
   

   View Slide

5. Theory of Fault Coupling
   March 1, 2017
   5
   • Fault masking probability = 1/n

   (where n is the size of the domain)
   • The coupling ratio = 1 - Fault Masking = (n - 1)/n
   • Fault masking becomes stronger as q approaches n

   (where q is the number of independent functions that
   compose the program)
   • i.e Coupling effect is unreliable in very large programs
   (Wah 1995)
   

   View Slide

6. Fault Interactions
   March 1, 2017
   6
   def sum(lst):
   i, result=1, 0
   while i <= len(lst):
   result+=lst[i-1]
   i += 1
   return result
   assert sum[0,1] = 1
   assert sum[1,0] = 1
   Original
   

   View Slide

7. Fault Interactions
   March 1, 2017
   7
   def sum(lst):
   i, result=2, 0
   while i <= len(lst):
   result+=lst[i]
   i += 1
   return result
   assert sum[1,0] = 1 0
   def sum(lst):
   i, result=1, 0
   while i <= len(lst)-1:
   result+=lst[i]
   i += 1
   return result
   assert sum[0,1] = 1 0
   def sum(lst):
   i, result=1, 0
   while i <= len(lst):
   result+=lst[i-1]
   i += 1
   return result
   assert sum[0,1] = 1
   assert sum[1,0] = 1
   Original
   

   View Slide

8. Fault Interactions
   March 1, 2017
   8
   def sum(lst):
   i, result=2, 0
   while i <= len(lst):
   result+=lst[i]
   i += 1
   return result
   assert sum[1,0] = 1 0
   def sum(lst):
   i, result=1, 0
   while i <= len(lst)-1:
   result+=lst[i]
   i += 1
   return result
   assert sum[0,1] = 1 0
   def sum(lst):
   i, result=1, 0
   while i <= len(lst):
   result+=lst[i-1]
   i += 1
   return result
   assert sum[1,0] = 1
   assert sum[0,1] = 1
   def sum(lst):
   i, result=2, 0
   while i <= len(lst)-1:
   result+=lst[i-1]
   i += 1
   return result
   assert sum[1,0] = 1 0
   assert sum[0,1] = 1 0
   Original
   

   View Slide

9. What is Fault Masking?
   March 1, 2017
   9
   def sum(lst):
   i, result=1, 1
   while i <= len(lst):
   result+=lst[i-1]
   i += 1
   return result
   assert sum[1,1] = 2 3
   def sum(lst):
   i, result=2, 0
   while i result+=lst[i]
   i += 1
   return result
   assert sum[1,1] = 2 1
   def sum(lst):
   i, result=1, 0
   while i <= len(lst):
   result+=lst[i-1]
   i += 1
   return result
   assert sum[1,1] = 2
   Original
   

   View Slide

10. What is Fault Masking?
    March 1, 2017
    10
    def sum(lst):
    i, result=1, 1
    while i <= len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[1,1] = 2 3
    def sum(lst):
    i, result=2, 0
    while i result+=lst[i]
    i += 1
    return result
    assert sum[1,1] = 2 1
    def sum(lst):
    i, result=1, 0
    while i <= len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[1,1] = 2
    def sum(lst):
    i, result=2, 1
    while i <=len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[1,1] = 2
    Original
    

    View Slide

11. What is Fault Masking?
    March 1, 2017
    11
    def sum(lst):
    i, result=1, 1
    while i <= len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[1,1] = 2 3
    def sum(lst):
    i, result=2, 0
    while i result+=lst[i]
    i += 1
    return result
    assert sum[1,1] = 2 1
    def sum(lst):
    i, result=1, 0
    while i <= len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[1,1] = 2
    def sum(lst):
    i, result=2, 1
    while i <=len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[1,1] = 2
    assert sum[0,1] = 1 2
    Original
    

    View Slide

12. Equivalent programs
    March 1, 2017
    12
    def sum(lst):
    i, result=0, 0
    while i < len(lst):
    i += 1
    result+=lst[i-1]
    return result
    def sum(lst):
    i, result=0, 0
    while i < len(lst):
    result+=lst[i]
    i += 1
    return result
    Equivalents
    def sum(lst):
    i, result=0, 0
    while i < len(lst):
    result+=lst[i-1]
    i += 1
    return result
    Original
    

    View Slide

13. Strong Interaction
    March 1, 2017
    13
    def sum(lst):
    i, result=0, 0
    while i < len(lst):
    i += 1
    result+=lst[i-1]
    return result
    def sum(lst):
    i, result=0, 0
    while i < len(lst):
    result+=lst[i]
    i += 1
    return result
    Equivalents
    def sum(lst):
    i, result=0, 0
    while i < len(lst):
    result+=lst[i-1]
    i += 1
    return result
    Original
    def sum(lst):
    i, result=0, 0
    while i < len(lst):
    i += 1
    result+=lst[i]
    return result
    assert sum[0,1] = 1 error
    Faulty
    These faults can't be separated to independent faults
    

    View Slide

14. Theory of Fault Coupling: Drawbacks
    March 1, 2017
    14
    • Model programs as composed of q independent functions
    • Caveat: Ignores recursion and iteration
    • A complex fault can be split into independent faults
    • Caveat: Ignores strongly interacting faults
    • The functions have same domain and range n
    • Caveat: Real world is often more complex
    • All faults have equal probability of occurring
    • Caveat: Ignores the syntactic neighborhood
    

    View Slide

15. The Theory of Composite Faults
    March 1, 2017
    15
    • Model based on fault propagation on function pairs
    • Only model composite faults (where faults can be split)
    • Functions have different domain and co-domain
    • All faults have equal probability of occurring
    • But also considers syntactic neighborhood
    A program with a composite fault
    Faults in independent functions
    with different domain and co-domain
    a b
    

    View Slide

16. The Theory of Composite Faults
    March 1, 2017
    16
    function: h
    function: g
    a
    b
    c
    d
    a'
    b'
    c'
    d'
    a'
    b'
    c'
    d'
    a''
    b''
    c''
    d''
    Domain Co-domain Domain Co-domain
    

    View Slide

17. The Theory of Composite Faults
    March 1, 2017
    17
    function: f = g ⨾ h
    a
    b
    c
    d
    a'
    b'
    c'
    d'
    a''
    b''
    c''
    d''
    Domain Co-domain
    

    View Slide

18. The Theory of Composite Faults
    March 1, 2017
    18
    h'
    g'
    a
    b
    c
    d
    a'
    b'
    c'
    d'
    a'
    b'
    c'
    d'
    a''
    b''
    c''
    d''
    

    View Slide

19. The Theory of Composite Faults
    March 1, 2017
    19
    function: f'' = g' ⨾ h'
    a
    b
    c
    d
    a'
    b'
    c'
    d'
    a''
    b''
    c''
    d''
    not masked
    masked
    Fault from input a in g' is masked by function h'
    

    View Slide

20. The Theory of Composite Faults
    March 1, 2017
    20
    g : L x M h: M x N
    a
    b
    c
    d
    a'
    b'
    c'
    d'
    a''
    b''
    c''
    d''
    function: f = g ⨾ h
    nm unique alternatives for h
    

    View Slide

21. The Theory of Composite Faults
    March 1, 2017
    21
    g' : L x M h': M x N
    function: f'' = g' ⨾ h'
    nm-1 alternatives of h' can correct g'
    a
    b
    c
    d
    a'
    b'
    c'
    d'
    a''
    b''
    c''
    d''
    

    View Slide

22. The Theory of Composite Faults
    March 1, 2017
    22
    m is the domain and n is the co-domain
    nm unique alternatives for h
    nm-1 alternatives of h' can correct g'
    Composite coupling ratio = 1 - (nm-1/nm) = (n -1)/n
    (i.e fault masking probability = 1/n)
    

    View Slide

23. Recursion and Iteration
    March 1, 2017
    23
    First iteration
    Masked values
    Non masked values
    Faulty input
    

    View Slide

24. Recursion and Iteration
    March 1, 2017
    24
    First iteration
    Masked values
    Non masked values
    Faulty input
    Next iteration
    

    View Slide

25. Premature loop exits
    March 1, 2017
    25
    If there are premature exits (e.g. crashes), there is
    a smaller probability of masking than 1/n
    Premature exits for x% inputs
    x + (ny - 1) ≥ (n-1)
    _______ _____
    ny n
    x%
    where y = x - 1
    

    View Slide

26. The impact of Syntactical Neighborhood
    March 1, 2017
    26
    g h
    i0 j0 k0
    g’
    h
    i0
    j1 k1
    Both g and h are non-faulty
    g’ is a faulty version of g, but h is non-faulty
    

    View Slide

27. Summary
    March 1, 2017
    27
    • Probability of fault masking is 1/n where n is the size of
    the co-domain for the program considered
    • The probability of fault masking does not increase with
    increase in program size or length of execution for the
    common patterns we studied.
    • The probability of fault masking does not increase even
    when the effects of syntactical neighborhood are
    considered.
    

    View Slide

28. Empirical Evaluation
    March 1, 2017
    28
    

    View Slide

29. Questions for Empirical Evaluation
    March 1, 2017
    29
    The composite coupling ratio:
    Given two faults, and and their combined fault, what
    percentage of test cases detecting them in isolation will
    detect the combined fault?
    The general coupling ratio:
    Given two faults, and their combined fault, what is the ratio
    between the number of test cases detecting them in
    isolation vs tests detecting the combined fault?
    

    View Slide

30. Methodology
    March 1, 2017
    30
    • Subjects: 25 Apache commons-libraries
    • Size 84 KLOC - 1.4 KLOC
    • Collected commits
    • Generated reverse patches that can be independently
    applied from bug-fixes without compilation errors
    • Collected test cases failing each patch
    • Combined pairs of patches together
    • Collected test cases failing combined patches
    

    View Slide

31. Results: Paired Patches for All Projects
    31
    • Composite Coupling Ratio = 0.99916
    • General Coupling Raio = 0.99931
    

    View Slide

32. Larger Fault Combinations
    March 1, 2017
    32
    • Subject: Apache commons-math
    • Size 84 KLOC
    • Combined 2,4,8,16,32,64 patches
    • Collected test cases failing combined patches
    

    View Slide

33. Results: Apache commons-math
    33
    For Apache commons-math:
    • Composite Coupling Ratio = 0.98956
    • General Coupling Raio = 0.9944
    

    View Slide

34. Summary of Empirical Analysis
    March 1, 2017
    34
    The Composite fault hypothesis
    Tests detecting a fault in isolation will (with probability ~ 99%) continue
    to detect the fault even when in combination with other faults.
    

    View Slide

35. Summary
    March 1, 2017
    35
    • The theory of composite faults
    • Probability of fault masking is 1/n where n is the size of
    the co-domain for the program considered
    • The probability of fault masking does not increase with
    increase in program size or length of execution for the
    common patterns we studied.
    • Empirical evaluation of the composite fault ratio
    • Between 98.8% and 99.1% of test cases detecting a fault
    in isolation can be expected to detect a composite fault
    including that fault.

    (95% confidence level, p < 0.0001)
    

    View Slide