Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Theory of Composite Faults

The Theory of Composite Faults

ICST 2017 Tokyo

Rahul Gopinath

March 14, 2017
Tweet

More Decks by Rahul Gopinath

Other Decks in Research

Transcript

  1. The Theory of
    Composite Faults
    Rahul Gopinath
    Carlos Jensen
    Alex Groce

    View full-size slide

  2. The Coupling Effect
    March 1, 2017
    2
    Complex faults are coupled to simple faults in such a way
    that a test data set that detects all simple faults in a
    program will detect a high percentage of complex faults
    (Jia et al. 2011)
    (Mutation Testing)

    View full-size slide

  3. The Coupling Effect
    March 1, 2017
    3
    Complex faults are coupled to simple faults in such a way
    that a test data set that detects all simple faults in a
    program will detect a high percentage of complex faults
    (Jia et al. 2011)
    (Mutation Testing)
    Coupling effect was verified for combined faults with up to 3
    simple faults. (Offutt 92)

    View full-size slide

  4. Theory of Fault Coupling
    March 1, 2017
    4
    • Model programs as composed of q independent functions
    with same domain and range n
    • A complex fault can be split into independent faults
    • All faults have equal probability of occurring
    (Wah 1995)
    The program composed of q
    independent functions
    A single function with
    same domain and range
    q
    b
    A fault
    a

    View full-size slide

  5. Theory of Fault Coupling
    March 1, 2017
    5
    • Fault masking probability = 1/n

    (where n is the size of the domain)
    • The coupling ratio = 1 - Fault Masking = (n - 1)/n
    • Fault masking becomes stronger as q approaches n

    (where q is the number of independent functions that
    compose the program)
    • i.e Coupling effect is unreliable in very large programs
    (Wah 1995)

    View full-size slide

  6. Fault Interactions
    March 1, 2017
    6
    def sum(lst):
    i, result=1, 0
    while i <= len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[0,1] = 1
    assert sum[1,0] = 1
    Original

    View full-size slide

  7. Fault Interactions
    March 1, 2017
    7
    def sum(lst):
    i, result=2, 0
    while i <= len(lst):
    result+=lst[i]
    i += 1
    return result
    assert sum[1,0] = 1 0
    def sum(lst):
    i, result=1, 0
    while i <= len(lst)-1:
    result+=lst[i]
    i += 1
    return result
    assert sum[0,1] = 1 0
    def sum(lst):
    i, result=1, 0
    while i <= len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[0,1] = 1
    assert sum[1,0] = 1
    Original

    View full-size slide

  8. Fault Interactions
    March 1, 2017
    8
    def sum(lst):
    i, result=2, 0
    while i <= len(lst):
    result+=lst[i]
    i += 1
    return result
    assert sum[1,0] = 1 0
    def sum(lst):
    i, result=1, 0
    while i <= len(lst)-1:
    result+=lst[i]
    i += 1
    return result
    assert sum[0,1] = 1 0
    def sum(lst):
    i, result=1, 0
    while i <= len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[1,0] = 1
    assert sum[0,1] = 1
    def sum(lst):
    i, result=2, 0
    while i <= len(lst)-1:
    result+=lst[i-1]
    i += 1
    return result
    assert sum[1,0] = 1 0
    assert sum[0,1] = 1 0
    Original

    View full-size slide

  9. What is Fault Masking?
    March 1, 2017
    9
    def sum(lst):
    i, result=1, 1
    while i <= len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[1,1] = 2 3
    def sum(lst):
    i, result=2, 0
    while i result+=lst[i]
    i += 1
    return result
    assert sum[1,1] = 2 1
    def sum(lst):
    i, result=1, 0
    while i <= len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[1,1] = 2
    Original

    View full-size slide

  10. What is Fault Masking?
    March 1, 2017
    10
    def sum(lst):
    i, result=1, 1
    while i <= len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[1,1] = 2 3
    def sum(lst):
    i, result=2, 0
    while i result+=lst[i]
    i += 1
    return result
    assert sum[1,1] = 2 1
    def sum(lst):
    i, result=1, 0
    while i <= len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[1,1] = 2
    def sum(lst):
    i, result=2, 1
    while i <=len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[1,1] = 2
    Original

    View full-size slide

  11. What is Fault Masking?
    March 1, 2017
    11
    def sum(lst):
    i, result=1, 1
    while i <= len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[1,1] = 2 3
    def sum(lst):
    i, result=2, 0
    while i result+=lst[i]
    i += 1
    return result
    assert sum[1,1] = 2 1
    def sum(lst):
    i, result=1, 0
    while i <= len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[1,1] = 2
    def sum(lst):
    i, result=2, 1
    while i <=len(lst):
    result+=lst[i-1]
    i += 1
    return result
    assert sum[1,1] = 2
    assert sum[0,1] = 1 2
    Original

    View full-size slide

  12. Equivalent programs
    March 1, 2017
    12
    def sum(lst):
    i, result=0, 0
    while i < len(lst):
    i += 1
    result+=lst[i-1]
    return result
    def sum(lst):
    i, result=0, 0
    while i < len(lst):
    result+=lst[i]
    i += 1
    return result
    Equivalents
    def sum(lst):
    i, result=0, 0
    while i < len(lst):
    result+=lst[i-1]
    i += 1
    return result
    Original

    View full-size slide

  13. Strong Interaction
    March 1, 2017
    13
    def sum(lst):
    i, result=0, 0
    while i < len(lst):
    i += 1
    result+=lst[i-1]
    return result
    def sum(lst):
    i, result=0, 0
    while i < len(lst):
    result+=lst[i]
    i += 1
    return result
    Equivalents
    def sum(lst):
    i, result=0, 0
    while i < len(lst):
    result+=lst[i-1]
    i += 1
    return result
    Original
    def sum(lst):
    i, result=0, 0
    while i < len(lst):
    i += 1
    result+=lst[i]
    return result
    assert sum[0,1] = 1 error
    Faulty
    These faults can't be separated to independent faults

    View full-size slide

  14. Theory of Fault Coupling: Drawbacks
    March 1, 2017
    14
    • Model programs as composed of q independent functions
    • Caveat: Ignores recursion and iteration
    • A complex fault can be split into independent faults
    • Caveat: Ignores strongly interacting faults
    • The functions have same domain and range n
    • Caveat: Real world is often more complex
    • All faults have equal probability of occurring
    • Caveat: Ignores the syntactic neighborhood

    View full-size slide

  15. The Theory of Composite Faults
    March 1, 2017
    15
    • Model based on fault propagation on function pairs
    • Only model composite faults (where faults can be split)
    • Functions have different domain and co-domain
    • All faults have equal probability of occurring
    • But also considers syntactic neighborhood
    A program with a composite fault
    Faults in independent functions
    with different domain and co-domain
    a b

    View full-size slide

  16. The Theory of Composite Faults
    March 1, 2017
    16
    function: h
    function: g
    a
    b
    c
    d
    a'
    b'
    c'
    d'
    a'
    b'
    c'
    d'
    a''
    b''
    c''
    d''
    Domain Co-domain Domain Co-domain

    View full-size slide

  17. The Theory of Composite Faults
    March 1, 2017
    17
    function: f = g ⨾ h
    a
    b
    c
    d
    a'
    b'
    c'
    d'
    a''
    b''
    c''
    d''
    Domain Co-domain

    View full-size slide

  18. The Theory of Composite Faults
    March 1, 2017
    18
    h'
    g'
    a
    b
    c
    d
    a'
    b'
    c'
    d'
    a'
    b'
    c'
    d'
    a''
    b''
    c''
    d''

    View full-size slide

  19. The Theory of Composite Faults
    March 1, 2017
    19
    function: f'' = g' ⨾ h'
    a
    b
    c
    d
    a'
    b'
    c'
    d'
    a''
    b''
    c''
    d''
    not masked
    masked
    Fault from input a in g' is masked by function h'

    View full-size slide

  20. The Theory of Composite Faults
    March 1, 2017
    20
    g : L x M h: M x N
    a
    b
    c
    d
    a'
    b'
    c'
    d'
    a''
    b''
    c''
    d''
    function: f = g ⨾ h
    nm unique alternatives for h

    View full-size slide

  21. The Theory of Composite Faults
    March 1, 2017
    21
    g' : L x M h': M x N
    function: f'' = g' ⨾ h'
    nm-1 alternatives of h' can correct g'
    a
    b
    c
    d
    a'
    b'
    c'
    d'
    a''
    b''
    c''
    d''

    View full-size slide

  22. The Theory of Composite Faults
    March 1, 2017
    22
    m is the domain and n is the co-domain
    nm unique alternatives for h
    nm-1 alternatives of h' can correct g'
    Composite coupling ratio = 1 - (nm-1/nm) = (n -1)/n
    (i.e fault masking probability = 1/n)

    View full-size slide

  23. Recursion and Iteration
    March 1, 2017
    23
    First iteration
    Masked values
    Non masked values
    Faulty input

    View full-size slide

  24. Recursion and Iteration
    March 1, 2017
    24
    First iteration
    Masked values
    Non masked values
    Faulty input
    Next iteration

    View full-size slide

  25. Premature loop exits
    March 1, 2017
    25
    If there are premature exits (e.g. crashes), there is
    a smaller probability of masking than 1/n
    Premature exits for x% inputs
    x + (ny - 1) ≥ (n-1)
    _______ _____
    ny n
    x%
    where y = x - 1

    View full-size slide

  26. The impact of Syntactical Neighborhood
    March 1, 2017
    26
    g h
    i0 j0 k0
    g’
    h
    i0
    j1 k1
    Both g and h are non-faulty
    g’ is a faulty version of g, but h is non-faulty

    View full-size slide

  27. Summary
    March 1, 2017
    27
    • Probability of fault masking is 1/n where n is the size of
    the co-domain for the program considered
    • The probability of fault masking does not increase with
    increase in program size or length of execution for the
    common patterns we studied.
    • The probability of fault masking does not increase even
    when the effects of syntactical neighborhood are
    considered.

    View full-size slide

  28. Empirical Evaluation
    March 1, 2017
    28

    View full-size slide

  29. Questions for Empirical Evaluation
    March 1, 2017
    29
    The composite coupling ratio:
    Given two faults, and and their combined fault, what
    percentage of test cases detecting them in isolation will
    detect the combined fault?
    The general coupling ratio:
    Given two faults, and their combined fault, what is the ratio
    between the number of test cases detecting them in
    isolation vs tests detecting the combined fault?

    View full-size slide

  30. Methodology
    March 1, 2017
    30
    • Subjects: 25 Apache commons-libraries
    • Size 84 KLOC - 1.4 KLOC
    • Collected commits
    • Generated reverse patches that can be independently
    applied from bug-fixes without compilation errors
    • Collected test cases failing each patch
    • Combined pairs of patches together
    • Collected test cases failing combined patches

    View full-size slide

  31. Results: Paired Patches for All Projects
    31
    • Composite Coupling Ratio = 0.99916
    • General Coupling Raio = 0.99931

    View full-size slide

  32. Larger Fault Combinations
    March 1, 2017
    32
    • Subject: Apache commons-math
    • Size 84 KLOC
    • Combined 2,4,8,16,32,64 patches
    • Collected test cases failing combined patches

    View full-size slide

  33. Results: Apache commons-math
    33
    For Apache commons-math:
    • Composite Coupling Ratio = 0.98956
    • General Coupling Raio = 0.9944

    View full-size slide

  34. Summary of Empirical Analysis
    March 1, 2017
    34
    The Composite fault hypothesis
    Tests detecting a fault in isolation will (with probability ~ 99%) continue
    to detect the fault even when in combination with other faults.

    View full-size slide

  35. Summary
    March 1, 2017
    35
    • The theory of composite faults
    • Probability of fault masking is 1/n where n is the size of
    the co-domain for the program considered
    • The probability of fault masking does not increase with
    increase in program size or length of execution for the
    common patterns we studied.
    • Empirical evaluation of the composite fault ratio
    • Between 98.8% and 99.1% of test cases detecting a fault
    in isolation can be expected to detect a composite fault
    including that fault.

    (95% confidence level, p < 0.0001)

    View full-size slide