$30 off During Our Annual Pro Sale. View Details »

The False False Positives of Static Analysis (sattose2017)

The False False Positives of Static Analysis (sattose2017)

Static analysis tools may produce false positive results, which negatively impact the overall usability of these tools. However, even a correct static analysis report is sometimes classified as a false positive if a developer does not understand it or does not agree with it. Lately developers' classification of false positives is treated on a par with the actual static analysis performance which may distort the knowledge about the real state of static analysis.
In this paper we discuss various use cases where a false positive report is not false and the issue is caused by another aspects of static analysis. We provide an in-depth explanation of the issue for each use case followed by recommendations on how to solve it, and thus exemplify the importance of careful false positive classification.

Yuriy Tymchuk

June 07, 2017
Tweet

More Decks by Yuriy Tymchuk

Other Decks in Research

Transcript

  1. The False False Positives
    of
    Static Analysis
    Yuriy Tymchuk
    SCG @ UNIBE

    View Slide

  2. @yuriy_tymchuk
    The False False Positives
    of
    Static Analysis
    SCG @ UNIBE
    The False False Positives
    of
    Static Analysis

    View Slide

  3. You know nothing
    (about static analysis)
    little programmer!

    View Slide

  4. if(isValid == true)
    {
    //Do something.
    }
    Wanna Evolve This?

    View Slide

  5. if(isValid == true)
    {
    //Do something.
    }
    Wanna Evolve This?

    View Slide

  6. if( == true)
    {
    //Do something.
    }
    Wanna Evolve This?

    View Slide

  7. Wanna Evolve This?
    if(isValid == true)
    {
    //Do something.
    }

    View Slide

  8. Wanna Evolve This?
    if(isValid == true)
    {
    //Do something.
    }
    don’t compare with true…

    View Slide

  9. Wanna Evolve This?
    if(isValid == true)
    {
    //Do something.
    }
    don’t compare with true…
    detected by tatic analysis
    S
    s

    View Slide

  10. Static analysis?

    View Slide

  11. Static analysis?
    Yeah, but you know,
    there are…
    false positives!

    View Slide

  12. False Positives?
    if(isValid == true)
    {
    //Do something.
    }
    don’t compare with true…
    This is a “true positive”

    View Slide

  13. False Positives?
    False Positive False Negative

    View Slide

  14. Pregnancy vs Programming
    False Negative

    View Slide

  15. Pregnancy vs Programming
    True Positive
    are

    View Slide

  16. Pregnancy vs Programming
    True Positive
    are
    No, I’m not. Your test
    is false positive!

    View Slide

  17. “Developers will decide what
    a false positive is”
    Tricorder: Building a program analysis ecosystem,

    C. Sadowski, J. van Gogh, C. Jaspan, E. So ̈derberg, and C. Winter
    ICSE 2015
    pst… the authors are from

    View Slide

  18. “Developers will decide what
    a false positive is”
    Tricorder: Building a program analysis ecosystem,

    C. Sadowski, J. van Gogh, C. Jaspan, E. So ̈derberg, and C. Winter
    ICSE 2015
    pst… the authors are from
    this is Bad

    View Slide

  19. False positives? True positives?
    False false positives?

    View Slide

  20. self assert: (checkedValue = Date today)

    View Slide

  21. self assert: (checkedValue = Date today)

    View Slide

  22. self assert: checkedValue equals: Date today
    self assert: (checkedValue = Date today)

    View Slide

  23. self assert: checkedValue equals: Date today
    C’est cool!
    I didn’t know that you
    can do that…

    View Slide

  24. sender: aSenderContext
    | nArgs |
    self requestor: aSenderContext method methodClass.
    nArgs := aSenderContext method numArgs.
    methodArgs := aSenderContext method ffiArgumentNames.
    self receiver: aSenderContext receiver.
    self assert: (methodArgs size = nArgs).
    Assert Beyond Tests

    View Slide

  25. sender: aSenderContext
    | nArgs |
    self requestor: aSenderContext method methodClass.
    nArgs := aSenderContext method numArgs.
    methodArgs := aSenderContext method ffiArgumentNames.
    self receiver: aSenderContext receiver.
    self assert: (methodArgs size = nArgs).
    Assert Beyond Tests

    View Slide

  26. sender: aSenderContext
    | nArgs |
    self requestor: aSenderContext method methodClass.
    nArgs := aSenderContext method numArgs.
    methodArgs := aSenderContext method ffiArgumentNames.
    self receiver: aSenderContext receiver.
    self assert: (methodArgs size = nArgs).
    shiftLevelDown
    level := (level - 1) max: 0.
    self assert: level >=0
    And this is a quality assistance tool that
    Yuriy developed… and here you see an
    example of a false positive report.

    View Slide

  27. And this is a quality assistance tool that
    Yuriy developed… and here you see an
    example of a false positive report.
    It’s a bug :(

    View Slide

  28. detections
    false
    positives
    p
    t
    +
    -
    Static Analysis Dilemma

    View Slide

  29. detections
    false
    positives
    p
    t
    +
    -
    Static Analysis Dilemma
    precision recall

    View Slide

  30. FALSE FALSE

    View Slide

  31. background ifNil: [ ˆ true ].
    (background isColor and: [ background isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    (border isColor and: [ border isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    ˆ false
    Anti-Quick Return Rule

    View Slide

  32. background ifNil: [ ˆ true ].
    (background isColor and: [ background isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    (border isColor and: [ border isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    ˆ false
    ˆ background isNil or: [
    (background isColor and: [
    background isTranslucentButNotTransparent ]) or: [
    border isColor and: [
    border isTranslucentButNotTransparent ] ] ]
    Anti-Quick Return Rule

    View Slide

  33. background ifNil: [ ˆ true ].
    (background isColor and: [ background isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    (border isColor and: [ border isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    ˆ false
    ˆ background isNil or: [
    (background isColor and: [
    background isTranslucentButNotTransparent ]) or: [
    border isColor and: [
    border isTranslucentButNotTransparent ] ] ]
    Anti-Quick Return Rule

    View Slide

  34. background ifNil: [ ˆ true ].
    (background isColor and: [ background isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    (border isColor and: [ border isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    ˆ false
    ˆ background isNil or: [
    (background isColor and: [
    background isTranslucentButNotTransparent ]) or: [
    border isColor and: [
    border isTranslucentButNotTransparent ] ] ]
    Anti-Quick Return Rule
    FP?

    View Slide

  35. background ifNil: [ ˆ true ].
    (background isColor and: [ background isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    (border isColor and: [ border isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    ˆ false
    ˆ background isNil or: [
    (background isColor and: [
    background isTranslucentButNotTransparent ]) or: [
    border isColor and: [
    border isTranslucentButNotTransparent ] ] ]
    Anti-Quick Return Rule
    fFP

    View Slide

  36. background ifNil: [ ˆ true ].
    (background isColor and: [ background isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    (border isColor and: [ border isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    ˆ false
    ˆ background isNil or: [
    (background isColor and: [
    background isTranslucentButNotTransparent ]) or: [
    border isColor and: [
    border isTranslucentButNotTransparent ] ] ]
    Anti-Quick Return Rule
    fFP

    View Slide

  37. Expected Blocks Rule
    (denominator = 0)
    ifTrue: [ Float infinity ]
    ifFalse: [ numerator / denominator ]

    View Slide

  38. Expected Blocks Rule
    (denominator = 0)
    ifTrue: Float infinity
    ifFalse: numerator / denominator

    View Slide

  39. Expected Blocks Rule
    (denominator = 0)
    ifTrue: [ Float infinity ]
    ifFalse: [ numerator / denominator ]

    View Slide

  40. Expected Blocks Rule
    size = 1 ifTrue: ’:’ ifFalse: ’s:’

    View Slide

  41. Expected Blocks Rule
    size = 1 ifTrue: ’:’ ifFalse: ’s:’
    FP

    View Slide

  42. Expected Blocks Rule
    size = 1 ifTrue: ’:’ ifFalse: ’s:’
    FP

    View Slide

  43. Expected Blocks Rule
    size = 1 ifTrue: ’:’ ifFalse: ’s:’
    FP?

    View Slide

  44. Expected Blocks Rule
    size = 1 ifTrue: ’:’ ifFalse: ’s:’
    fFP

    View Slide

  45. Trivial False False Positive Rules
    Uncommented Class
    Unused Variable

    View Slide

  46. Trivial False False Positive Rules
    Uncommented Class
    Unused Variable

    View Slide

  47. Trivial False False Positive Rules
    Uncommented Class
    Unused Variable
    FP?

    View Slide

  48. Trivial False False Positive Rules
    Uncommented Class
    Unused Variable
    fFP

    View Slide

  49. Do not Introduce

    False False Positives!

    View Slide

  50. Do not Introduce

    False False Positives!
    FP
    exaggerate
    FP ratio

    View Slide

  51. exaggerate
    FP ratio
    mask intentions

    Do not Introduce

    False False Positives!
    FP

    View Slide

  52. Do not Introduce

    False False Positives!
    exaggerate
    FP ratio
    mask intentions
 neglect tool
    de!ciencies
    FP

    View Slide

  53. exaggerate
    FP ratio
    mask intentions

    ignore
    user groups
    neglect tool
    de!ciencies
    Do not Introduce

    False False Positives!
    FP

    View Slide

  54. exaggerate
    FP ratio
    mask intentions

    ignore
    user groups
    neglect tool
    de!ciencies
    @yuriy_tymchuk
    Do not Introduce

    False False Positives!
    FP

    View Slide