The False False Positives of Static Analysis (sattose2017)

Static analysis tools may produce false positive results, which negatively impact the overall usability of these tools. However, even a correct static analysis report is sometimes classified as a false positive if a developer does not understand it or does not agree with it. Lately developers' classification of false positives is treated on a par with the actual static analysis performance which may distort the knowledge about the real state of static analysis.
In this paper we discuss various use cases where a false positive report is not false and the issue is caused by another aspects of static analysis. We provide an in-depth explanation of the issue for each use case followed by recommendations on how to solve it, and thus exemplify the importance of careful false positive classification.

Transcript

1. The False False Positives
   of
   Static Analysis
   Yuriy Tymchuk
   SCG @ UNIBE
   

   View Slide

2. @yuriy_tymchuk
   The False False Positives
   of
   Static Analysis
   SCG @ UNIBE
   The False False Positives
   of
   Static Analysis
   

   View Slide

3. You know nothing
   (about static analysis)
   little programmer!
   

   View Slide

4. if(isValid == true)
   {
   //Do something.
   }
   Wanna Evolve This?
   

   View Slide

5. if(isValid == true)
   {
   //Do something.
   }
   Wanna Evolve This?
   

   View Slide

6. if( == true)
   {
   //Do something.
   }
   Wanna Evolve This?
   

   View Slide

7. Wanna Evolve This?
   if(isValid == true)
   {
   //Do something.
   }
   

   View Slide

8. Wanna Evolve This?
   if(isValid == true)
   {
   //Do something.
   }
   don’t compare with true…
   

   View Slide

9. Wanna Evolve This?
   if(isValid == true)
   {
   //Do something.
   }
   don’t compare with true…
   detected by tatic analysis
   S
   s
   

   View Slide

10. Static analysis?
    

    View Slide

11. Static analysis?
    Yeah, but you know,
    there are…
    false positives!
    

    View Slide

12. False Positives?
    if(isValid == true)
    {
    //Do something.
    }
    don’t compare with true…
    This is a “true positive”
    

    View Slide

13. False Positives?
    False Positive False Negative
    

    View Slide

14. Pregnancy vs Programming
    False Negative
    

    View Slide

15. Pregnancy vs Programming
    True Positive
    are
    

    View Slide

16. Pregnancy vs Programming
    True Positive
    are
    No, I’m not. Your test
    is false positive!
    

    View Slide

17. “Developers will decide what
    a false positive is”
    Tricorder: Building a program analysis ecosystem,

    C. Sadowski, J. van Gogh, C. Jaspan, E. So ̈derberg, and C. Winter
    ICSE 2015
    pst… the authors are from
    

    View Slide

18. “Developers will decide what
    a false positive is”
    Tricorder: Building a program analysis ecosystem,

    C. Sadowski, J. van Gogh, C. Jaspan, E. So ̈derberg, and C. Winter
    ICSE 2015
    pst… the authors are from
    this is Bad
    

    View Slide

19. False positives? True positives?
    False false positives?
    

    View Slide

20. self assert: (checkedValue = Date today)
    

    View Slide

21. self assert: (checkedValue = Date today)
    

    View Slide

22. self assert: checkedValue equals: Date today
    self assert: (checkedValue = Date today)
    

    View Slide

23. self assert: checkedValue equals: Date today
    C’est cool!
    I didn’t know that you
    can do that…
    

    View Slide

24. sender: aSenderContext
    | nArgs |
    self requestor: aSenderContext method methodClass.
    nArgs := aSenderContext method numArgs.
    methodArgs := aSenderContext method ffiArgumentNames.
    self receiver: aSenderContext receiver.
    self assert: (methodArgs size = nArgs).
    Assert Beyond Tests
    

    View Slide

25. sender: aSenderContext
    | nArgs |
    self requestor: aSenderContext method methodClass.
    nArgs := aSenderContext method numArgs.
    methodArgs := aSenderContext method ffiArgumentNames.
    self receiver: aSenderContext receiver.
    self assert: (methodArgs size = nArgs).
    Assert Beyond Tests
    

    View Slide

26. sender: aSenderContext
    | nArgs |
    self requestor: aSenderContext method methodClass.
    nArgs := aSenderContext method numArgs.
    methodArgs := aSenderContext method ffiArgumentNames.
    self receiver: aSenderContext receiver.
    self assert: (methodArgs size = nArgs).
    shiftLevelDown
    level := (level - 1) max: 0.
    self assert: level >=0
    And this is a quality assistance tool that
    Yuriy developed… and here you see an
    example of a false positive report.
    

    View Slide

27. And this is a quality assistance tool that
    Yuriy developed… and here you see an
    example of a false positive report.
    It’s a bug :(
    

    View Slide

28. detections
    false
    positives
    p
    t
    +
    -
    Static Analysis Dilemma
    

    View Slide

29. detections
    false
    positives
    p
    t
    +
    -
    Static Analysis Dilemma
    precision recall
    

    View Slide

30. FALSE FALSE
    

    View Slide

31. background ifNil: [ ˆ true ].
    (background isColor and: [ background isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    (border isColor and: [ border isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    ˆ false
    Anti-Quick Return Rule
    

    View Slide

32. background ifNil: [ ˆ true ].
    (background isColor and: [ background isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    (border isColor and: [ border isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    ˆ false
    ˆ background isNil or: [
    (background isColor and: [
    background isTranslucentButNotTransparent ]) or: [
    border isColor and: [
    border isTranslucentButNotTransparent ] ] ]
    Anti-Quick Return Rule
    

    View Slide

33. background ifNil: [ ˆ true ].
    (background isColor and: [ background isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    (border isColor and: [ border isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    ˆ false
    ˆ background isNil or: [
    (background isColor and: [
    background isTranslucentButNotTransparent ]) or: [
    border isColor and: [
    border isTranslucentButNotTransparent ] ] ]
    Anti-Quick Return Rule
    

    View Slide

34. background ifNil: [ ˆ true ].
    (background isColor and: [ background isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    (border isColor and: [ border isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    ˆ false
    ˆ background isNil or: [
    (background isColor and: [
    background isTranslucentButNotTransparent ]) or: [
    border isColor and: [
    border isTranslucentButNotTransparent ] ] ]
    Anti-Quick Return Rule
    FP?
    

    View Slide

35. background ifNil: [ ˆ true ].
    (background isColor and: [ background isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    (border isColor and: [ border isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    ˆ false
    ˆ background isNil or: [
    (background isColor and: [
    background isTranslucentButNotTransparent ]) or: [
    border isColor and: [
    border isTranslucentButNotTransparent ] ] ]
    Anti-Quick Return Rule
    fFP
    

    View Slide

36. background ifNil: [ ˆ true ].
    (background isColor and: [ background isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    (border isColor and: [ border isTranslucentButNotTransparent ])
    ifTrue: [ ˆ true ].
    ˆ false
    ˆ background isNil or: [
    (background isColor and: [
    background isTranslucentButNotTransparent ]) or: [
    border isColor and: [
    border isTranslucentButNotTransparent ] ] ]
    Anti-Quick Return Rule
    fFP
    

    View Slide

37. Expected Blocks Rule
    (denominator = 0)
    ifTrue: [ Float infinity ]
    ifFalse: [ numerator / denominator ]
    

    View Slide

38. Expected Blocks Rule
    (denominator = 0)
    ifTrue: Float infinity
    ifFalse: numerator / denominator
    

    View Slide

39. Expected Blocks Rule
    (denominator = 0)
    ifTrue: [ Float infinity ]
    ifFalse: [ numerator / denominator ]
    

    View Slide

40. Expected Blocks Rule
    size = 1 ifTrue: ’:’ ifFalse: ’s:’
    

    View Slide

41. Expected Blocks Rule
    size = 1 ifTrue: ’:’ ifFalse: ’s:’
    FP
    

    View Slide

42. Expected Blocks Rule
    size = 1 ifTrue: ’:’ ifFalse: ’s:’
    FP
    

    View Slide

43. Expected Blocks Rule
    size = 1 ifTrue: ’:’ ifFalse: ’s:’
    FP?
    

    View Slide

44. Expected Blocks Rule
    size = 1 ifTrue: ’:’ ifFalse: ’s:’
    fFP
    

    View Slide

45. Trivial False False Positive Rules
    Uncommented Class
    Unused Variable
    

    View Slide

46. Trivial False False Positive Rules
    Uncommented Class
    Unused Variable
    

    View Slide

47. Trivial False False Positive Rules
    Uncommented Class
    Unused Variable
    FP?
    

    View Slide

48. Trivial False False Positive Rules
    Uncommented Class
    Unused Variable
    fFP
    

    View Slide

49. Do not Introduce

    False False Positives!
    

    View Slide

50. Do not Introduce

    False False Positives!
    FP
    exaggerate
    FP ratio
    

    View Slide

51. exaggerate
    FP ratio
    mask intentions

    Do not Introduce

    False False Positives!
    FP
    

    View Slide

52. Do not Introduce

    False False Positives!
    exaggerate
    FP ratio
    mask intentions
 neglect tool
    de!ciencies
    FP
    

    View Slide

53. exaggerate
    FP ratio
    mask intentions

    ignore
    user groups
    neglect tool
    de!ciencies
    Do not Introduce

    False False Positives!
    FP
    

    View Slide

54. exaggerate
    FP ratio
    mask intentions

    ignore
    user groups
    neglect tool
    de!ciencies
    @yuriy_tymchuk
    Do not Introduce

    False False Positives!
    FP
    

    View Slide