The False False Positives of Static Analysis (sattose2017)

The False False Positives of Static Analysis (sattose2017)

Static analysis tools may produce false positive results, which negatively impact the overall usability of these tools. However, even a correct static analysis report is sometimes classified as a false positive if a developer does not understand it or does not agree with it. Lately developers' classification of false positives is treated on a par with the actual static analysis performance which may distort the knowledge about the real state of static analysis.
In this paper we discuss various use cases where a false positive report is not false and the issue is caused by another aspects of static analysis. We provide an in-depth explanation of the issue for each use case followed by recommendations on how to solve it, and thus exemplify the importance of careful false positive classification.

9847702de1f7f19db8c3e158325baa8e?s=128

Yuriy Tymchuk

June 07, 2017
Tweet

Transcript

  1. The False False Positives of Static Analysis Yuriy Tymchuk SCG

    @ UNIBE
  2. @yuriy_tymchuk The False False Positives of Static Analysis SCG @

    UNIBE The False False Positives of Static Analysis
  3. You know nothing (about static analysis) little programmer!

  4. if(isValid == true) { //Do something. } Wanna Evolve This?

  5. if(isValid == true) { //Do something. } Wanna Evolve This?

  6. if( == true) { //Do something. } Wanna Evolve This?

  7. Wanna Evolve This? if(isValid == true) { //Do something. }

  8. Wanna Evolve This? if(isValid == true) { //Do something. }

    don’t compare with true…
  9. Wanna Evolve This? if(isValid == true) { //Do something. }

    don’t compare with true… detected by tatic analysis S s
  10. Static analysis?

  11. Static analysis? Yeah, but you know, there are… false positives!

  12. False Positives? if(isValid == true) { //Do something. } don’t

    compare with true… This is a “true positive”
  13. False Positives? False Positive False Negative

  14. Pregnancy vs Programming False Negative

  15. Pregnancy vs Programming True Positive are

  16. Pregnancy vs Programming True Positive are No, I’m not. Your

    test is false positive!
  17. “Developers will decide what a false positive is” Tricorder: Building

    a program analysis ecosystem,
 C. Sadowski, J. van Gogh, C. Jaspan, E. So ̈derberg, and C. Winter ICSE 2015 pst… the authors are from
  18. “Developers will decide what a false positive is” Tricorder: Building

    a program analysis ecosystem,
 C. Sadowski, J. van Gogh, C. Jaspan, E. So ̈derberg, and C. Winter ICSE 2015 pst… the authors are from this is Bad
  19. False positives? True positives? False false positives?

  20. self assert: (checkedValue = Date today)

  21. self assert: (checkedValue = Date today)

  22. self assert: checkedValue equals: Date today self assert: (checkedValue =

    Date today)
  23. self assert: checkedValue equals: Date today C’est cool! I didn’t

    know that you can do that…
  24. sender: aSenderContext | nArgs | self requestor: aSenderContext method methodClass.

    nArgs := aSenderContext method numArgs. methodArgs := aSenderContext method ffiArgumentNames. self receiver: aSenderContext receiver. self assert: (methodArgs size = nArgs). Assert Beyond Tests
  25. sender: aSenderContext | nArgs | self requestor: aSenderContext method methodClass.

    nArgs := aSenderContext method numArgs. methodArgs := aSenderContext method ffiArgumentNames. self receiver: aSenderContext receiver. self assert: (methodArgs size = nArgs). Assert Beyond Tests
  26. sender: aSenderContext | nArgs | self requestor: aSenderContext method methodClass.

    nArgs := aSenderContext method numArgs. methodArgs := aSenderContext method ffiArgumentNames. self receiver: aSenderContext receiver. self assert: (methodArgs size = nArgs). shiftLevelDown level := (level - 1) max: 0. self assert: level >=0 And this is a quality assistance tool that Yuriy developed… and here you see an example of a false positive report.
  27. And this is a quality assistance tool that Yuriy developed…

    and here you see an example of a false positive report. It’s a bug :(
  28. detections false positives p t + - Static Analysis Dilemma

  29. detections false positives p t + - Static Analysis Dilemma

    precision recall
  30. FALSE FALSE

  31. background ifNil: [ ˆ true ]. (background isColor and: [

    background isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. (border isColor and: [ border isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. ˆ false Anti-Quick Return Rule
  32. background ifNil: [ ˆ true ]. (background isColor and: [

    background isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. (border isColor and: [ border isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. ˆ false ˆ background isNil or: [ (background isColor and: [ background isTranslucentButNotTransparent ]) or: [ border isColor and: [ border isTranslucentButNotTransparent ] ] ] Anti-Quick Return Rule
  33. background ifNil: [ ˆ true ]. (background isColor and: [

    background isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. (border isColor and: [ border isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. ˆ false ˆ background isNil or: [ (background isColor and: [ background isTranslucentButNotTransparent ]) or: [ border isColor and: [ border isTranslucentButNotTransparent ] ] ] Anti-Quick Return Rule
  34. background ifNil: [ ˆ true ]. (background isColor and: [

    background isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. (border isColor and: [ border isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. ˆ false ˆ background isNil or: [ (background isColor and: [ background isTranslucentButNotTransparent ]) or: [ border isColor and: [ border isTranslucentButNotTransparent ] ] ] Anti-Quick Return Rule FP?
  35. background ifNil: [ ˆ true ]. (background isColor and: [

    background isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. (border isColor and: [ border isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. ˆ false ˆ background isNil or: [ (background isColor and: [ background isTranslucentButNotTransparent ]) or: [ border isColor and: [ border isTranslucentButNotTransparent ] ] ] Anti-Quick Return Rule fFP
  36. background ifNil: [ ˆ true ]. (background isColor and: [

    background isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. (border isColor and: [ border isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. ˆ false ˆ background isNil or: [ (background isColor and: [ background isTranslucentButNotTransparent ]) or: [ border isColor and: [ border isTranslucentButNotTransparent ] ] ] Anti-Quick Return Rule fFP
  37. Expected Blocks Rule (denominator = 0) ifTrue: [ Float infinity

    ] ifFalse: [ numerator / denominator ]
  38. Expected Blocks Rule (denominator = 0) ifTrue: Float infinity ifFalse:

    numerator / denominator
  39. Expected Blocks Rule (denominator = 0) ifTrue: [ Float infinity

    ] ifFalse: [ numerator / denominator ]
  40. Expected Blocks Rule size = 1 ifTrue: ’:’ ifFalse: ’s:’

  41. Expected Blocks Rule size = 1 ifTrue: ’:’ ifFalse: ’s:’

    FP
  42. Expected Blocks Rule size = 1 ifTrue: ’:’ ifFalse: ’s:’

    FP
  43. Expected Blocks Rule size = 1 ifTrue: ’:’ ifFalse: ’s:’

    FP?
  44. Expected Blocks Rule size = 1 ifTrue: ’:’ ifFalse: ’s:’

    fFP
  45. Trivial False False Positive Rules Uncommented Class Unused Variable

  46. Trivial False False Positive Rules Uncommented Class Unused Variable

  47. Trivial False False Positive Rules Uncommented Class Unused Variable FP?

  48. Trivial False False Positive Rules Uncommented Class Unused Variable fFP

  49. Do not Introduce
 False False Positives!

  50. Do not Introduce
 False False Positives! FP exaggerate FP ratio

  51. exaggerate FP ratio mask intentions
 Do not Introduce
 False False

    Positives! FP
  52. Do not Introduce
 False False Positives! exaggerate FP ratio mask

    intentions
 neglect tool de!ciencies FP
  53. exaggerate FP ratio mask intentions
 ignore user groups neglect tool

    de!ciencies Do not Introduce
 False False Positives! FP
  54. exaggerate FP ratio mask intentions
 ignore user groups neglect tool

    de!ciencies @yuriy_tymchuk Do not Introduce
 False False Positives! FP