The False False Positives of Static Analysis (sattose2017)

Static analysis tools may produce false positive results, which negatively impact the overall usability of these tools. However, even a correct static analysis report is sometimes classified as a false positive if a developer does not understand it or does not agree with it. Lately developers' classification of false positives is treated on a par with the actual static analysis performance which may distort the knowledge about the real state of static analysis.
In this paper we discuss various use cases where a false positive report is not false and the issue is caused by another aspects of static analysis. We provide an in-depth explanation of the issue for each use case followed by recommendations on how to solve it, and thus exemplify the importance of careful false positive classification.

Transcript

1. The False False Positives of Static Analysis Yuriy Tymchuk SCG @ UNIBE
2. @yuriy_tymchuk The False False Positives of Static Analysis SCG @ UNIBE The False False Positives of Static Analysis
3. You know nothing (about static analysis) little programmer!
4. if(isValid == true) { //Do something. } Wanna Evolve This?
5. if(isValid == true) { //Do something. } Wanna Evolve This?
6. if( == true) { //Do something. } Wanna Evolve This?
7. Wanna Evolve This? if(isValid == true) { //Do something. }
8. Wanna Evolve This? if(isValid == true) { //Do something. } don’t compare with true…
9. Wanna Evolve This? if(isValid == true) { //Do something. } don’t compare with true… detected by tatic analysis S s
10. Static analysis?
11. Static analysis? Yeah, but you know, there are… false positives!
12. False Positives? if(isValid == true) { //Do something. } don’t compare with true… This is a “true positive”
13. False Positives? False Positive False Negative
14. Pregnancy vs Programming False Negative
15. Pregnancy vs Programming True Positive are
16. Pregnancy vs Programming True Positive are No, I’m not. Your test is false positive!
17. “Developers will decide what a false positive is” Tricorder: Building a program analysis ecosystem,
 C. Sadowski, J. van Gogh, C. Jaspan, E. So ̈derberg, and C. Winter ICSE 2015 pst… the authors are from
18. “Developers will decide what a false positive is” Tricorder: Building a program analysis ecosystem,
 C. Sadowski, J. van Gogh, C. Jaspan, E. So ̈derberg, and C. Winter ICSE 2015 pst… the authors are from this is Bad
19. False positives? True positives? False false positives?
20. self assert: (checkedValue = Date today)
21. self assert: (checkedValue = Date today)
22. self assert: checkedValue equals: Date today self assert: (checkedValue = Date today)
23. self assert: checkedValue equals: Date today C’est cool! I didn’t know that you can do that…
24. sender: aSenderContext | nArgs | self requestor: aSenderContext method methodClass. nArgs := aSenderContext method numArgs. methodArgs := aSenderContext method ffiArgumentNames. self receiver: aSenderContext receiver. self assert: (methodArgs size = nArgs). Assert Beyond Tests
25. sender: aSenderContext | nArgs | self requestor: aSenderContext method methodClass. nArgs := aSenderContext method numArgs. methodArgs := aSenderContext method ffiArgumentNames. self receiver: aSenderContext receiver. self assert: (methodArgs size = nArgs). Assert Beyond Tests
26. sender: aSenderContext | nArgs | self requestor: aSenderContext method methodClass. nArgs := aSenderContext method numArgs. methodArgs := aSenderContext method ffiArgumentNames. self receiver: aSenderContext receiver. self assert: (methodArgs size = nArgs). shiftLevelDown level := (level - 1) max: 0. self assert: level >=0 And this is a quality assistance tool that Yuriy developed… and here you see an example of a false positive report.
27. And this is a quality assistance tool that Yuriy developed… and here you see an example of a false positive report. It’s a bug :(
28. detections false positives p t + - Static Analysis Dilemma
29. detections false positives p t + - Static Analysis Dilemma precision recall
30. FALSE FALSE
31. background ifNil: [ ˆ true ]. (background isColor and: [ background isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. (border isColor and: [ border isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. ˆ false Anti-Quick Return Rule
32. background ifNil: [ ˆ true ]. (background isColor and: [ background isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. (border isColor and: [ border isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. ˆ false ˆ background isNil or: [ (background isColor and: [ background isTranslucentButNotTransparent ]) or: [ border isColor and: [ border isTranslucentButNotTransparent ] ] ] Anti-Quick Return Rule
33. background ifNil: [ ˆ true ]. (background isColor and: [ background isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. (border isColor and: [ border isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. ˆ false ˆ background isNil or: [ (background isColor and: [ background isTranslucentButNotTransparent ]) or: [ border isColor and: [ border isTranslucentButNotTransparent ] ] ] Anti-Quick Return Rule
34. background ifNil: [ ˆ true ]. (background isColor and: [ background isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. (border isColor and: [ border isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. ˆ false ˆ background isNil or: [ (background isColor and: [ background isTranslucentButNotTransparent ]) or: [ border isColor and: [ border isTranslucentButNotTransparent ] ] ] Anti-Quick Return Rule FP?
35. background ifNil: [ ˆ true ]. (background isColor and: [ background isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. (border isColor and: [ border isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. ˆ false ˆ background isNil or: [ (background isColor and: [ background isTranslucentButNotTransparent ]) or: [ border isColor and: [ border isTranslucentButNotTransparent ] ] ] Anti-Quick Return Rule fFP
36. background ifNil: [ ˆ true ]. (background isColor and: [ background isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. (border isColor and: [ border isTranslucentButNotTransparent ]) ifTrue: [ ˆ true ]. ˆ false ˆ background isNil or: [ (background isColor and: [ background isTranslucentButNotTransparent ]) or: [ border isColor and: [ border isTranslucentButNotTransparent ] ] ] Anti-Quick Return Rule fFP
37. Expected Blocks Rule (denominator = 0) ifTrue: [ Float infinity ] ifFalse: [ numerator / denominator ]
38. Expected Blocks Rule (denominator = 0) ifTrue: Float infinity ifFalse: numerator / denominator
39. Expected Blocks Rule (denominator = 0) ifTrue: [ Float infinity ] ifFalse: [ numerator / denominator ]
40. Expected Blocks Rule size = 1 ifTrue: ’:’ ifFalse: ’s:’
41. Expected Blocks Rule size = 1 ifTrue: ’:’ ifFalse: ’s:’ FP
42. Expected Blocks Rule size = 1 ifTrue: ’:’ ifFalse: ’s:’ FP
43. Expected Blocks Rule size = 1 ifTrue: ’:’ ifFalse: ’s:’ FP?
44. Expected Blocks Rule size = 1 ifTrue: ’:’ ifFalse: ’s:’ fFP
45. Trivial False False Positive Rules Uncommented Class Unused Variable
46. Trivial False False Positive Rules Uncommented Class Unused Variable
47. Trivial False False Positive Rules Uncommented Class Unused Variable FP?
48. Trivial False False Positive Rules Uncommented Class Unused Variable fFP
49. Do not Introduce
 False False Positives!
50. Do not Introduce
 False False Positives! FP exaggerate FP ratio
51. exaggerate FP ratio mask intentions
 Do not Introduce
 False False Positives! FP
52. Do not Introduce
 False False Positives! exaggerate FP ratio mask intentions
 neglect tool de!ciencies FP
53. exaggerate FP ratio mask intentions
 ignore user groups neglect tool de!ciencies Do not Introduce
 False False Positives! FP
54. exaggerate FP ratio mask intentions
 ignore user groups neglect tool de!ciencies @yuriy_tymchuk Do not Introduce
 False False Positives! FP