JIT Feedback — what Experienced Developers like about Static Analysis (icpc2018)

Although software developers are usually reluctant to use static analysis to detect issues in their source code, our automatic just-in-time static analysis assistant was integrated into an Integrated Development Environment, and was evaluated positively by its users. We conducted interviews to understand the impact of the tool on experienced developers, and how it performs in comparison with other static analyzers.

We learned that the availability of our tool as a default IDE feature and its automatic execution are the main reasons for its adoption. Moreover, the fact that immediate feedback is provided directly in the related development context is essential to keeping developers satisfied, although in certain cases feedback delivered later was deemed more useful. We also discovered that static analyzers can play an educational role, especially in combination with domain- specific rules.

Transcript

1. JIT Feedback — what Experienced Developers like about Static Analysis Yuriy Tymchuk Swisscom, Switzerland Mohammad Ghafari UniBern, Switzerland Oscar Nierstrasz UniBern, Switzerland 3
2. How do Developers use Static Analysis When They use Static Analysis Mohammad Ghafari UniBern, Switzerland Oscar Nierstrasz UniBern, Switzerland *WITH A TWIST 4 @yuriy_tymchuk Swisscom, Switzerland
3. Wouldn’t Make it Alone Mohammad Oscar 5
4. 6 The research presented here was all done in the University of Bern. Nothing Swisscom-related in this presentation, folks.
5. Static Analyzers 7
6. 8 Static Analyzers
7. 9 Static Analysis Camps
8. 9 Static Analysis Camps
9. 10 Static Analysis Camps Continuous quality assessment with inCode.
 Ganea, Verebi, Marinescu.
 Science of Computer Prog. 2017 Usage contracts: Offering imme- diate feedback on violations of structural source-code regularities.
 Lozano, Mens, Kellens 
 Science of Computer Prog. 2015 Using Static Analysis to Find Bugs.
 Ayewah, Hovemeyer, Morgenthaler, Penix, Pugh. IEEE Software 2008
10. 11 Static Analysis Camps We have new useful analyzers!
11. 12 Static Analysis Camps We have new useful analyzers! Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software.
 Beller, Bholanath, McIntosh, Zaidman.
 SANER 2016. Why Don’t Software Developers Use Static Analysis Tools to Find Bugs? 
 Johnson, Song, Murphy-Hill, Bowdidge.
 ICSE 2013
12. 13 Static Analysis Camps We have new useful analyzers! Nobody uses static analyzers
13. 14 Static Analysis Camps We have new useful analyzers! Nobody uses static analyzers But they are not used?
14. 15 Static Analysis Camps We have new useful analyzers! Nobody uses static analyzers But they are not used? But do they know about…
15. We shape our tools, and thereafter our tools shape us. — McLuhan/Culkin 16
16. 17 Context: Pharo
17. 18 Context: Pharo Object-Oriented Dynamically Typed
18. 19 Context: Pharo Object-Oriented Dynamically Typed Language and IDE
19. 20 Central Tool: Code Editor
20. 21 Central Tool: Code Editor
21. 22 Intrusive Quality Assistant
22. 23 Is QA Useful or Distracting?
23. 24 0% 20% 40% < 1 1–3 4–6 7–10 10+ Smalltalk Pharo N/A 4% Academia 75% Industry 21% 24 Participants (years) Is QA Useful or Distracting? Survey
24. 25 Useful SOMETIMES Sometimes Distracting VERY Neutral 0% 25% 50% 75% 100% 8% Is QA Useful or Distracting? Survey 0% 20% 40% < 1 1–3 4–6 7–10 10+ Smalltalk Pharo N/A 4% Academia 75% Industry 21% 24 Participants (years)
25. 26 Why/How Quality Assistant?
26. 27 14 Researcher 36% Sen. Engineer 14% Engineer 50% 0% 20% 40% 60% 80% 100% <5 5–10 10–20 20+ 0% 20% 40% 60% 80% 100% <3 3–6 6+ Development Experience Pharo Experience Interview Why/How Quality Assistant? 5 3 2 5 3 4 2 3 9 2 7 2 5
27. 28 It’s Always There
28. 29 “No installation required” “No execution required” It’s Always There
29. 30 Not Always There
30. 31 Adds a Final Drop Maybe just once?
31. 32 No! Adds a Final Drop Maybe just once?
32. 34 Instant and Scoped Feedback
34. 36 Scoped Feedback
35. 37 Scoped Feedback
36. 38 Scoped Feedback ~8.5 ~1.5
37. 39 The Bad Side of QA
38. 40 “There is this ‘missing yourself ’ rule with many false positives” The Bad Side of QA
39. 41 “Uncommented class? Meh…” The Bad Side of QA “There is this ‘missing yourself ’ rule with many false positives”
40. 42 “Sometimes not clear what is the problem” “Uncommented class? Meh…” The Bad Side of QA “There is this ‘missing yourself ’ rule with many false positives”
41. 43 “Sometimes not clear what is the problem” “Takes too much of the editor space” “Uncommented class? Meh…” The Bad Side of QA “There is this ‘missing yourself ’ rule with many false positives”
42. 44 Part of Something More
43. 45 Part of Something More
44. 46 Part of Something More
45. 47 Part of Something More Fast Slow Slow
46. 48 Part of Something More Fast Slow Slow Personal Personal Shared
47. 49 Part of Something More Fast Slow Slow Personal Personal Shared Overview Glimpse Summary
48. 50 QualityAssistant as a Teacher
49. 51 QualityAssistant as a Teacher nodes select: [ :node | node isUsed not ]
50. 52 QualityAssistant as a Teacher nodes select: [ :node | node isUsed not ] nodes reject: [ :node | node isUsed ] “I learned how to program better” — Student
51. 53 QA as Communication Medium
52. 54 QA as Communication Medium
53. 55 QA as Communication Medium
54. 55 QA as Communication Medium
55. 56 QA as Communication Medium
56. 56 QA as Communication Medium
57. 57 QA as Communication Medium return presentation.list() .title("Best movies") .display(...) .sorting(...) .items(downloadMovies())
58. 58 QA as Communication Medium return presentation.list() .title("Best movies") .display(...) .sorting(...) .items(downloadMovies())
59. 59 QA as Communication Medium return presentation.list() .title("Best movies") .display(...) .sorting(...) .items(λ(downloadMovies()))
60. 60 QA as Communication Medium
61. 61 QA as Communication Medium
62. 62 QA as Communication Medium “The most
 useful rules!”
63. 63 Automatic, constant feedback
64. 64 Automatic, constant feedback Narrow, contextual scope
65. 65 Automatic, constant feedback Static analysis as a teacher Narrow, contextual scope
66. 66 Automatic, constant feedback Static analysis as a teacher Narrow, contextual scope Common static analysis issues
67. 67 Do you want to see developers using static analyzers?