JIT Feedback — what Experienced Developers like about Static Analysis (icpc2018)

JIT Feedback — what Experienced Developers like about Static Analysis (icpc2018)

Although software developers are usually reluctant to use static analysis to detect issues in their source code, our automatic just-in-time static analysis assistant was integrated into an Integrated Development Environment, and was evaluated positively by its users. We conducted interviews to understand the impact of the tool on experienced developers, and how it performs in comparison with other static analyzers.

We learned that the availability of our tool as a default IDE feature and its automatic execution are the main reasons for its adoption. Moreover, the fact that immediate feedback is provided directly in the related development context is essential to keeping developers satisfied, although in certain cases feedback delivered later was deemed more useful. We also discovered that static analyzers can play an educational role, especially in combination with domain- specific rules.

9847702de1f7f19db8c3e158325baa8e?s=128

Yuriy Tymchuk

May 27, 2018
Tweet

Transcript

  1. 1.

    JIT Feedback — what Experienced Developers like about Static Analysis

    Yuriy Tymchuk Swisscom, Switzerland Mohammad Ghafari UniBern, Switzerland Oscar Nierstrasz UniBern, Switzerland 3
  2. 2.

    How do Developers use Static Analysis When They use Static

    Analysis Mohammad Ghafari UniBern, Switzerland Oscar Nierstrasz UniBern, Switzerland *WITH A TWIST 4 @yuriy_tymchuk Swisscom, Switzerland
  3. 4.

    6 The research presented here was all done in the

    University of Bern. Nothing Swisscom-related in this presentation, folks.
  4. 9.

    10 Static Analysis Camps Continuous quality assessment with inCode.
 Ganea,

    Verebi, Marinescu.
 Science of Computer Prog. 2017 Usage contracts: Offering imme- diate feedback on violations of structural source-code regularities.
 Lozano, Mens, Kellens 
 Science of Computer Prog. 2015 Using Static Analysis to Find Bugs.
 Ayewah, Hovemeyer, Morgenthaler, Penix, Pugh. IEEE Software 2008
  5. 11.

    12 Static Analysis Camps We have new useful analyzers! Analyzing

    the State of Static Analysis: A Large-Scale Evaluation in Open Source Software.
 Beller, Bholanath, McIntosh, Zaidman.
 SANER 2016. Why Don’t Software Developers Use Static Analysis Tools to Find Bugs? 
 Johnson, Song, Murphy-Hill, Bowdidge.
 ICSE 2013
  6. 13.

    14 Static Analysis Camps We have new useful analyzers! Nobody

    uses static analyzers But they are not used?
  7. 14.

    15 Static Analysis Camps We have new useful analyzers! Nobody

    uses static analyzers But they are not used? But do they know about…
  8. 23.

    24 0% 20% 40% < 1 1–3 4–6 7–10 10+

    Smalltalk Pharo N/A 4% Academia 75% Industry 21% 24 Participants (years) Is QA Useful or Distracting? Survey
  9. 24.

    25 Useful SOMETIMES Sometimes Distracting VERY Neutral 0% 25% 50%

    75% 100% 8% Is QA Useful or Distracting? Survey 0% 20% 40% < 1 1–3 4–6 7–10 10+ Smalltalk Pharo N/A 4% Academia 75% Industry 21% 24 Participants (years)
  10. 26.

    27 14 Researcher 36% Sen. Engineer 14% Engineer 50% 0%

    20% 40% 60% 80% 100% <5 5–10 10–20 20+ 0% 20% 40% 60% 80% 100% <3 3–6 6+ Development Experience Pharo Experience Interview Why/How Quality Assistant? 5 3 2 5 3 4 2 3 9 2 7 2 5
  11. 33.

    35

  12. 38.

    40 “There is this ‘missing yourself ’ rule with many

    false positives” The Bad Side of QA
  13. 39.

    41 “Uncommented class? Meh…” The Bad Side of QA “There

    is this ‘missing yourself ’ rule with many false positives”
  14. 40.

    42 “Sometimes not clear what is the problem” “Uncommented class?

    Meh…” The Bad Side of QA “There is this ‘missing yourself ’ rule with many false positives”
  15. 41.

    43 “Sometimes not clear what is the problem” “Takes too

    much of the editor space” “Uncommented class? Meh…” The Bad Side of QA “There is this ‘missing yourself ’ rule with many false positives”
  16. 50.

    52 QualityAssistant as a Teacher nodes select: [ :node |

    node isUsed not ] nodes reject: [ :node | node isUsed ] “I learned how to program better” — Student
  17. 66.

    66 Automatic, constant feedback Static analysis as a teacher Narrow,

    contextual scope Common static analysis issues