JIT Feedback — what Experienced Developers like about Static Analysis (icpc2018)

JIT Feedback — what Experienced Developers like about Static Analysis (icpc2018)

Although software developers are usually reluctant to use static analysis to detect issues in their source code, our automatic just-in-time static analysis assistant was integrated into an Integrated Development Environment, and was evaluated positively by its users. We conducted interviews to understand the impact of the tool on experienced developers, and how it performs in comparison with other static analyzers.

We learned that the availability of our tool as a default IDE feature and its automatic execution are the main reasons for its adoption. Moreover, the fact that immediate feedback is provided directly in the related development context is essential to keeping developers satisfied, although in certain cases feedback delivered later was deemed more useful. We also discovered that static analyzers can play an educational role, especially in combination with domain- specific rules.

9847702de1f7f19db8c3e158325baa8e?s=128

Yuriy Tymchuk

May 27, 2018
Tweet

Transcript

  1. JIT Feedback — what Experienced Developers like about Static Analysis

    Yuriy Tymchuk Swisscom, Switzerland Mohammad Ghafari UniBern, Switzerland Oscar Nierstrasz UniBern, Switzerland 3
  2. How do Developers use Static Analysis When They use Static

    Analysis Mohammad Ghafari UniBern, Switzerland Oscar Nierstrasz UniBern, Switzerland *WITH A TWIST 4 @yuriy_tymchuk Swisscom, Switzerland
  3. Wouldn’t Make it Alone Mohammad Oscar 5

  4. 6 The research presented here was all done in the

    University of Bern. Nothing Swisscom-related in this presentation, folks.
  5. Static Analyzers 7

  6. 8 Static Analyzers

  7. 9 Static Analysis Camps

  8. 9 Static Analysis Camps

  9. 10 Static Analysis Camps Continuous quality assessment with inCode.
 Ganea,

    Verebi, Marinescu.
 Science of Computer Prog. 2017 Usage contracts: Offering imme- diate feedback on violations of structural source-code regularities.
 Lozano, Mens, Kellens 
 Science of Computer Prog. 2015 Using Static Analysis to Find Bugs.
 Ayewah, Hovemeyer, Morgenthaler, Penix, Pugh. IEEE Software 2008
  10. 11 Static Analysis Camps We have new useful analyzers!

  11. 12 Static Analysis Camps We have new useful analyzers! Analyzing

    the State of Static Analysis: A Large-Scale Evaluation in Open Source Software.
 Beller, Bholanath, McIntosh, Zaidman.
 SANER 2016. Why Don’t Software Developers Use Static Analysis Tools to Find Bugs? 
 Johnson, Song, Murphy-Hill, Bowdidge.
 ICSE 2013
  12. 13 Static Analysis Camps We have new useful analyzers! Nobody

    uses static analyzers
  13. 14 Static Analysis Camps We have new useful analyzers! Nobody

    uses static analyzers But they are not used?
  14. 15 Static Analysis Camps We have new useful analyzers! Nobody

    uses static analyzers But they are not used? But do they know about…
  15. We shape our tools, and thereafter our tools shape us.

    — McLuhan/Culkin 16
  16. 17 Context: Pharo

  17. 18 Context: Pharo Object-Oriented Dynamically Typed

  18. 19 Context: Pharo Object-Oriented Dynamically Typed Language and IDE

  19. 20 Central Tool: Code Editor

  20. 21 Central Tool: Code Editor

  21. 22 Intrusive Quality Assistant

  22. 23 Is QA Useful or Distracting?

  23. 24 0% 20% 40% < 1 1–3 4–6 7–10 10+

    Smalltalk Pharo N/A 4% Academia 75% Industry 21% 24 Participants (years) Is QA Useful or Distracting? Survey
  24. 25 Useful SOMETIMES Sometimes Distracting VERY Neutral 0% 25% 50%

    75% 100% 8% Is QA Useful or Distracting? Survey 0% 20% 40% < 1 1–3 4–6 7–10 10+ Smalltalk Pharo N/A 4% Academia 75% Industry 21% 24 Participants (years)
  25. 26 Why/How Quality Assistant?

  26. 27 14 Researcher 36% Sen. Engineer 14% Engineer 50% 0%

    20% 40% 60% 80% 100% <5 5–10 10–20 20+ 0% 20% 40% 60% 80% 100% <3 3–6 6+ Development Experience Pharo Experience Interview Why/How Quality Assistant? 5 3 2 5 3 4 2 3 9 2 7 2 5
  27. 28 It’s Always There

  28. 29 “No installation required” “No execution required” It’s Always There

  29. 30 Not Always There

  30. 31 Adds a Final Drop Maybe just once?

  31. 32 No! Adds a Final Drop Maybe just once?

  32. 34 Instant and Scoped Feedback

  33. 35

  34. 36 Scoped Feedback

  35. 37 Scoped Feedback

  36. 38 Scoped Feedback ~8.5 ~1.5

  37. 39 The Bad Side of QA

  38. 40 “There is this ‘missing yourself ’ rule with many

    false positives” The Bad Side of QA
  39. 41 “Uncommented class? Meh…” The Bad Side of QA “There

    is this ‘missing yourself ’ rule with many false positives”
  40. 42 “Sometimes not clear what is the problem” “Uncommented class?

    Meh…” The Bad Side of QA “There is this ‘missing yourself ’ rule with many false positives”
  41. 43 “Sometimes not clear what is the problem” “Takes too

    much of the editor space” “Uncommented class? Meh…” The Bad Side of QA “There is this ‘missing yourself ’ rule with many false positives”
  42. 44 Part of Something More

  43. 45 Part of Something More

  44. 46 Part of Something More

  45. 47 Part of Something More Fast Slow Slow

  46. 48 Part of Something More Fast Slow Slow Personal Personal

    Shared
  47. 49 Part of Something More Fast Slow Slow Personal Personal

    Shared Overview Glimpse Summary
  48. 50 QualityAssistant as a Teacher

  49. 51 QualityAssistant as a Teacher nodes select: [ :node |

    node isUsed not ]
  50. 52 QualityAssistant as a Teacher nodes select: [ :node |

    node isUsed not ] nodes reject: [ :node | node isUsed ] “I learned how to program better” — Student
  51. 53 QA as Communication Medium

  52. 54 QA as Communication Medium

  53. 55 QA as Communication Medium

  54. 55 QA as Communication Medium

  55. 56 QA as Communication Medium

  56. 56 QA as Communication Medium

  57. 57 QA as Communication Medium return presentation.list() .title("Best movies") .display(...)

    .sorting(...) .items(downloadMovies())
  58. 58 QA as Communication Medium return presentation.list() .title("Best movies") .display(...)

    .sorting(...) .items(downloadMovies())
  59. 59 QA as Communication Medium return presentation.list() .title("Best movies") .display(...)

    .sorting(...) .items(λ(downloadMovies()))
  60. 60 QA as Communication Medium

  61. 61 QA as Communication Medium

  62. 62 QA as Communication Medium “The most
 useful rules!”

  63. 63 Automatic, constant feedback

  64. 64 Automatic, constant feedback Narrow, contextual scope

  65. 65 Automatic, constant feedback Static analysis as a teacher Narrow,

    contextual scope
  66. 66 Automatic, constant feedback Static analysis as a teacher Narrow,

    contextual scope Common static analysis issues
  67. 67 Do you want to see developers using static analyzers?