JIT Feedback — what Experienced Developers like about Static Analysis (icpc2018)

Although software developers are usually reluctant to use static analysis to detect issues in their source code, our automatic just-in-time static analysis assistant was integrated into an Integrated Development Environment, and was evaluated positively by its users. We conducted interviews to understand the impact of the tool on experienced developers, and how it performs in comparison with other static analyzers.

We learned that the availability of our tool as a default IDE feature and its automatic execution are the main reasons for its adoption. Moreover, the fact that immediate feedback is provided directly in the related development context is essential to keeping developers satisfied, although in certain cases feedback delivered later was deemed more useful. We also discovered that static analyzers can play an educational role, especially in combination with domain- specific rules.

Transcript

1. JIT Feedback — what Experienced
   Developers like about Static Analysis
   Yuriy Tymchuk
   Swisscom, Switzerland
   Mohammad Ghafari
   UniBern, Switzerland
   Oscar Nierstrasz
   UniBern, Switzerland
   3
   

   View Slide

2. How do Developers use Static Analysis
   When They use Static Analysis
   Mohammad Ghafari
   UniBern, Switzerland
   Oscar Nierstrasz
   UniBern, Switzerland
   *WITH A TWIST
   4
   @yuriy_tymchuk
   Swisscom, Switzerland
   

   View Slide

3. Wouldn’t Make it Alone
   Mohammad Oscar
   5
   

   View Slide

4. 6
   The research presented here was all
   done in the University of Bern.
   Nothing Swisscom-related in this
   presentation, folks.
   

   View Slide

5. Static Analyzers
   7
   

   View Slide

6. 8
   Static Analyzers
   

   View Slide

7. 9
   Static Analysis Camps
   

   View Slide

8. 9
   Static Analysis Camps
   

   View Slide

9. 10
   Static Analysis Camps
   Continuous quality assessment with
   inCode.

   Ganea, Verebi, Marinescu.

   Science of Computer Prog. 2017
   Usage contracts: Offering imme-
   diate feedback on violations of
   structural source-code regularities.

   Lozano, Mens, Kellens 

   Science of Computer Prog. 2015
   Using Static Analysis to Find Bugs.

   Ayewah, Hovemeyer, Morgenthaler,
   Penix, Pugh.
   IEEE Software 2008
   

   View Slide

10. 11
    Static Analysis Camps
    We have new
    useful analyzers!
    

    View Slide

11. 12
    Static Analysis Camps
    We have new
    useful analyzers! Analyzing the State of Static
    Analysis: A Large-Scale Evaluation
    in Open Source Software.

    Beller, Bholanath, McIntosh,
    Zaidman.

    SANER 2016.
    Why Don’t Software Developers Use
    Static Analysis Tools to Find Bugs? 

    Johnson, Song, Murphy-Hill,
    Bowdidge.

    ICSE 2013
    

    View Slide

12. 13
    Static Analysis Camps
    We have new
    useful analyzers!
    Nobody uses
    static analyzers
    

    View Slide

13. 14
    Static Analysis Camps
    We have new
    useful analyzers!
    Nobody uses
    static analyzers
    But they are
    not used?
    

    View Slide

14. 15
    Static Analysis Camps
    We have new
    useful analyzers!
    Nobody uses
    static analyzers
    But they are
    not used?
    But do they
    know about…
    

    View Slide

15. We shape our tools, and thereafter
    our tools shape us.
    — McLuhan/Culkin
    16
    

    View Slide

16. 17
    Context: Pharo
    

    View Slide

17. 18
    Context: Pharo
    Object-Oriented
    Dynamically Typed
    

    View Slide

18. 19
    Context: Pharo
    Object-Oriented
    Dynamically Typed
    Language and IDE
    

    View Slide

19. 20
    Central Tool: Code Editor
    

    View Slide

20. 21
    Central Tool: Code Editor
    

    View Slide

21. 22
    Intrusive Quality Assistant
    

    View Slide

22. 23
    Is QA Useful or Distracting?
    

    View Slide

23. 24
    0%
    20%
    40%
    < 1 1–3 4–6 7–10 10+
    Smalltalk Pharo
    N/A
    4% Academia 75% Industry 21%
    24 Participants
    (years)
    Is QA Useful or Distracting?
    Survey
    

    View Slide

24. 25
    Useful
    SOMETIMES
    Sometimes Distracting
    VERY
    Neutral
    0% 25% 50% 75% 100%
    8%
    Is QA Useful or Distracting?
    Survey
    0%
    20%
    40%
    < 1 1–3 4–6 7–10 10+
    Smalltalk Pharo
    N/A
    4% Academia 75% Industry 21%
    24 Participants
    (years)
    

    View Slide

25. 26
    Why/How Quality Assistant?
    

    View Slide

26. 27
    14 Researcher 36%
    Sen. Engineer
    14%
    Engineer 50%
    0% 20% 40% 60% 80% 100%
    <5 5–10 10–20 20+
    0% 20% 40% 60% 80% 100%
    <3 3–6 6+
    Development Experience Pharo Experience
    Interview
    Why/How Quality Assistant?
    5
    3
    2 5
    3
    4
    2 3
    9
    2
    7 2
    5
    

    View Slide

27. 28
    It’s Always There
    

    View Slide

28. 29
    “No installation required”
    “No execution required”
    It’s Always There
    

    View Slide

29. 30
    Not Always There
    

    View Slide

30. 31
    Adds a Final Drop
    Maybe just once?
    

    View Slide

31. 32
    No!
    Adds a Final Drop
    Maybe just once?
    

    View Slide

32. 34
    Instant and Scoped Feedback
    

    View Slide

33. View Slide

34. 36
    Scoped Feedback
    

    View Slide

35. 37
    Scoped Feedback
    

    View Slide

36. 38
    Scoped Feedback
    ~8.5
    ~1.5
    

    View Slide

37. 39
    The Bad Side of QA
    

    View Slide

38. 40
    “There is this ‘missing yourself ’ rule with many
    false positives”
    The Bad Side of QA
    

    View Slide

39. 41
    “Uncommented class? Meh…”
    The Bad Side of QA
    “There is this ‘missing yourself ’ rule with many
    false positives”
    

    View Slide

40. 42
    “Sometimes not clear what is the problem”
    “Uncommented class? Meh…”
    The Bad Side of QA
    “There is this ‘missing yourself ’ rule with many
    false positives”
    

    View Slide

41. 43
    “Sometimes not clear what is the problem”
    “Takes too much of the editor space”
    “Uncommented class? Meh…”
    The Bad Side of QA
    “There is this ‘missing yourself ’ rule with many
    false positives”
    

    View Slide

42. 44
    Part of Something More
    

    View Slide

43. 45
    Part of Something More
    

    View Slide

44. 46
    Part of Something More
    

    View Slide

45. 47
    Part of Something More
    Fast Slow
    Slow
    

    View Slide

46. 48
    Part of Something More
    Fast Slow
    Slow
    Personal
    Personal Shared
    

    View Slide

47. 49
    Part of Something More
    Fast Slow
    Slow
    Personal
    Personal Shared
    Overview Glimpse Summary
    

    View Slide

48. 50
    QualityAssistant as a Teacher
    

    View Slide

49. 51
    QualityAssistant as a Teacher
    nodes select: [ :node | node isUsed not ]
    

    View Slide

50. 52
    QualityAssistant as a Teacher
    nodes select: [ :node | node isUsed not ]
    nodes reject: [ :node | node isUsed ]
    “I learned how to program better”
    — Student
    

    View Slide

51. 53
    QA as Communication Medium
    

    View Slide

52. 54
    QA as Communication Medium
    

    View Slide

53. 55
    QA as Communication Medium
    

    View Slide

54. 55
    QA as Communication Medium
    

    View Slide

55. 56
    QA as Communication Medium
    

    View Slide

56. 56
    QA as Communication Medium
    

    View Slide

57. 57
    QA as Communication Medium
    return presentation.list()
    .title("Best movies")
    .display(...)
    .sorting(...)
    .items(downloadMovies())
    

    View Slide

58. 58
    QA as Communication Medium
    return presentation.list()
    .title("Best movies")
    .display(...)
    .sorting(...)
    .items(downloadMovies())
    

    View Slide

59. 59
    QA as Communication Medium
    return presentation.list()
    .title("Best movies")
    .display(...)
    .sorting(...)
    .items(λ(downloadMovies()))
    

    View Slide

60. 60
    QA as Communication Medium
    

    View Slide

61. 61
    QA as Communication Medium
    

    View Slide

62. 62
    QA as Communication Medium
    “The most

    useful rules!”
    

    View Slide

63. 63
    Automatic, constant feedback
    

    View Slide

64. 64
    Automatic, constant feedback
    Narrow, contextual scope
    

    View Slide

65. 65
    Automatic, constant feedback
    Static analysis as a teacher
    Narrow, contextual scope
    

    View Slide

66. 66
    Automatic, constant feedback
    Static analysis as a teacher
    Narrow, contextual scope
    Common static analysis issues
    

    View Slide

67. 67
    Do you want to see developers
    using static analyzers?
    

    View Slide