Upgrade to Pro — share decks privately, control downloads, hide ads and more …

JIT Feedback — what Experienced Developers like about Static Analysis (icpc2018)

JIT Feedback — what Experienced Developers like about Static Analysis (icpc2018)

Although software developers are usually reluctant to use static analysis to detect issues in their source code, our automatic just-in-time static analysis assistant was integrated into an Integrated Development Environment, and was evaluated positively by its users. We conducted interviews to understand the impact of the tool on experienced developers, and how it performs in comparison with other static analyzers.

We learned that the availability of our tool as a default IDE feature and its automatic execution are the main reasons for its adoption. Moreover, the fact that immediate feedback is provided directly in the related development context is essential to keeping developers satisfied, although in certain cases feedback delivered later was deemed more useful. We also discovered that static analyzers can play an educational role, especially in combination with domain- specific rules.

Yuriy Tymchuk

May 27, 2018
Tweet

More Decks by Yuriy Tymchuk

Other Decks in Programming

Transcript

  1. JIT Feedback — what Experienced
    Developers like about Static Analysis
    Yuriy Tymchuk
    Swisscom, Switzerland
    Mohammad Ghafari
    UniBern, Switzerland
    Oscar Nierstrasz
    UniBern, Switzerland
    3

    View Slide

  2. How do Developers use Static Analysis
    When They use Static Analysis
    Mohammad Ghafari
    UniBern, Switzerland
    Oscar Nierstrasz
    UniBern, Switzerland
    *WITH A TWIST
    4
    @yuriy_tymchuk
    Swisscom, Switzerland

    View Slide

  3. Wouldn’t Make it Alone
    Mohammad Oscar
    5

    View Slide

  4. 6
    The research presented here was all
    done in the University of Bern.
    Nothing Swisscom-related in this
    presentation, folks.

    View Slide

  5. Static Analyzers
    7

    View Slide

  6. 8
    Static Analyzers

    View Slide

  7. 9
    Static Analysis Camps

    View Slide

  8. 9
    Static Analysis Camps

    View Slide

  9. 10
    Static Analysis Camps
    Continuous quality assessment with
    inCode.

    Ganea, Verebi, Marinescu.

    Science of Computer Prog. 2017
    Usage contracts: Offering imme-
    diate feedback on violations of
    structural source-code regularities.

    Lozano, Mens, Kellens 

    Science of Computer Prog. 2015
    Using Static Analysis to Find Bugs.

    Ayewah, Hovemeyer, Morgenthaler,
    Penix, Pugh.
    IEEE Software 2008

    View Slide

  10. 11
    Static Analysis Camps
    We have new
    useful analyzers!

    View Slide

  11. 12
    Static Analysis Camps
    We have new
    useful analyzers! Analyzing the State of Static
    Analysis: A Large-Scale Evaluation
    in Open Source Software.

    Beller, Bholanath, McIntosh,
    Zaidman.

    SANER 2016.
    Why Don’t Software Developers Use
    Static Analysis Tools to Find Bugs? 

    Johnson, Song, Murphy-Hill,
    Bowdidge.

    ICSE 2013

    View Slide

  12. 13
    Static Analysis Camps
    We have new
    useful analyzers!
    Nobody uses
    static analyzers

    View Slide

  13. 14
    Static Analysis Camps
    We have new
    useful analyzers!
    Nobody uses
    static analyzers
    But they are
    not used?

    View Slide

  14. 15
    Static Analysis Camps
    We have new
    useful analyzers!
    Nobody uses
    static analyzers
    But they are
    not used?
    But do they
    know about…

    View Slide

  15. We shape our tools, and thereafter
    our tools shape us.
    — McLuhan/Culkin
    16

    View Slide

  16. 17
    Context: Pharo

    View Slide

  17. 18
    Context: Pharo
    Object-Oriented
    Dynamically Typed

    View Slide

  18. 19
    Context: Pharo
    Object-Oriented
    Dynamically Typed
    Language and IDE

    View Slide

  19. 20
    Central Tool: Code Editor

    View Slide

  20. 21
    Central Tool: Code Editor

    View Slide

  21. 22
    Intrusive Quality Assistant

    View Slide

  22. 23
    Is QA Useful or Distracting?

    View Slide

  23. 24
    0%
    20%
    40%
    < 1 1–3 4–6 7–10 10+
    Smalltalk Pharo
    N/A
    4% Academia 75% Industry 21%
    24 Participants
    (years)
    Is QA Useful or Distracting?
    Survey

    View Slide

  24. 25
    Useful
    SOMETIMES
    Sometimes Distracting
    VERY
    Neutral
    0% 25% 50% 75% 100%
    8%
    Is QA Useful or Distracting?
    Survey
    0%
    20%
    40%
    < 1 1–3 4–6 7–10 10+
    Smalltalk Pharo
    N/A
    4% Academia 75% Industry 21%
    24 Participants
    (years)

    View Slide

  25. 26
    Why/How Quality Assistant?

    View Slide

  26. 27
    14 Researcher 36%
    Sen. Engineer
    14%
    Engineer 50%
    0% 20% 40% 60% 80% 100%
    <5 5–10 10–20 20+
    0% 20% 40% 60% 80% 100%
    <3 3–6 6+
    Development Experience Pharo Experience
    Interview
    Why/How Quality Assistant?
    5
    3
    2 5
    3
    4
    2 3
    9
    2
    7 2
    5

    View Slide

  27. 28
    It’s Always There

    View Slide

  28. 29
    “No installation required”
    “No execution required”
    It’s Always There

    View Slide

  29. 30
    Not Always There

    View Slide

  30. 31
    Adds a Final Drop
    Maybe just once?

    View Slide

  31. 32
    No!
    Adds a Final Drop
    Maybe just once?

    View Slide

  32. 34
    Instant and Scoped Feedback

    View Slide

  33. 35

    View Slide

  34. 36
    Scoped Feedback

    View Slide

  35. 37
    Scoped Feedback

    View Slide

  36. 38
    Scoped Feedback
    ~8.5
    ~1.5

    View Slide

  37. 39
    The Bad Side of QA

    View Slide

  38. 40
    “There is this ‘missing yourself ’ rule with many
    false positives”
    The Bad Side of QA

    View Slide

  39. 41
    “Uncommented class? Meh…”
    The Bad Side of QA
    “There is this ‘missing yourself ’ rule with many
    false positives”

    View Slide

  40. 42
    “Sometimes not clear what is the problem”
    “Uncommented class? Meh…”
    The Bad Side of QA
    “There is this ‘missing yourself ’ rule with many
    false positives”

    View Slide

  41. 43
    “Sometimes not clear what is the problem”
    “Takes too much of the editor space”
    “Uncommented class? Meh…”
    The Bad Side of QA
    “There is this ‘missing yourself ’ rule with many
    false positives”

    View Slide

  42. 44
    Part of Something More

    View Slide

  43. 45
    Part of Something More

    View Slide

  44. 46
    Part of Something More

    View Slide

  45. 47
    Part of Something More
    Fast Slow
    Slow

    View Slide

  46. 48
    Part of Something More
    Fast Slow
    Slow
    Personal
    Personal Shared

    View Slide

  47. 49
    Part of Something More
    Fast Slow
    Slow
    Personal
    Personal Shared
    Overview Glimpse Summary

    View Slide

  48. 50
    QualityAssistant as a Teacher

    View Slide

  49. 51
    QualityAssistant as a Teacher
    nodes select: [ :node | node isUsed not ]

    View Slide

  50. 52
    QualityAssistant as a Teacher
    nodes select: [ :node | node isUsed not ]
    nodes reject: [ :node | node isUsed ]
    “I learned how to program better”
    — Student

    View Slide

  51. 53
    QA as Communication Medium

    View Slide

  52. 54
    QA as Communication Medium

    View Slide

  53. 55
    QA as Communication Medium

    View Slide

  54. 55
    QA as Communication Medium

    View Slide

  55. 56
    QA as Communication Medium

    View Slide

  56. 56
    QA as Communication Medium

    View Slide

  57. 57
    QA as Communication Medium
    return presentation.list()
    .title("Best movies")
    .display(...)
    .sorting(...)
    .items(downloadMovies())

    View Slide

  58. 58
    QA as Communication Medium
    return presentation.list()
    .title("Best movies")
    .display(...)
    .sorting(...)
    .items(downloadMovies())

    View Slide

  59. 59
    QA as Communication Medium
    return presentation.list()
    .title("Best movies")
    .display(...)
    .sorting(...)
    .items(λ(downloadMovies()))

    View Slide

  60. 60
    QA as Communication Medium

    View Slide

  61. 61
    QA as Communication Medium

    View Slide

  62. 62
    QA as Communication Medium
    “The most

    useful rules!”

    View Slide

  63. 63
    Automatic, constant feedback

    View Slide

  64. 64
    Automatic, constant feedback
    Narrow, contextual scope

    View Slide

  65. 65
    Automatic, constant feedback
    Static analysis as a teacher
    Narrow, contextual scope

    View Slide

  66. 66
    Automatic, constant feedback
    Static analysis as a teacher
    Narrow, contextual scope
    Common static analysis issues

    View Slide

  67. 67
    Do you want to see developers
    using static analyzers?

    View Slide