Context Is King: The Developer Perspective on the Usage of Static Analysis Tools

Our study about the usage of Automatic Static Analysis Tools in different software development contexts presented at the 25th edition of the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER '18) in Campobasso, Italy. Preprint of the corresponding paper available at http://www.ifi.uzh.ch/seal/people/vassallo/VassalloSANER18.pdf

Transcript

1. 1
   Context Is King:
   The Developer Perspective on the Usage of Static Analysis Tools.
   Carmine Vassallo, Sebastiano Panichella, Fabio Palomba,
   Sebastian Proksch, Andy Zaidman, and Harald Gall. @ccvassallo
   

   View Slide

2. 2
   Development Context Is King:
   The Developer Perspective on the Usage of Static Analysis Tools.
   Carmine Vassallo, Sebastiano Panichella, Fabio Palomba,
   Sebastian Proksch, Andy Zaidman, and Harald Gall. @ccvassallo
   

   View Slide

3. 3
   Development Context Is King:
   The Developer Perspective on the Usage of Static Analysis Tools.
   Carmine Vassallo, Sebastiano Panichella, Fabio Palomba,
   Sebastian Proksch, Andy Zaidman, and Harald Gall. @ccvassallo
   

   View Slide

4. 4
   Motivation
   

   View Slide

5. 5
   ASAT
   (A) Automatic
   (S) Static
   (A) Analysis
   (T) Tool
   

   View Slide

6. 6
   ASATs detect so+ware defects faster and cheaper than human
   inspec6on and tes6ng would do (Johnson et al., ICSE 2013).
   ASATs are common, but not ubiquitous (Beller et al., SANER 2016)
   

   View Slide

7. 7
   Barriers when using ASATs
   Lack of effec6vely
   implemented quick fixes
   Johnson et al.,
   “Why don’t software developers use
   Static Analysis Tools to Find Bugs?”
   ICSE 2013
   High rate of false posi6ve
   warnings
   Low understandability of the
   warnings
   

   View Slide

8. 8
   Usage of ASATs in one context
   Panichella et al.,
   “Would static analysis tools help
   developers with code reviews?”
   SANER 2015
   Zampetti et al.,
   “How open source projects use static
   code analysis tools in continuous
   integration”
   MSR 2017
   Build failures caused by ASATs are mainly
   due to coding standard viola.ons
   Developers use ASATs mainly for checking
   coding structure
   Code Review
   Continuous
   Integration
   

   View Slide

9. ASATs configured differently in
   different contexts.
   9
   

   View Slide

10. TO DO List
    10
    Development Contexts where ASATs are used
    Usage of ASATs in different contexts
    

    View Slide

11. First Study:
    Development Contexts
    11
    

    View Slide

12. First Study: Research Questions
    • RQ1: In which development contexts do
    developers use ASATs? 

    • RQ2: How do developers configure ASATs in
    different development contexts?
    12
    

    View Slide

13. The Questionnaire
    13
    19 questions, 2 main topics:
    • Adoption of ASATs
    • Configuration of ASATs
    43 (69% industrial and 31%
    open-source) participants.
    

    View Slide

14. Usage of ASATs
    14
    Frequency
    Multiple times per day
    Daily
    Weekly
    Monthly
    % Respondents
    0 10 20 30 40
    12
    19
    31
    38
    ASATs are integrated with the regular development
    

    View Slide

15. 15
    Where ASATs are used
    

    View Slide

16. Where ASATs are used
    16
    30% 33% 37%
    % Respondents
    Local Development Code Review Continuous Integration
    

    View Slide

17. When ASATs are configured
    17
    Frequency
    Kick-off
    Monthly
    Never
    Weekly
    % Respondents
    0 15 30 45 60
    7
    20
    22
    51
    The majority of developers configure ASATs only once.
    

    View Slide

18. 18
    How ASATs are configured
    of our respondents use the
    same configuration in
    different contexts.
    %
    75
    

    View Slide

19. How ASATs are configured
    19
    Local Development Code Review Continuous Integration
    

    View Slide

20. Second Study:
    ASATs usage in different contexts
    20
    

    View Slide

21. Extended questionnaire & Interviews
    21
    Context-Based Usage
    25 participants
    11 professional
    developers
    • 6 companies
    Semi-structured
    interviews
    

    View Slide

22. Second Study: Research Question
    • RQ3 Do developers pay attention to the same
    warnings in different development contexts?
    22
    

    View Slide

23. Warnings in different contexts
    23
    Local Development Code Review Continuous Integration
    Developers pay attention to different warnings depending on the context.
    Code Structure
    Logic
    Error Handling
    Style Convention
    Redundancies
    Naming Conventions
    Error Handling
    Logic
    Style Convention
    1st
    2nd
    3rd
    1st
    2nd
    3rd
    1st
    2nd
    3rd
    

    View Slide

24. Other factors while selecting warnings
    24
    Factors
    Severity of the Warnings
    Policies of the Development Team
    Application Type
    Team Composition
    None of the above
    Tool Reputation
    % Respondents
    0 15 30 45 60
    0
    6.1
    6.1
    12.1
    24.2
    51.5
    2.4
    2.4
    9.9
    19.5
    31.7
    34.1
    2.3
    7
    11.6
    18.6
    27.9
    32.6
    Continuous Integration Code Review Local Development
    Blocker, Cri>cal, Major, etc.
    “Team leader decides to adopt a strict
    policy regarding naming conven6ons.”
    “Short-term applica6ons don’t need to
    follow strict rules.”
    

    View Slide

25. 25
    Findings
    

    View Slide

26. How developers configure ASATs
    26
    Local Development Code Review Continuous Integration
    

    View Slide

27. How developers perceive ASATs
    27
    Local Development Code Review Continuous Integration
    

    View Slide

28. 28
    Biased Percep6on
    Towards Context-Awareness
    Holis6c Analysis of the
    Developers’ Behaviour
    

    View Slide

29. 29
    Context Is King:
    The Developer Perspective on the Usage of Static Analysis
    Tools.
    Carmine Vassallo, Sebastiano Panichella, Fabio Palomba,
    Sebastian Proksch, Andy Zaidman, and Harald Gall.
    @ccvassallo
    vassallo@ifi.uzh.ch
    X
    Usage of ASATs in one context
    Panichella et al.,
    “Would static analysis tools help
    developers with code reviews?”
    SANER 2015
    Zampetti et al.,
    “How open source projects use static
    code analysis tools in continuous
    integration”
    MSR 2017
    Code Review
    Continuous
    Integration
    How ASATs are configured
    X
    Local Development Code Review Continuous Integration
    How developers perceive ASATs
    X
    Local Development Code Review Continuous Integration
    

    View Slide