Context Is King: The Developer Perspective on the Usage of Static Analysis Tools

Our study about the usage of Automatic Static Analysis Tools in different software development contexts presented at the 25th edition of the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER '18) in Campobasso, Italy. Preprint of the corresponding paper available at http://www.ifi.uzh.ch/seal/people/vassallo/VassalloSANER18.pdf

Transcript

1. 1 Context Is King: The Developer Perspective on the Usage of Static Analysis Tools. Carmine Vassallo, Sebastiano Panichella, Fabio Palomba, Sebastian Proksch, Andy Zaidman, and Harald Gall. @ccvassallo
2. 2 Development Context Is King: The Developer Perspective on the Usage of Static Analysis Tools. Carmine Vassallo, Sebastiano Panichella, Fabio Palomba, Sebastian Proksch, Andy Zaidman, and Harald Gall. @ccvassallo
3. 3 Development Context Is King: The Developer Perspective on the Usage of Static Analysis Tools. Carmine Vassallo, Sebastiano Panichella, Fabio Palomba, Sebastian Proksch, Andy Zaidman, and Harald Gall. @ccvassallo
4. 4 Motivation
5. 5 ASAT (A) Automatic (S) Static (A) Analysis (T) Tool
6. 6 ASATs detect so+ware defects faster and cheaper than human inspec6on and tes6ng would do (Johnson et al., ICSE 2013). ASATs are common, but not ubiquitous (Beller et al., SANER 2016)
7. 7 Barriers when using ASATs Lack of effec6vely implemented quick fixes Johnson et al., “Why don’t software developers use Static Analysis Tools to Find Bugs?” ICSE 2013 High rate of false posi6ve warnings Low understandability of the warnings
8. 8 Usage of ASATs in one context Panichella et al., “Would static analysis tools help developers with code reviews?” SANER 2015 Zampetti et al., “How open source projects use static code analysis tools in continuous integration” MSR 2017 Build failures caused by ASATs are mainly due to coding standard viola.ons Developers use ASATs mainly for checking coding structure Code Review Continuous Integration
9. ASATs configured differently in different contexts. 9
10. TO DO List 10 Development Contexts where ASATs are used Usage of ASATs in different contexts
11. First Study: Development Contexts 11
12. First Study: Research Questions • RQ1: In which development contexts do developers use ASATs? 
 • RQ2: How do developers configure ASATs in different development contexts? 12
13. The Questionnaire 13 19 questions, 2 main topics: • Adoption of ASATs • Configuration of ASATs 43 (69% industrial and 31% open-source) participants.
14. Usage of ASATs 14 Frequency Multiple times per day Daily Weekly Monthly % Respondents 0 10 20 30 40 12 19 31 38 ASATs are integrated with the regular development
15. 15 Where ASATs are used
16. Where ASATs are used 16 30% 33% 37% % Respondents Local Development Code Review Continuous Integration
17. When ASATs are configured 17 Frequency Kick-off Monthly Never Weekly % Respondents 0 15 30 45 60 7 20 22 51 The majority of developers configure ASATs only once.
18. 18 How ASATs are configured of our respondents use the same configuration in different contexts. % 75
19. How ASATs are configured 19 Local Development Code Review Continuous Integration
20. Second Study: ASATs usage in different contexts 20
21. Extended questionnaire & Interviews 21 Context-Based Usage 25 participants 11 professional developers • 6 companies Semi-structured interviews
22. Second Study: Research Question • RQ3 Do developers pay attention to the same warnings in different development contexts? 22
23. Warnings in different contexts 23 Local Development Code Review Continuous Integration Developers pay attention to different warnings depending on the context. Code Structure Logic Error Handling Style Convention Redundancies Naming Conventions Error Handling Logic Style Convention 1st 2nd 3rd 1st 2nd 3rd 1st 2nd 3rd
24. Other factors while selecting warnings 24 Factors Severity of the Warnings Policies of the Development Team Application Type Team Composition None of the above Tool Reputation % Respondents 0 15 30 45 60 0 6.1 6.1 12.1 24.2 51.5 2.4 2.4 9.9 19.5 31.7 34.1 2.3 7 11.6 18.6 27.9 32.6 Continuous Integration Code Review Local Development Blocker, Cri>cal, Major, etc. “Team leader decides to adopt a strict policy regarding naming conven6ons.” “Short-term applica6ons don’t need to follow strict rules.”
25. 25 Findings
26. How developers configure ASATs 26 Local Development Code Review Continuous Integration
27. How developers perceive ASATs 27 Local Development Code Review Continuous Integration
28. 28 Biased Percep6on Towards Context-Awareness Holis6c Analysis of the Developers’ Behaviour
29. 29 Context Is King: The Developer Perspective on the Usage of Static Analysis Tools. Carmine Vassallo, Sebastiano Panichella, Fabio Palomba, Sebastian Proksch, Andy Zaidman, and Harald Gall. @ccvassallo vassallo@ifi.uzh.ch X Usage of ASATs in one context Panichella et al., “Would static analysis tools help developers with code reviews?” SANER 2015 Zampetti et al., “How open source projects use static code analysis tools in continuous integration” MSR 2017 Code Review Continuous Integration How ASATs are configured X Local Development Code Review Continuous Integration How developers perceive ASATs X Local Development Code Review Continuous Integration