Can We Identify Stack Overflow Questions Requiring Code Snippets? Investigating the Cause & Effect of Missing Code Snippets

Saikat Mondal, M. Masudur Rahman and Chanchal K. Roy. Can We Identify Stack Overflow Questions Requiring Code Snippets? Investigating the Cause & Effect of Missing Code Snippets. In Proceeding of The 31st IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER 2024), pp. 12, Rovaniemi, Finland, March 2024

Transcript

1. Saikat Mondal University of Saskatchewan Canada Masud Rahman Dalhousie University Canada Chanchal Roy University of Saskatchewan Canada Can We Identify Stack Overflow Questions Requiring Code Snippets? Investigating the Cause & Effect of Missing Code Snippets
2. Presentation outline Research Goal Findings Methodology 2 3 4 Target Domain, Problems & Motivation 1
3. Why is the programming-related knowledge base of Stack Overflow invaluable? 24 million questions 36 million answers 53rd most- visited site in the world Most popular among 182 sites of Stack Exchange! Get indexed top by Google! 22 million users Problem & Motivation Methodology RQ1 RQ2 Research Goal RQ3
4. Motivation Problem & Motivation Methodology RQ1 RQ2 Research Goal RQ3 Stack Overflow users often post questions to find solutions to code- related problems (e.g., coding errors and unexpected code behavior). Traditionally, users analyze the code snippets to identify or reproduce the reported problems. Upon success, they can submit appropriate solutions. Unfortunately, they often miss required code snippets when submitting questions, which could prevent their questions from getting prompt and appropriate answers in a timely fashion. Question submitters can add code snippets upon request, which could lead to unexpected delays in answering questions. Such a scenario also explains the 31% unanswered and more than 50% unresolved questions at Stack Overflow. Therefore, it is warranted to have a comprehensive understanding of why question submitters miss required code snippets and how missing code snippets affect the questions at Stack Overflow.
5. Our study goal To investigate the cause & effect of missing code snippets in Stack Overflow questions (whenever required) Research Goal Problem & Motivation Methodology RQ1 RQ2 Research Goal RQ3
6. How do the answers get affected when the questions miss the required code snippets at Stack Overflow? What other factors affect questions receiving answers besides missing required code? Why do question submitters miss the required code snippets during question submission? RQ1 RQ2 Research questions Can we predict questions that need code snippets during submission? RQ3 Problem & Motivation Methodology RQ1 RQ2 Research Goal RQ3
7. Methodology Problem & Motivation Methodology RQ1 RQ2 Research Goal RQ3 Missing code (MICO) Code after comment (COAC) Code during submission (CODS) Do not need code Need code Effect Analysis of Missing Code & Confounding Factors Extract answer meta-data Feature extraction RQ1 Development of prediction models Performance analysis of models RQ3 Step 1 Selection of dataset Step 2 Effect analysis Step 4 Model construction User study to understand why question submitters miss code RQ2 Step 3 Reasons analysis Target Stack Overflow questions
8. Effect Analysis of Missing Code Snippets Problem & Motivation Methodology RQ1 RQ2 Research Goal Does the inclusion of required code snippets in Stack Overflow questions encourage acceptable answers? RQ1(a) Fig. Percentage and count of resolved & unresolved questions Problem & Motivation Methodology RQ1 RQ2 Research Goal RQ3
9. Effect Analysis of Missing Code Snippets Does the inclusion of required code snippets in Stack Overflow questions reduce the time delay in getting acceptable answers? RQ1(b) Fig. Time delay of receiving acceptable answers Table: Statistical tests summary of the time delay of receiving accepted answer Problem & Motivation Methodology RQ1 RQ2 Research Goal RQ3
10. Effect Analysis of Missing Code Snippets Does the inclusion of required code snippets in Stack Overflow questions encourage answers? RQ1(c) Fig. Percentage of unanswered questions Problem & Motivation Methodology RQ1 RQ2 Research Goal RQ3
11. Effect Analysis of Missing Code Snippets What factors affect questions receiving answers besides including required code? RQ1(d) Problem & Motivation Methodology RQ1 RQ2 Research Goal RQ3 Reputation Question Submission Time These factors impact questions receiving answers. However, including required code snippets consistently increases the chance of getting prompt and acceptable answers. We investigate the two most potential factors
12. Perceived Reasons Behind Missing the Required Code Snippets Problem & Motivation Methodology RQ1 RQ2 Research Goal RQ3 Tables: Reasons behind missing the code snippets with Stack Overflow questions whenever required
13. Perceived Reasons Behind Missing the Required Code Snippets Problem & Motivation Methodology RQ1 RQ2 Research Goal RQ3 Tables: Other reasons (excluding the reasons in previous Table) behind missing the code snippets
14. Prediction Models to Identify the Questions Requiring Code Snippets Problem & Motivation Methodology RQ1 RQ2 Research Goal RQ3 We develop six ML models using four text-based features. Our models can identify the questions that need code snippets with 86.5% of precision and 90.8% of recall.
15. Findings Summary Only 23.8% of questions get acceptable answers that miss code snippets, as opposed to 61.4% that include code during their submission. 28% of questions remain unanswered when they lack the required code snippets. Users are not always aware of whether their questions need code snippets. ML models can predict the target questions needing code snippets with the highest precision of 85%. Including necessary code snippets consistently encourages acceptable answers to questions with maximum delay despite potential confounding factors.
16. Implications & Future Works Our work has the potential to – • save significant time in programming question-answering and • improve the quality of the valuable knowledge base by decreasing unanswered and unresolved questions. We aim to introduce tool support to the Stack Overflow question submission system to detect questions that need code snippets during submission.
17. Find the replication package https://shorturl.at/giwIM masud.rahman@dal.ca, saikat.mondal@usask.ca Thank You! Any questions? Can We Identify Stack Overflow Questions Requiring Code Snippets? Investigating the Cause & Effect of Missing Code Snippets