Developers’ Sentiment and Issue Reopening

Transcript

1. Jonathan Cheruvelil and Bruno C. da Silva <bcdasilv@calpoly.edu> Developers’ Sentiment and Issue Reopening @BrunoDaSilvaSE github.com/bcdasilv/sentiment-analysis-on-issues #SEmotion19 · ICSE 2019 Workshop · 28 May 2019 · Montréal, QC, Canada https://www.abc.net.au/radionational/programs/allinthemind/our-emotional-brain-6th-may/3985474
2. https://www.builtinboston.com/2017/12/12/day-life-engineer-facebook-cambridge
3. https://www.builtinboston.com/2017/12/12/day-life-engineer-facebook-cambridge
4. SENTIMENT → BUILD STATUS “This fixes a really nasty bug” build broken Commits with negative sentiment are slightly more likely to result in broken builds SENTIMENT → BUILD STATUS Rodrigo Souza and Bruno Silva. Sentiment analysis of Travis CI builds. In MSR '17. IEEE Press, Piscataway, NJ, USA, 459-462.
5. Push commits “Definitely hating [issue] #320” build broken Commits following a broken build tend to be slightly more negative BUILD STATUS → SENTIMENT BUILD STATUS → SENTIMENT Rodrigo Souza and Bruno Silva. Sentiment analysis of Travis CI builds. In MSR '17. IEEE Press, Piscataway, NJ, USA, 459-462.
6. Happier and more polite developers… fix issues faster solve problems better
8. Rework
9. vs
10. Research Questions RQ1: Are comments with negative sentiments more likely to appear in issues that have been reopened?
11. Research Questions RQ1: Are comments with negative sentiments more likely to appear in issues that have been reopened? RQ2: Does a larger comment size correlate with more extreme sentiment scores?
12. Research Questions RQ1: Are comments with negative sentiments more likely to appear in issues that have been reopened? RQ2: Does a larger comment size correlate with more extreme sentiment scores? RQ3: Do different projects have different proportions in regards to sentiment scores and issue reopening status?
13. 8 projects What we did…
14. 8 projects REST API What we did…
15. 8 projects REST API 35k+ issues (excluded issues < 1y old) What we did…
16. 8 projects REST API SentiStrengh-SE 35k+ issues (excluded issues < 1y old) What we did…
17. Md Rakibul Islam, Minhaz F. Zibran, SentiStrength-SE: Exploiting domain specificity for improved sentiment analysis in software engineering text, Journal of Systems and Software, V. 145, 2018, Pages 125-146, How SentiStrength-SE works…
18. “I love tests, but dislike the awful API” Md Rakibul Islam, Minhaz F. Zibran, SentiStrength-SE: Exploiting domain specificity for improved sentiment analysis in software engineering text, Journal of Systems and Software, V. 145, 2018, Pages 125-146, How SentiStrength-SE works…
19. “I love tests, but dislike the awful API” +3 -3 -4 love dislike awful Md Rakibul Islam, Minhaz F. Zibran, SentiStrength-SE: Exploiting domain specificity for improved sentiment analysis in software engineering text, Journal of Systems and Software, V. 145, 2018, Pages 125-146, How SentiStrength-SE works…
20. “I love tests, but dislike the awful API” +3 -3 -4 love dislike awful sentiment score:
 [3, -4] Md Rakibul Islam, Minhaz F. Zibran, SentiStrength-SE: Exploiting domain specificity for improved sentiment analysis in software engineering text, Journal of Systems and Software, V. 145, 2018, Pages 125-146, How SentiStrength-SE works…
21. SentiStrengh-SE 35k+ issues (excluded issues < 1y old) What we did… [3, -4] (sentiment tuple) #comment size Transition history —> #reopenings for each issue
22. RESULTS
23. RQ1: Are comments with negative sentiments more likely to appear in issues that have been reopened?
24. RQ1: Are comments with negative sentiments more likely to appear in issues that have been reopened? (-) scores. All issues.
25. RQ1: Are comments with negative sentiments more likely to appear in issues that have been reopened? (-) scores. All issues. Yes. Small effect size (based on Chi-squared test and Cramer’s V).
26. RQ1: Are comments with negative sentiments more likely to appear in issues that have been reopened? (-) scores. All issues. Yes. Small effect size (based on Chi-squared and Cramer’s V). What about the opposite?
27. (+) scores. All issues. Comments with positive sentiments are also more likely to appear in issues that have been reopened. Small effect size (based on Chi- squared and Cramer’s V).
28. (+) scores. All issues. Comments with positive sentiments are also more likely to appear in issues that have been reopened. Small effect size (based on Chi- squared and Cramer’s V).
29. RQ2: Does a larger comment size correlate with more extreme sentiment scores?
30. comment_size <= 500 words
31. comment_size <= 500 words 500 < comment_size < 1000
32. comment_size <= 500 words 500 < comment_size < 1000 comment_size >= 1000 words
33. RQ2: Does a larger comment size correlate with more extreme sentiment scores? (-) scores. Comment_size <= 500 words (+) scores. Comment_size <= 500 words
34. RQ2: Does a larger comment size correlate with more extreme sentiment scores? (-) scores. Comment_size <= 500 words (+) scores. Comment_size <= 500 words 0.7% 4.7%
35. RQ2: Does a larger comment size correlate with more extreme sentiment scores? (-) scores. Comment_size <= 500 words (+) scores. Comment_size <= 500 words 0.7% 4.7% 0.4% 6.4%
36. RQ2: Does a larger comment size correlate with more extreme sentiment scores? (-) scores. Comment_size <= 500 words (+) scores. Comment_size <= 500 words 0.7% 4.7% 0.4% 6.4%
37. (-) scores. 500 < Comment_size < 1000 (+) scores. 500 words < Comment_size < 1000 RQ2: Does a larger comment size correlate with more extreme sentiment scores? 4.2% 20% 2% 29%
38. (-) scores. 500 < Comment_size < 1000 (+) scores. 500 words < Comment_size < 1000 RQ2: Does a larger comment size correlate with more extreme sentiment scores? 4.2% 20% 2% 29%
39. RQ2: Does a larger comment size correlate with more extreme sentiment scores? (-) scores. Comment_size >= 1000 words (+) scores. Comment_size >= 1000 words 12% 37% 6% 48%
40. RQ2: Does a larger comment size correlate with more extreme sentiment scores? (-) scores. Comment_size >= 1000 words (+) scores. Comment_size >= 1000 words Yes.
41. RQ2: Does a larger comment size correlate with more extreme sentiment scores? (-) scores. Comment_size >= 1000 words (+) scores. Comment_size >= 1000 words … and a slight correlation with issue reopening
42. RQ3: Do different projects have different proportions in regards to sentiment scores and issue reopening status? Project Title % of issues selected Cramer’s V (-) scores Cramers’ V (+) scores Zookeeper 88% .303 .288 Hadoop 30% .224 .202 MNG 77% .164 .152 Qpid 60% .112 .109 Felix 94% .108 .122 Groovy 69% .101 .107 Zeppelin 51% .072 .065 CloudStack 46% .034 .029
43. RQ3: Do different projects have different proportions in regards to sentiment scores and issue reopening status? Project Title % of issues selected Cramer’s V (-) scores Cramers’ V (+) scores Zookeeper 88% .303 .288 Hadoop 30% .224 .202 MNG 77% .164 .152 Qpid 60% .112 .109 Felix 94% .108 .122 Groovy 69% .101 .107 Zeppelin 51% .072 .065 CloudStack 46% .034 .029 (-) scores. CloudStack issues.
44. RQ3: Do different projects have different proportions in regards to sentiment scores and issue reopening status? Project Title % of issues selected Cramer’s V (-) scores Cramers’ V (+) scores Zookeeper 88% .303 .288 Hadoop 30% .224 .202 MNG 77% .164 .152 Qpid 60% .112 .109 Felix 94% .108 .122 Groovy 69% .101 .107 Zeppelin 51% .072 .065 CloudStack 46% .034 .029 (-) scores. Zookeeper issues.
45. RQ3: Do different projects have different proportions in regards to sentiment scores and issue reopening status? Project Title % of issues selected Cramer’s V (-) scores Cramers’ V (+) scores Zookeeper 88% .303 .288 MNG 77% .164 .152 CloudStack 46% .034 .029 Felix 94% .108 .122 Qpid 60% .112 .109 Zeppelin 51% .072 .065 Groovy 69% .101 .107 Hadoop 30% .224 .202 Yes. (-) scores. Zookeeper issues.
46. Takeaways • If using a lexicon-based approach such as SentiStrength-SE, consider comment size as a possible confounding factor. • We noticed SentiStrength-SE improved SentiStrength for SE text (compared to our previous experience with SentiStrengh)
47. Future work • Replicate with other sentiment analysis approaches (including ML models) • Separate comments made before issue reopenings from those made after the reopenings
48. REST API SentiStrengh-SE 35k+ issues Developers’ Sentiment and Issue Reopening Jonathan Cheruvelil and Bruno C. da Silva <bcdasilv@calpoly.edu> #SEmotion19 ICSE 2019 Workshop · 28 May 2019 · Montréal, QC, Canada Material available: github.com/bcdasilv/sentiment-analysis-on-issues