Sentiment analysis of Travis CI builds

Talk presented at the Mining Software Repositories (MSR Mining Challenge session) in Buenos Aires, Argentina.

Paper: https://rodrigorgs.github.io/files/msr2017-rodrigo.pdf

Abstract: Human factors such as sentiments, emotions, mood, and stress along with their potential effect on software development are of paramount importance in software engineering, as we still strongly rely on human-to-human interaction for performing software development activities and driving results. With the advance of sentiment analysis tools, software engineering researchers have investigated the interplay between developers’ sentiment and software engineering tasks such as issue fixing times. However, there is a lack of studies analyzing whether there is a relation between developers’ sentiment and builds performed by continuous integration servers. Build breakage is not desired as it represents a signal that something went wrong in the software development activity and that extra work or rework should be done. In this paper, we report an empirical assessment over Travis CI builds and the corresponding commits in order to understand a potential association between developers’ sentiment and build breakage. We found evidence that negative sentiment both affects and is affected by the result of the build process, although the influence seems to be small. Also, we found that developers tend to be more positive when writing about the CI server in commit messages.

Transcript

1. SENTIMENT ANALYSIS OF TRAVIS CI BUILDS SENTIMENT ANALYSIS OF TRAVIS CI BUILDS Rodrigo Souza <rodrigo@dcc.ufba.br> @xrodrigorgs Bruno Silva <bruno.carreiro@pro.unifacs.br> @BrCarreiroSilva Federal University of Bahia University of Salvador
2. EMOTIONS ARE CENTRAL TO HUMANS EMOTIONS ARE CENTRAL TO HUMANS
3. EMOTIONS AFFECT OUR JOB PERFORMANCE EMOTIONS AFFECT OUR JOB PERFORMANCE
4. EMOTIONS AFFECT OUR JOB PERFORMANCE EMOTIONS AFFECT OUR JOB PERFORMANCE
5. JOB PERFORMANCE AFFECTS OUR EMOTIONS JOB PERFORMANCE AFFECTS OUR EMOTIONS
6. SOFTWARE ENGINEERING?
8. Happier developers… fix issues faster solve problems better
9. build successful build broken X
10. Travis CI Push commits “This fixes a really nasty bug” 1.2k projects 609k builds 1M commits SentiStrengh tool build successful build broken or GitHub
11. “I love tests, but dislike the awful API”
12. “I love tests, but dislike the awful API” +3 -3 -4 love dislike awful
13. “I love tests, but dislike the awful API” +3 -3 -4 love dislike awful sentiment score:
 [3, -4]
14. “I love tests, but dislike the awful API” (A sentence can be both positive and negative) +3 -3 -4 love dislike awful sentiment score:
 [3, -4]
15. RESULTS
16. SENTIMENT → BUILD STATUS “This fixes a really nasty bug” SENTIMENT → BUILD STATUS
17. SENTIMENT → BUILD STATUS “This fixes a really nasty bug” build broken Commits with negative sentiment are slightly more likely to result in broken builds SENTIMENT → BUILD STATUS
18. Push commits build broken BUILD STATUS → SENTIMENT BUILD STATUS → SENTIMENT
19. Push commits “Definitely hating [issue] #320” build broken Commits following a broken build tend to be slightly more negative BUILD STATUS → SENTIMENT BUILD STATUS → SENTIMENT
20. build broken vicious circle
21. statistically significant small effect size build broken vicious circle
22. SENTIMENT TOWARDS TRAVIS CI SENTIMENT TOWARDS TRAVIS CI
23. “Travis, I love you, but you’re bringing me down”
24. “Make specs even slower, hopefully Travis will enjoy this.”
25. “MOAR travis FUN”
26. SENTIMENT TOWARDS TRAVIS CI SENTIMENT TOWARDS TRAVIS CI Commits mentioning Travis CI tend to be
 more positive and less negative
27. DISCUSSION
28. SentiStrength
 doesn’t understand
 sarcasm LIMITATIONS LIMITATIONS
29. SentiStrength
 misclassifies as negative
 technical terms in SE LIMITATIONS LIMITATIONS
30. “test against mocked command output” SentiStrength
 misclassifies as negative
 technical terms in SE LIMITATIONS LIMITATIONS
31. “test against mocked command output” SentiStrength
 misclassifies as negative
 technical terms in SE We reclassified >50 words
 from SentiStrength’s database LIMITATIONS LIMITATIONS
32. Manual, detailed sentiment analysis (joy, anger, love, sadness…) FUTURE WORK FUTURE WORK
33. FUTURE WORK FUTURE WORK Separate commits by type
 (bug fix, feature, refactoring…)
34. Scripts available:
 https://gitlab.com/rodrigorgs/msr17-challenge
35. SENTIMENT ANALYSIS OF TRAVIS CI BUILDS SENTIMENT ANALYSIS OF TRAVIS CI BUILDS Rodrigo Souza <rodrigo@dcc.ufba.br> @xrodrigorgs Bruno Silva <bruno.carreiro@pro.unifacs.br> @BrCarreiroSilva Federal University of Bahia University of Salvador https://gitlab.com/rodrigorgs/msr17-challenge