Do Rapid Releases Affect Bug Reopening? A Case Study of Firefox

Presented at SBES 2014: http://www.ic.ufal.br/evento/cbsoft2014/en/program-sbes-en.html

Link to paper: http://rodrigorgs.github.io/cbsoft14/souza2014.pdf

Transcript

1. Do Rapid Releases
 Affect Bug Reopening?
 A Case Study of Firefox Rodrigo Souza, UFBA Christina Chavez, UFBA Roberto A. Bittencourt, UEFS E S L Software Engineering
 Laboratory Federal University of Bahia
2. Releases
3. Traditional Releases Fixed scope: release when features are complete 12–18 month Bugfix patches can be released as needed
4. Rapid Releases Fixed time: release at fixed time intervals 1–3 month If a feature is not complete, it is not released
5. Rapid Releases and Quality Frequent deliveries 㱺 more feedback 
 㱺 higher quality Tighter deadlines 㱺 less time for system testing
 㱺 lower quality
6. Rapid Releases and Bug Reopening Bug reopening: bug report is mark resolved, something is wrong, it is reopened so it can be properly resolved rework (implementation, testing…) instability in the process or product
7. Research Questions
8. Research Questions Do rapid releases impact… !
9. RQ1 … the bug reopening rate? Do rapid releases impact… !
10. RQ2 … the proportion of bugs reopened due to failing automated tests? Do rapid releases impact… !
11. RQ3 … the time it takes for a bug to be reopened? Do rapid releases impact… !
12. Mozilla Firefox
13. Release cycle 2011: changed from traditional to rapid releases 6-week release cycle Four builds: central (nightly) aurora beta release
14. Data and Methods
15. Data Bug reports and commit log 
 for Mozilla Firefox
 from 2009 to 2013,
 with 2 traditional and 17 rapid releases
16. Commit-Bug Mapping Bug report #314 Commit log NEW RESOLVED/! FIXED CLOSED REOPENED Bug 314: show dialog when the… Back out bug 314 because of the…
17. Bug Selection last resolution = FIXED referenced by a bugfix commit
18. Method Compute metrics under both traditional and rapid releases 0.1 Y/N Plot mosaic plots and time series Run Fisher’s exact test to determine if observed differences are significant Collect feedback from Firefox developers
19. RQ1 (reopening rate) Population: all selected bugs Metric: proportion of reopened bugs Reopened bug:
 transitioned from FIXED to REOPENED or referenced in a back out commit
20. RQ2 (reopening because of testing) Population: reopened bugs Metric: proportion of test-reopened bugs Test-reopened bugs:
 bugs referenced in back out commit messages that include the word “test”
21. RQ3 (time-to-reopen) Population: reopened bugs Metric: time from bug fix to bug reopening Bug reopening:
 when bug status changes to REOPENED or when bug is referenced in a back out commit
22. Results
23. RQ1 (reopening rate) After changing to rapid releases,
 the bug reopening rate increased
 from 12.6% to 13.5%
24. RQ2 (reopening because of testing) There is no significant difference in the proportion of bugs reopened due to failing tests under rapid releases
25. RQ3 (time-to-reopen) traditional
 releases short
 releases latent
 time
 (hours)
26. RQ3 (time-to-reopen) traditional
 releases short
 releases topmost 10%
 latent
 times
 (days) p < 0.05
27. RQ3 (time-to-reopen) Long latent times (measured in weeks and months) tend to be shorter under rapid releases
28. Developer feedback
29. Discussion firefox-dev mailing list 20+ email messages problems detected
30. Preliminary analysis Initially, only bug reports were analyzed Conclusion: 50% less reopening under rapid releases
31. Developer feedback Since 2011, bug reports are RESOLVED only if the bug fixes pass automated integration test If a bug fix fails testing: •Before 2011, it is REOPENED •Since 2011, it is backed out
32. Conclusion
33. Summary After rapid releases, bug reopening increased Long latent periods became shorter
34. Threats to validity Construct 
 detection of reopened bugs (mitigation: comparative) Internal
 difference could be due to other factors (mitigation: feedback) External
 the purpose was not to generalize, but to provide insight
35. Lessons learned Study the development process Explore the data Get feedback on preliminary results relevance language level of detail
36. Do Rapid Releases
 Affect Bug Reopening?
 A Case Study of Firefox Rodrigo Souza, UFBA Christina Chavez, UFBA Roberto A. Bittencourt, UEFS E S L Software Engineering
 Laboratory Federal University of Bahia