An Empirical Study on Recommendations of Similar Bugs (SANER 2016)

The work to be performed on open source systems, whether feature developments or defects, is typically described as an issue (or bug). Developers self-select bugs from the many open bugs in a repository when they wish to perform work on the system. This paper evaluates a recommender, called NextBug, that considers the textual similarity of bug descriptions to predict bugs that require handling of similar code fragments. First, we evaluate this recommender using 69 projects in the Mozilla ecosystem. We show that for detecting similar bugs, a technique that considers just the bug components and short descriptions perform just as well as a more complex technique that considers other features. Second, we report a field study where we monitored the bugs fixed for Mozilla during a week. We sent mails to the developers who fixed these bugs, asking whether they would consider working on the recommendations provided by NextBug; 39 developers (59%) stated that they would consider working on these recommendations; 44 developers (67%) also expressed interest in seeing the recommendations in their bug tracking system

Transcript

1. An Empirical Study on
   Recommendations of Similar Bugs
   Henrique Rocha (UFMG, Brazil)
   Marco Túlio Valente (UFMG, Brazil)
   Humberto Marques-Neto (PUC-Minas, Brazil)
   Gail Murphy (UBC, Canada)
   SANER 2016
   

   View Slide

2. Introduction
   n  Evaluate the feasibility of recommending
   similar bugs to open source developers
   o  More efficient to work on similar code
   o  Less effort to locate and understand code
   2
   

   View Slide

3. Motivation
   3
   

   View Slide

4. Challenge
   n  Open source systems have many issues
   (or bugs) and developers
   o  Mozilla 2.8K fixed issues per month
   and 2K contributors (2012).
   n  Can we predict similar bugs?
   o  Bugs with similar code fragments
   o  Using only bug report fields
   4
   

   View Slide

5. Goal
   n  Evaluate 2 techniques for recommending
   similar bugs
   o  NextBug [Rocha et al. JSERD 2015]
   o  REP [Sun et al. ASE 2011]
   5
   

   View Slide

6. Evaluated Techniques
   6
   

   View Slide

7. NextBug
   7
   

   View Slide

8. NextBug
   8
   

   View Slide

9. REP
   n  Proposed to detect duplicate bug reports
   n  Uses 7 features to compute bug similarity
   o  Textual description (unigram & bigram),
   component, product, version, type, & priority.
   n  Two-round parameter training
   9
   

   View Slide

10. Comparative Study
    10
    

    View Slide

11. Study Design
    n  Dataset: 65K Mozilla bugs (2009 – 2012)
    n  Metrics: Feedback, Precision, Likelihood,
    Recall, and F-score
    n  Oracle: Two bugs are similar if the set of
    changed files are at least 50% similar
    11
    

    View Slide

12. Feedback
    12
    

    View Slide

13. Precision
    13
    

    View Slide

14. Execution Time (ms)
    14
    

    View Slide

15. Field Study
    15
    

    View Slide

16. Survey Setup
    n  We showed NextBug recommendations
    for real bugs to 176 Mozilla developers
    o  66 devs (37%) responded
    o  We asked two survey questions
    16
    

    View Slide

17. Question #1
    Would you consider working on one of the
    recommended bugs?
    17
    “The suggestions seem pretty accurate (...) the
    #3 bug is new to me — I didn’t know about that
    one” (Subject #47)
    “Bugs may interest me but are definitely not my
    current focus” (Subject #1)
    

    View Slide

18. Question #2
    Would you consider useful a Bugzilla
    extension with recommendations?
    18
    “(...) would be immensely helpful for new
    contributors as they don’t know the project
    very well” (Subject #10)
    “I don’t manage a lot of open bugs at the same
    time, so I don’t need a list of
    suggestions” (Subject #22)
    

    View Slide

19. Final Remarks
    19
    

    View Slide

20. Conclusions
    n  Comparative study (65K bugs)
    o  NextBug & REP show similar results
    o  NextBug has a better execution time
    n  Field Study: Devs confirmed that NextBug
    can provide useful recommendations
    20
    

    View Slide

21. henrique.rocha@dcc.ufmg.br
    mtov@dcc.ufmg.br
    humberto@pucminas.br
    murphy@cs.ubc.ca
    Thank you! Arigatou gozaimasu!
    Questions?
    SANER 2016
    

    View Slide