Pro Yearly is on sale from $80 to $50! »

The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code Review

The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code Review

Modern Code Review (MCR) plays a key role in software quality practices. In MCR process, a new patch (i.e., a set of code changes) is encouraged to be examined by reviewers in order to identify weaknesses in source code prior to an integration into main software repositories. To mitigate the risk of having future defects, prior work suggests that MCR should be performed with sufficient review participation. Indeed, recent work shows that a low number of participated reviewers is associated with poor software quality. However, there is a likely case that a new patch still suffers from poor review participation even though reviewers were invited. Hence, in this paper, we set out to investigate the factors that are associated with the participation decision of an invited reviewer. Through a case study of 230,090 patches spread across the Android, LibreOffice, OpenStack and Qt systems, we find that (1) 16%-66% of patches have at least one invited reviewer who did not respond to the review invitation; (2) human factors play an important role in predicting whether or not an invited reviewer will participate in a review; (3) a review participation rate of an invited reviewers and code authoring experience of an invited reviewer are highly associated with the participation decision of an invited reviewer. These results can help practitioners better understand about how human factors associate with the participation decision of reviewers and serve as guidelines for inviting reviewers, leading to a better inviting decision and a better reviewer participation.

2315505e8dc7f0071bd4b9e7e1bf1782?s=128

sruangwan

May 31, 2019
Tweet

Transcript

  1. The Impact of Human Factors on the Participation Decision of

    Reviewers in Modern Code Review Journal-First Paper (Empirical Software Engineering) Shade Ruangwan Patanamon
 Thongtanunam Akinori
 Ihara Kenichi Matsumoto PhD Student
  2. Code review plays a key role in software development to

    improve software quality Remove software defects Improve readability 
 and maintainability 2 Modern Code Review (MCR) a tool-based code review Promote geographically-distributed
 collaboration practices Promote knowledge transfer
 with traceable process
  3. 3 Before a new patch can be integrated into codebase,

    it has to undergone code review A patch author A code review tool (e.g., Gerrit) Uploads a patch
 (code changes)
  4. 4 A patch author A code review tool (e.g., Gerrit)

    Reviewers Uploads a patch
 (code changes) Invites via
 code review tool Before a new patch can be integrated into codebase, it has to undergone code review
  5. 5 A patch author A code review tool (e.g., Gerrit)

    Reviewers Uploads a patch
 (code changes) Invites via
 code review tool Examine the patch,
 provide comments
 and/or review scores Before a new patch can be integrated into codebase, it has to undergone code review
  6. 6 A patch author Uploads a patch
 (code changes) A

    code review tool (e.g., Gerrit) Reviewers Invites via
 code review tool Receives comments
 and revise the patch Examine the patch,
 provide comments
 and/or review scores Before a new patch can be integrated into codebase, it has to undergone code review
  7. 7 A patch author Uploads a patch
 (code changes) A

    code review tool (e.g., Gerrit) Reviewers Invites via
 code review tool Receives comments
 and revise the patch A clean patch 
 can be integrated
 into codebase Main repository +1 +1 +2 Examine the patch,
 provide comments
 and/or review scores Before a new patch can be integrated into codebase, it has to undergone code review
  8. A patch author Uploads a patch
 (code changes) Invites via


    code review tool Receives comments
 and revise the patch 8 The number of reviewers has a large impact on software quality [McIntosh et al. MSR 2014] [Thongtanunam et al. MSR 2015] [Bavota and Russo ICSME 2015] Main repository A code review tool (e.g., Gerrit) A buggy patch
 maybe integrated
 into codebase Fewer Reviewers Examine the patch,
 provide comments
 and/or review scores ?
  9. 9 A patch author invites many reviewers to increase review

    effectiveness A patch author Uploads a patch
 (code changes) A code review tool (e.g., Gerrit) Reviewers Invites via
 code review tool
  10. 10 However, reviewers may not respond to the invitation due

    to the nature of OSS development processes A patch author Uploads a patch
 (code changes) A code review tool (e.g., Gerrit) Reviewers Invites via
 code review tool
  11. 11 We find that many patches have an invited reviewer

    who did not respond to the invitation Studied systems #Patches 36,771 18,716 108,788 65,815 Patches with at least one reviewer who did not respond to the invitation 66% 16% 23% 47% Correlation between #invited reviewers and #reviewers who did not respond to the invitation large (+0.719) large (+0.544) medium (+0.344) large (+0.671)
  12. Studied systems #Patches 36,771 18,716 108,788 65,815 Patches with at

    least one reviewer who did not respond to the invitation 66% 16% 23% 47% Correlation between #invited reviewers and #reviewers who did not respond to the invitation large
 (+0.719) large
 (+0.544) medium
 (+0.344) large
 (+0.671) 12 Inviting more reviewers increases the chance of having an invited reviewer who did not respond
  13. Studied systems #Patches 36,771 18,716 108,788 65,815 Patches with at

    least one reviewer who did not respond to the invitation 66% 16% 23% 47% Correlation between #invited reviewers and #reviewers who did not respond to the invitation large
 (+0.719) large
 (+0.544) medium
 (+0.344) large
 (+0.671) 13 Inviting more reviewers increases the chance of having an invited reviewer who did not respond The approach of inviting more reviewers is less effective when the number of invited reviewers increases
  14. 14 Several technical factors share a link to review participation

    in MCR “Patch properties metrics share 
 a link to poor review participation” [Thongtanunam et al. EMSE 2016] Patch Characteristics Reviewer Experience “Patch author should invite reviewers 
 based on their past experience” [Balachandran ICSE 2013] [Zanjani et al. TSE 2016]
  15. 15 Little is known whether human factors have an impact

    on review participation Uploads a patch
 (code changes) A code review tool (e.g., Gerrit) Invites via
 code review tool Receives comments
 and revise the patch A clean patch 
 can be integrated
 into codebase Main repository Examine the patch,
 provide comments
 and review scores A patch author +1 +1 +2 Reviewers
  16. 16 For example, a reviewer with a large number of

    invitations is less likely to respond to a new invitation A patch author Uploads a patch
 (code changes) A code review tool (e.g., Gerrit) Invites via
 code review tool Reviewers
  17. 17 Investigating the impact of human factors Impact of human

    factors
 on prediction models Influence of human factors on reviewer participation decision
  18. 18 Extracting 12 metrics grouped along 3 dimensions Human Factors

    e.g., review participation rate, workload, familiarity Reviewer Experience e.g., reviewer code authoring exp. Patch Characteristics e.g., patch author code authoring exp., patch size
  19. 19 To investigate the impact of human factors on prediction

    models, we construct 2 models Human Factors Our proposed model Patch Characteristics Reviewer Experience Baseline model Patch Characteristics Reviewer Experience
  20. The proposed models that include human factors outperform the baseline

    models AUC value 0.82-0.89 0.62-0.78 Brier score
 (lower better) 0.06-0.13 0.09-0.19 Precision 0.68-0.78 0.60-0.77 Recall 0.24-0.73 0.01-0.67 F-measure 0.35-0.75 0.02-0.64 Proposed Baseline
  21. 21 Impact of human factors
 on prediction models Influence of

    human factors on reviewer participation decision Including human factors into the prediction models of review participation decision increases the performance of the models Investigating the impact of human factors
  22. 22 Human factors and review experience metrics contribute large amount

    of explanatory power Human Factors Reviewer Experience Patch Characteristics 1st 2nd 1st 1st 1st 3rd 4th 3rd 6th Review Participation Rate 2nd 2nd 2nd Review Participation Rate Review Participation Rate Review Participation Rate Reviewer Code Authoring Experience Reviewer Reviewing Experience Reviewer Code Authoring Experience Reviewer Code Authoring Experience Patch Author Reviewing Experience Patch Author Reviewing Experience Patch Author Reviewing Experience Patch Author Reviewing Experience
  23. 23 Influence of human factors on reviewer participation decision Human

    factors play an important role on reviewer participation decision in addition to reviewer experience Impact of human factors
 on prediction models Including human factors into the prediction models of review participation decision increases the performance of the models Investigating the impact of human factors
  24. 24 While reviewer experience is well-known among practitioners, only few

    are aware of human factors of the patch authors invite reviewers 
 based on reviewer experience 72% of the patch authors invite reviewers 
 based on responsiveness of the reviewers 24% Example of online survey results
  25. Prediction models can help developers making a better decision at

    inviting reviewers A developer Reviewers List potential reviewers Invites only reviewers with
 high participation likelihood Prediction model Low participation likelihood Additionally reduces up to 31% of the unnecessary review invitations i.e., reviewers who may not 
 respond to the invitation
  26. 26 The Impact of Human Factors on the Participation Decision

    of Reviewers in Modern Code Review Human factors should be considered in addition to reviewer experience and patch characteristics when inviting reviewers. Thus, leading to a better inviting decision and reviewer participation. S. Ruangwan, P. Thongtanunam, A. Ihara, and K. Matsumoto (2019) The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code Review. Empirical Software Engineering (EMSE) 24(2): 973-1016 Shade Ruangwan shade.ruangwan.sj1@is.naist.jp www.ruangwan.com
  27. 27