The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code Review

Modern Code Review (MCR) plays a key role in software quality practices. In MCR process, a new patch (i.e., a set of code changes) is encouraged to be examined by reviewers in order to identify weaknesses in source code prior to an integration into main software repositories. To mitigate the risk of having future defects, prior work suggests that MCR should be performed with sufficient review participation. Indeed, recent work shows that a low number of participated reviewers is associated with poor software quality. However, there is a likely case that a new patch still suffers from poor review participation even though reviewers were invited. Hence, in this paper, we set out to investigate the factors that are associated with the participation decision of an invited reviewer. Through a case study of 230,090 patches spread across the Android, LibreOffice, OpenStack and Qt systems, we find that (1) 16%-66% of patches have at least one invited reviewer who did not respond to the review invitation; (2) human factors play an important role in predicting whether or not an invited reviewer will participate in a review; (3) a review participation rate of an invited reviewers and code authoring experience of an invited reviewer are highly associated with the participation decision of an invited reviewer. These results can help practitioners better understand about how human factors associate with the participation decision of reviewers and serve as guidelines for inviting reviewers, leading to a better inviting decision and a better reviewer participation.

Transcript

1. The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code Review Journal-First Paper (Empirical Software Engineering) Shade Ruangwan Patanamon
 Thongtanunam Akinori
 Ihara Kenichi Matsumoto PhD Student
2. Code review plays a key role in software development to improve software quality Remove software defects Improve readability 
 and maintainability 2 Modern Code Review (MCR) a tool-based code review Promote geographically-distributed
 collaboration practices Promote knowledge transfer
 with traceable process
3. 3 Before a new patch can be integrated into codebase, it has to undergone code review A patch author A code review tool (e.g., Gerrit) Uploads a patch
 (code changes)
4. 4 A patch author A code review tool (e.g., Gerrit) Reviewers Uploads a patch
 (code changes) Invites via
 code review tool Before a new patch can be integrated into codebase, it has to undergone code review
5. 5 A patch author A code review tool (e.g., Gerrit) Reviewers Uploads a patch
 (code changes) Invites via
 code review tool Examine the patch,
 provide comments
 and/or review scores Before a new patch can be integrated into codebase, it has to undergone code review
6. 6 A patch author Uploads a patch
 (code changes) A code review tool (e.g., Gerrit) Reviewers Invites via
 code review tool Receives comments
 and revise the patch Examine the patch,
 provide comments
 and/or review scores Before a new patch can be integrated into codebase, it has to undergone code review
7. 7 A patch author Uploads a patch
 (code changes) A code review tool (e.g., Gerrit) Reviewers Invites via
 code review tool Receives comments
 and revise the patch A clean patch 
 can be integrated
 into codebase Main repository +1 +1 +2 Examine the patch,
 provide comments
 and/or review scores Before a new patch can be integrated into codebase, it has to undergone code review
8. A patch author Uploads a patch
 (code changes) Invites via
 code review tool Receives comments
 and revise the patch 8 The number of reviewers has a large impact on software quality [McIntosh et al. MSR 2014] [Thongtanunam et al. MSR 2015] [Bavota and Russo ICSME 2015] Main repository A code review tool (e.g., Gerrit) A buggy patch
 maybe integrated
 into codebase Fewer Reviewers Examine the patch,
 provide comments
 and/or review scores ?
9. 9 A patch author invites many reviewers to increase review effectiveness A patch author Uploads a patch
 (code changes) A code review tool (e.g., Gerrit) Reviewers Invites via
 code review tool
10. 10 However, reviewers may not respond to the invitation due to the nature of OSS development processes A patch author Uploads a patch
 (code changes) A code review tool (e.g., Gerrit) Reviewers Invites via
 code review tool
11. 11 We find that many patches have an invited reviewer who did not respond to the invitation Studied systems #Patches 36,771 18,716 108,788 65,815 Patches with at least one reviewer who did not respond to the invitation 66% 16% 23% 47% Correlation between #invited reviewers and #reviewers who did not respond to the invitation large (+0.719) large (+0.544) medium (+0.344) large (+0.671)
12. Studied systems #Patches 36,771 18,716 108,788 65,815 Patches with at least one reviewer who did not respond to the invitation 66% 16% 23% 47% Correlation between #invited reviewers and #reviewers who did not respond to the invitation large
 (+0.719) large
 (+0.544) medium
 (+0.344) large
 (+0.671) 12 Inviting more reviewers increases the chance of having an invited reviewer who did not respond
13. Studied systems #Patches 36,771 18,716 108,788 65,815 Patches with at least one reviewer who did not respond to the invitation 66% 16% 23% 47% Correlation between #invited reviewers and #reviewers who did not respond to the invitation large
 (+0.719) large
 (+0.544) medium
 (+0.344) large
 (+0.671) 13 Inviting more reviewers increases the chance of having an invited reviewer who did not respond The approach of inviting more reviewers is less effective when the number of invited reviewers increases
14. 14 Several technical factors share a link to review participation in MCR “Patch properties metrics share 
 a link to poor review participation” [Thongtanunam et al. EMSE 2016] Patch Characteristics Reviewer Experience “Patch author should invite reviewers 
 based on their past experience” [Balachandran ICSE 2013] [Zanjani et al. TSE 2016]
15. 15 Little is known whether human factors have an impact on review participation Uploads a patch
 (code changes) A code review tool (e.g., Gerrit) Invites via
 code review tool Receives comments
 and revise the patch A clean patch 
 can be integrated
 into codebase Main repository Examine the patch,
 provide comments
 and review scores A patch author +1 +1 +2 Reviewers
16. 16 For example, a reviewer with a large number of invitations is less likely to respond to a new invitation A patch author Uploads a patch
 (code changes) A code review tool (e.g., Gerrit) Invites via
 code review tool Reviewers
17. 17 Investigating the impact of human factors Impact of human factors
 on prediction models Influence of human factors on reviewer participation decision
18. 18 Extracting 12 metrics grouped along 3 dimensions Human Factors e.g., review participation rate, workload, familiarity Reviewer Experience e.g., reviewer code authoring exp. Patch Characteristics e.g., patch author code authoring exp., patch size
19. 19 To investigate the impact of human factors on prediction models, we construct 2 models Human Factors Our proposed model Patch Characteristics Reviewer Experience Baseline model Patch Characteristics Reviewer Experience
20. The proposed models that include human factors outperform the baseline models AUC value 0.82-0.89 0.62-0.78 Brier score
 (lower better) 0.06-0.13 0.09-0.19 Precision 0.68-0.78 0.60-0.77 Recall 0.24-0.73 0.01-0.67 F-measure 0.35-0.75 0.02-0.64 Proposed Baseline
21. 21 Impact of human factors
 on prediction models Influence of human factors on reviewer participation decision Including human factors into the prediction models of review participation decision increases the performance of the models Investigating the impact of human factors
22. 22 Human factors and review experience metrics contribute large amount of explanatory power Human Factors Reviewer Experience Patch Characteristics 1st 2nd 1st 1st 1st 3rd 4th 3rd 6th Review Participation Rate 2nd 2nd 2nd Review Participation Rate Review Participation Rate Review Participation Rate Reviewer Code Authoring Experience Reviewer Reviewing Experience Reviewer Code Authoring Experience Reviewer Code Authoring Experience Patch Author Reviewing Experience Patch Author Reviewing Experience Patch Author Reviewing Experience Patch Author Reviewing Experience
23. 23 Influence of human factors on reviewer participation decision Human factors play an important role on reviewer participation decision in addition to reviewer experience Impact of human factors
 on prediction models Including human factors into the prediction models of review participation decision increases the performance of the models Investigating the impact of human factors
24. 24 While reviewer experience is well-known among practitioners, only few are aware of human factors of the patch authors invite reviewers 
 based on reviewer experience 72% of the patch authors invite reviewers 
 based on responsiveness of the reviewers 24% Example of online survey results
25. Prediction models can help developers making a better decision at inviting reviewers A developer Reviewers List potential reviewers Invites only reviewers with
 high participation likelihood Prediction model Low participation likelihood Additionally reduces up to 31% of the unnecessary review invitations i.e., reviewers who may not 
 respond to the invitation
26. 26 The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code Review Human factors should be considered in addition to reviewer experience and patch characteristics when inviting reviewers. Thus, leading to a better inviting decision and reviewer participation. S. Ruangwan, P. Thongtanunam, A. Ihara, and K. Matsumoto (2019) The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code Review. Empirical Software Engineering (EMSE) 24(2): 973-1016 Shade Ruangwan shade.ruangwan.sj1@is.naist.jp www.ruangwan.com