The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code Review Journal-First Paper (Empirical Software Engineering) Shade Ruangwan Patanamon
 Thongtanunam Akinori
 Ihara Kenichi Matsumoto PhD Student 

Code review plays a key role in software development to improve software quality Remove software defects Improve readability 
 and maintainability 2 Modern Code Review (MCR) a tool-based code review Promote geographically-distributed
 collaboration practices Promote knowledge transfer
 with traceable process 

3 Before a new patch can be integrated into codebase, it has to undergone code review A patch author A code review tool (e.g., Gerrit) Uploads a patch
 (code changes) 

4 A patch author A code review tool (e.g., Gerrit) Reviewers Uploads a patch
 (code changes) Invites via
 code review tool Before a new patch can be integrated into codebase, it has to undergone code review 

5 A patch author A code review tool (e.g., Gerrit) Reviewers Uploads a patch
 (code changes) Invites via
 code review tool Examine the patch,
 provide comments
 and/or review scores Before a new patch can be integrated into codebase, it has to undergone code review 

6 A patch author Uploads a patch
 (code changes) A code review tool (e.g., Gerrit) Reviewers Invites via
 code review tool Receives comments
 and revise the patch Examine the patch,
 provide comments
 and/or review scores Before a new patch can be integrated into codebase, it has to undergone code review 

7 A patch author Uploads a patch
 (code changes) A code review tool (e.g., Gerrit) Reviewers Invites via
 code review tool Receives comments
 and revise the patch A clean patch 
 can be integrated
 into codebase Main repository +1 +1 +2 Examine the patch,
 provide comments
 and/or review scores Before a new patch can be integrated into codebase, it has to undergone code review 

A patch author Uploads a patch
 (code changes) Invites via
 code review tool Receives comments
 and revise the patch 8 The number of reviewers has a large impact on software quality [McIntosh et al. MSR 2014] [Thongtanunam et al. MSR 2015] [Bavota and Russo ICSME 2015] Main repository A code review tool (e.g., Gerrit) A buggy patch
 maybe integrated
 into codebase Fewer Reviewers Examine the patch,
 provide comments
 and/or review scores ? 

9 A patch author invites many reviewers to increase review effectiveness A patch author Uploads a patch
 (code changes) A code review tool (e.g., Gerrit) Reviewers Invites via
 code review tool 

10 However, reviewers may not respond to the invitation due to the nature of OSS development processes A patch author Uploads a patch
 (code changes) A code review tool (e.g., Gerrit) Reviewers Invites via
 code review tool 

11 We find that many patches have an invited reviewer who did not respond to the invitation Studied systems #Patches 36,771 18,716 108,788 65,815 Patches with at least one reviewer who did not respond to the invitation 66% 16% 23% 47% Correlation between #invited reviewers and #reviewers who did not respond to the invitation large (+0.719) large (+0.544) medium (+0.344) large (+0.671) 

Studied systems #Patches 36,771 18,716 108,788 65,815 Patches with at least one reviewer who did not respond to the invitation 66% 16% 23% 47% Correlation between #invited reviewers and #reviewers who did not respond to the invitation large
 (+0.719) large
 (+0.544) medium
 (+0.344) large
 (+0.671) 12 Inviting more reviewers increases the chance of having an invited reviewer who did not respond 

Studied systems #Patches 36,771 18,716 108,788 65,815 Patches with at least one reviewer who did not respond to the invitation 66% 16% 23% 47% Correlation between #invited reviewers and #reviewers who did not respond to the invitation large
 (+0.719) large
 (+0.544) medium
 (+0.344) large
 (+0.671) 13 Inviting more reviewers increases the chance of having an invited reviewer who did not respond The approach of inviting more reviewers is less effective when the number of invited reviewers increases 

14 Several technical factors share a link to review participation in MCR “Patch properties metrics share 
 a link to poor review participation” [Thongtanunam et al. EMSE 2016] Patch Characteristics Reviewer Experience “Patch author should invite reviewers 
 based on their past experience” [Balachandran ICSE 2013] [Zanjani et al. TSE 2016] 

15 Little is known whether human factors have an impact on review participation Uploads a patch
 (code changes) A code review tool (e.g., Gerrit) Invites via
 code review tool Receives comments
 and revise the patch A clean patch 
 can be integrated
 into codebase Main repository Examine the patch,
 provide comments
 and review scores A patch author +1 +1 +2 Reviewers 

16 For example, a reviewer with a large number of invitations is less likely to respond to a new invitation A patch author Uploads a patch
 (code changes) A code review tool (e.g., Gerrit) Invites via
 code review tool Reviewers 

17 Investigating the impact of human factors Impact of human factors
 on prediction models Influence of human factors on reviewer participation decision 

18 Extracting 12 metrics grouped along 3 dimensions Human Factors e.g., review participation rate, workload, familiarity Reviewer Experience e.g., reviewer code authoring exp. Patch Characteristics e.g., patch author code authoring exp., patch size 

19 To investigate the impact of human factors on prediction models, we construct 2 models Human Factors Our proposed model Patch Characteristics Reviewer Experience Baseline model Patch Characteristics Reviewer Experience 

The proposed models that include human factors outperform the baseline models AUC value 0.82-0.89 0.62-0.78 Brier score
 (lower better) 0.06-0.13 0.09-0.19 Precision 0.68-0.78 0.60-0.77 Recall 0.24-0.73 0.01-0.67 F-measure 0.35-0.75 0.02-0.64 Proposed Baseline 

21 Impact of human factors
 on prediction models Influence of human factors on reviewer participation decision Including human factors into the prediction models of review participation decision increases the performance of the models Investigating the impact of human factors 

22 Human factors and review experience metrics contribute large amount of explanatory power Human Factors Reviewer Experience Patch Characteristics 1st 2nd 1st 1st 1st 3rd 4th 3rd 6th Review Participation Rate 2nd 2nd 2nd Review Participation Rate Review Participation Rate Review Participation Rate Reviewer Code Authoring Experience Reviewer Reviewing Experience Reviewer Code Authoring Experience Reviewer Code Authoring Experience Patch Author Reviewing Experience Patch Author Reviewing Experience Patch Author Reviewing Experience Patch Author Reviewing Experience 

23 Influence of human factors on reviewer participation decision Human factors play an important role on reviewer participation decision in addition to reviewer experience Impact of human factors
 on prediction models Including human factors into the prediction models of review participation decision increases the performance of the models Investigating the impact of human factors 

24 While reviewer experience is well-known among practitioners, only few are aware of human factors of the patch authors invite reviewers 
 based on reviewer experience 72% of the patch authors invite reviewers 
 based on responsiveness of the reviewers 24% Example of online survey results 

Prediction models can help developers making a better decision at inviting reviewers A developer Reviewers List potential reviewers Invites only reviewers with
 high participation likelihood Prediction model Low participation likelihood Additionally reduces up to 31% of the unnecessary review invitations i.e., reviewers who may not 
 respond to the invitation 

26 The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code Review Human factors should be considered in addition to reviewer experience and patch characteristics when inviting reviewers. Thus, leading to a better inviting decision and reviewer participation. S. Ruangwan, P. Thongtanunam, A. Ihara, and K. Matsumoto (2019) The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code Review. Empirical Software Engineering (EMSE) 24(2): 973-1016 Shade Ruangwan [email protected] www.ruangwan.com 

27