The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code Review

Modern Code Review (MCR) plays a key role in software quality practices. In MCR process, a new patch (i.e., a set of code changes) is encouraged to be examined by reviewers in order to identify weaknesses in source code prior to an integration into main software repositories. To mitigate the risk of having future defects, prior work suggests that MCR should be performed with sufficient review participation. Indeed, recent work shows that a low number of participated reviewers is associated with poor software quality. However, there is a likely case that a new patch still suffers from poor review participation even though reviewers were invited. Hence, in this paper, we set out to investigate the factors that are associated with the participation decision of an invited reviewer. Through a case study of 230,090 patches spread across the Android, LibreOffice, OpenStack and Qt systems, we find that (1) 16%-66% of patches have at least one invited reviewer who did not respond to the review invitation; (2) human factors play an important role in predicting whether or not an invited reviewer will participate in a review; (3) a review participation rate of an invited reviewers and code authoring experience of an invited reviewer are highly associated with the participation decision of an invited reviewer. These results can help practitioners better understand about how human factors associate with the participation decision of reviewers and serve as guidelines for inviting reviewers, leading to a better inviting decision and a better reviewer participation.

Transcript

1. The Impact of Human Factors on the
   Participation Decision of Reviewers in
   Modern Code Review
   Journal-First Paper (Empirical Software Engineering)
   Shade
   Ruangwan
   Patanamon

   Thongtanunam
   Akinori

   Ihara
   Kenichi
   
   Matsumoto
   PhD Student
   

   View Slide

2. Code review plays a key role in software
   development to improve software quality
   Remove software defects
   Improve readability 

   and maintainability
   2
   Modern Code Review (MCR)
   a tool-based code review
   Promote geographically-distributed

   collaboration practices
   Promote knowledge transfer

   with traceable process
   

   View Slide

3. 3
   Before a new patch can be integrated into codebase,
   it has to undergone code review
   A patch author
   A code review tool
   (e.g., Gerrit)
   Uploads a patch

   (code changes)
   

   View Slide

4. 4
   A patch author
   A code review tool
   (e.g., Gerrit)
   Reviewers
   Uploads a patch

   (code changes)
   Invites via

   code review tool
   Before a new patch can be integrated into codebase,
   it has to undergone code review
   

   View Slide

5. 5
   A patch author
   A code review tool
   (e.g., Gerrit)
   Reviewers
   Uploads a patch

   (code changes)
   Invites via

   code review tool
   Examine the patch,

   provide comments

   and/or review scores
   Before a new patch can be integrated into codebase,
   it has to undergone code review
   

   View Slide

6. 6
   A patch author
   Uploads a patch

   (code changes)
   A code review tool
   (e.g., Gerrit)
   Reviewers
   Invites via

   code review tool
   Receives comments

   and revise the patch
   Examine the patch,

   provide comments

   and/or review scores
   Before a new patch can be integrated into codebase,
   it has to undergone code review
   

   View Slide

7. 7
   A patch author
   Uploads a patch

   (code changes)
   A code review tool
   (e.g., Gerrit)
   Reviewers
   Invites via

   code review tool
   Receives comments

   and revise the patch
   A clean patch 

   can be integrated

   into codebase
   Main repository
   +1
   +1
   +2
   Examine the patch,

   provide comments

   and/or review scores
   Before a new patch can be integrated into codebase,
   it has to undergone code review
   

   View Slide

8. A patch author
   Uploads a patch

   (code changes)
   Invites via

   code review tool
   Receives comments

   and revise the patch
   8
   The number of reviewers has a large impact on
   software quality
   [McIntosh et al. MSR 2014]
   [Thongtanunam et al. MSR 2015]
   [Bavota and Russo ICSME 2015]
   Main repository
   A code review tool
   (e.g., Gerrit)
   A buggy patch

   maybe integrated

   into codebase
   Fewer Reviewers
   Examine the patch,

   provide comments

   and/or review scores
   ?
   

   View Slide

9. 9
   A patch author invites many reviewers to increase
   review effectiveness
   A patch author
   Uploads a patch

   (code changes)
   A code review tool
   (e.g., Gerrit)
   Reviewers
   Invites via

   code review tool
   

   View Slide

10. 10
    However, reviewers may not respond to the invitation
    due to the nature of OSS development processes
    A patch author
    Uploads a patch

    (code changes)
    A code review tool
    (e.g., Gerrit)
    Reviewers
    Invites via

    code review tool
    

    View Slide

11. 11
    We find that many patches have an invited reviewer
    who did not respond to the invitation
    Studied systems
    #Patches 36,771 18,716 108,788 65,815
    Patches with at least one
    reviewer who did not respond
    to the invitation
    66% 16% 23% 47%
    Correlation between #invited
    reviewers and #reviewers who
    did not respond to the
    invitation
    large
    (+0.719)
    large
    (+0.544)
    medium
    (+0.344)
    large
    (+0.671)
    

    View Slide

12. Studied systems
    #Patches 36,771 18,716 108,788 65,815
    Patches with at least one
    reviewer who did not respond
    to the invitation
    66% 16% 23% 47%
    Correlation between #invited
    reviewers and #reviewers who
    did not respond to the
    invitation
    large

    (+0.719)
    large

    (+0.544)
    medium

    (+0.344)
    large

    (+0.671)
    12
    Inviting more reviewers increases the chance of
    having an invited reviewer who did not respond
    

    View Slide

13. Studied systems
    #Patches 36,771 18,716 108,788 65,815
    Patches with at least one
    reviewer who did not respond
    to the invitation
    66% 16% 23% 47%
    Correlation between #invited
    reviewers and #reviewers who
    did not respond to the
    invitation
    large

    (+0.719)
    large

    (+0.544)
    medium

    (+0.344)
    large

    (+0.671)
    13
    Inviting more reviewers increases the chance of
    having an invited reviewer who did not respond
    The approach of inviting more reviewers is less effective
    when the number of invited reviewers increases
    

    View Slide

14. 14
    Several technical factors share a link to review
    participation in MCR
    “Patch properties metrics share 

    a link to poor review participation”
    [Thongtanunam et al. EMSE 2016]
    Patch
    Characteristics
    Reviewer
    Experience
    “Patch author should invite reviewers 

    based on their past experience”
    [Balachandran ICSE 2013]
    [Zanjani et al. TSE 2016]
    

    View Slide

15. 15
    Little is known whether human factors have an
    impact on review participation
    Uploads a patch

    (code changes)
    A code review tool
    (e.g., Gerrit)
    Invites via

    code review tool
    Receives comments

    and revise the patch
    A clean patch 

    can be integrated

    into codebase
    Main repository
    Examine the patch,

    provide comments

    and review scores
    A patch author
    +1
    +1
    +2
    Reviewers
    

    View Slide

16. 16
    For example, a reviewer with a large number of
    invitations is less likely to respond to a new invitation
    A patch author
    Uploads a patch

    (code changes)
    A code review tool
    (e.g., Gerrit)
    Invites via

    code review tool
    Reviewers
    

    View Slide

17. 17
    Investigating the impact of human factors
    Impact of human factors

    on prediction models
    Influence of human factors
    on reviewer participation
    decision
    

    View Slide

18. 18
    Extracting 12 metrics grouped along 3 dimensions
    Human Factors
    e.g., review participation rate,
    workload, familiarity
    Reviewer Experience
    e.g., reviewer code
    authoring exp.
    Patch Characteristics
    e.g., patch author code
    authoring exp., patch size
    

    View Slide

19. 19
    To investigate the impact of human factors on
    prediction models, we construct 2 models
    Human
    Factors
    Our proposed model
    Patch
    Characteristics
    Reviewer
    Experience
    Baseline model
    Patch
    Characteristics
    Reviewer
    Experience
    

    View Slide

20. The proposed models that include human factors
    outperform the baseline models
    AUC value 0.82-0.89 0.62-0.78
    Brier score

    (lower better)
    0.06-0.13 0.09-0.19
    Precision 0.68-0.78 0.60-0.77
    Recall 0.24-0.73 0.01-0.67
    F-measure 0.35-0.75 0.02-0.64
    Proposed Baseline
    

    View Slide

21. 21
    Impact of human factors

    on prediction models
    Influence of human factors
    on reviewer participation
    decision
    Including human factors into
    the prediction models of review
    participation decision increases
    the performance of the models
    Investigating the impact of human factors
    

    View Slide

22. 22
    Human factors and review experience metrics
    contribute large amount of explanatory power
    Human Factors
    Reviewer
    Experience
    Patch
    Characteristics
    1st 2nd 1st 1st
    1st
    3rd 4th 3rd 6th
    Review
    Participation
    Rate
    2nd 2nd 2nd
    Review
    Participation
    Rate
    Review
    Participation
    Rate
    Review
    Participation
    Rate
    Reviewer
    Code Authoring
    Experience
    Reviewer
    Reviewing
    Experience
    Reviewer
    Code Authoring
    Experience
    Reviewer
    Code Authoring
    Experience
    Patch Author
    Reviewing
    Experience
    Patch Author
    Reviewing
    Experience
    Patch Author
    Reviewing
    Experience
    Patch Author
    Reviewing
    Experience
    

    View Slide

23. 23
    Influence of human factors
    on reviewer participation
    decision
    Human factors play an
    important role on reviewer
    participation decision in addition
    to reviewer experience
    Impact of human factors

    on prediction models
    Including human factors into
    the prediction models of review
    participation decision increases
    the performance of the models
    Investigating the impact of human factors
    

    View Slide

24. 24
    While reviewer experience is well-known among
    practitioners, only few are aware of human factors
    of the patch authors invite reviewers 

    based on reviewer experience
    72%
    of the patch authors invite reviewers 

    based on responsiveness of the reviewers
    24%
    Example of online survey results
    

    View Slide

25. Prediction models can help developers making a
    better decision at inviting reviewers
    A developer Reviewers
    List potential reviewers
    Invites only reviewers with

    high participation likelihood
    Prediction model
    Low participation likelihood
    Additionally reduces up to 31% of the unnecessary review invitations
    i.e., reviewers who may not 

    respond to the invitation
    

    View Slide

26. 26
    The Impact of Human Factors on the Participation
    Decision of Reviewers in Modern Code Review
    Human factors should be considered in addition to reviewer
    experience and patch characteristics when inviting reviewers.
    Thus, leading to a better inviting decision and reviewer
    participation.
    S. Ruangwan, P. Thongtanunam, A. Ihara, and K. Matsumoto (2019) The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code
    Review. Empirical Software Engineering (EMSE) 24(2): 973-1016
    Shade Ruangwan
    shade.ruangwan.sj1@is.naist.jp
    www.ruangwan.com
    

    View Slide

27. View Slide