$30 off During Our Annual Pro Sale. View Details »

The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code Review

sruangwan
May 31, 2019

The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code Review

Modern Code Review (MCR) plays a key role in software quality practices. In MCR process, a new patch (i.e., a set of code changes) is encouraged to be examined by reviewers in order to identify weaknesses in source code prior to an integration into main software repositories. To mitigate the risk of having future defects, prior work suggests that MCR should be performed with sufficient review participation. Indeed, recent work shows that a low number of participated reviewers is associated with poor software quality. However, there is a likely case that a new patch still suffers from poor review participation even though reviewers were invited. Hence, in this paper, we set out to investigate the factors that are associated with the participation decision of an invited reviewer. Through a case study of 230,090 patches spread across the Android, LibreOffice, OpenStack and Qt systems, we find that (1) 16%-66% of patches have at least one invited reviewer who did not respond to the review invitation; (2) human factors play an important role in predicting whether or not an invited reviewer will participate in a review; (3) a review participation rate of an invited reviewers and code authoring experience of an invited reviewer are highly associated with the participation decision of an invited reviewer. These results can help practitioners better understand about how human factors associate with the participation decision of reviewers and serve as guidelines for inviting reviewers, leading to a better inviting decision and a better reviewer participation.

sruangwan

May 31, 2019
Tweet

Other Decks in Research

Transcript

  1. The Impact of Human Factors on the
    Participation Decision of Reviewers in
    Modern Code Review
    Journal-First Paper (Empirical Software Engineering)
    Shade
    Ruangwan
    Patanamon

    Thongtanunam
    Akinori

    Ihara
    Kenichi

    Matsumoto
    PhD Student

    View Slide

  2. Code review plays a key role in software
    development to improve software quality
    Remove software defects
    Improve readability 

    and maintainability
    2
    Modern Code Review (MCR)
    a tool-based code review
    Promote geographically-distributed

    collaboration practices
    Promote knowledge transfer

    with traceable process

    View Slide

  3. 3
    Before a new patch can be integrated into codebase,
    it has to undergone code review
    A patch author
    A code review tool
    (e.g., Gerrit)
    Uploads a patch

    (code changes)

    View Slide

  4. 4
    A patch author
    A code review tool
    (e.g., Gerrit)
    Reviewers
    Uploads a patch

    (code changes)
    Invites via

    code review tool
    Before a new patch can be integrated into codebase,
    it has to undergone code review

    View Slide

  5. 5
    A patch author
    A code review tool
    (e.g., Gerrit)
    Reviewers
    Uploads a patch

    (code changes)
    Invites via

    code review tool
    Examine the patch,

    provide comments

    and/or review scores
    Before a new patch can be integrated into codebase,
    it has to undergone code review

    View Slide

  6. 6
    A patch author
    Uploads a patch

    (code changes)
    A code review tool
    (e.g., Gerrit)
    Reviewers
    Invites via

    code review tool
    Receives comments

    and revise the patch
    Examine the patch,

    provide comments

    and/or review scores
    Before a new patch can be integrated into codebase,
    it has to undergone code review

    View Slide

  7. 7
    A patch author
    Uploads a patch

    (code changes)
    A code review tool
    (e.g., Gerrit)
    Reviewers
    Invites via

    code review tool
    Receives comments

    and revise the patch
    A clean patch 

    can be integrated

    into codebase
    Main repository
    +1
    +1
    +2
    Examine the patch,

    provide comments

    and/or review scores
    Before a new patch can be integrated into codebase,
    it has to undergone code review

    View Slide

  8. A patch author
    Uploads a patch

    (code changes)
    Invites via

    code review tool
    Receives comments

    and revise the patch
    8
    The number of reviewers has a large impact on
    software quality
    [McIntosh et al. MSR 2014]
    [Thongtanunam et al. MSR 2015]
    [Bavota and Russo ICSME 2015]
    Main repository
    A code review tool
    (e.g., Gerrit)
    A buggy patch

    maybe integrated

    into codebase
    Fewer Reviewers
    Examine the patch,

    provide comments

    and/or review scores
    ?

    View Slide

  9. 9
    A patch author invites many reviewers to increase
    review effectiveness
    A patch author
    Uploads a patch

    (code changes)
    A code review tool
    (e.g., Gerrit)
    Reviewers
    Invites via

    code review tool

    View Slide

  10. 10
    However, reviewers may not respond to the invitation
    due to the nature of OSS development processes
    A patch author
    Uploads a patch

    (code changes)
    A code review tool
    (e.g., Gerrit)
    Reviewers
    Invites via

    code review tool

    View Slide

  11. 11
    We find that many patches have an invited reviewer
    who did not respond to the invitation
    Studied systems
    #Patches 36,771 18,716 108,788 65,815
    Patches with at least one
    reviewer who did not respond
    to the invitation
    66% 16% 23% 47%
    Correlation between #invited
    reviewers and #reviewers who
    did not respond to the
    invitation
    large
    (+0.719)
    large
    (+0.544)
    medium
    (+0.344)
    large
    (+0.671)

    View Slide

  12. Studied systems
    #Patches 36,771 18,716 108,788 65,815
    Patches with at least one
    reviewer who did not respond
    to the invitation
    66% 16% 23% 47%
    Correlation between #invited
    reviewers and #reviewers who
    did not respond to the
    invitation
    large

    (+0.719)
    large

    (+0.544)
    medium

    (+0.344)
    large

    (+0.671)
    12
    Inviting more reviewers increases the chance of
    having an invited reviewer who did not respond

    View Slide

  13. Studied systems
    #Patches 36,771 18,716 108,788 65,815
    Patches with at least one
    reviewer who did not respond
    to the invitation
    66% 16% 23% 47%
    Correlation between #invited
    reviewers and #reviewers who
    did not respond to the
    invitation
    large

    (+0.719)
    large

    (+0.544)
    medium

    (+0.344)
    large

    (+0.671)
    13
    Inviting more reviewers increases the chance of
    having an invited reviewer who did not respond
    The approach of inviting more reviewers is less effective
    when the number of invited reviewers increases

    View Slide

  14. 14
    Several technical factors share a link to review
    participation in MCR
    “Patch properties metrics share 

    a link to poor review participation”
    [Thongtanunam et al. EMSE 2016]
    Patch
    Characteristics
    Reviewer
    Experience
    “Patch author should invite reviewers 

    based on their past experience”
    [Balachandran ICSE 2013]
    [Zanjani et al. TSE 2016]

    View Slide

  15. 15
    Little is known whether human factors have an
    impact on review participation
    Uploads a patch

    (code changes)
    A code review tool
    (e.g., Gerrit)
    Invites via

    code review tool
    Receives comments

    and revise the patch
    A clean patch 

    can be integrated

    into codebase
    Main repository
    Examine the patch,

    provide comments

    and review scores
    A patch author
    +1
    +1
    +2
    Reviewers

    View Slide

  16. 16
    For example, a reviewer with a large number of
    invitations is less likely to respond to a new invitation
    A patch author
    Uploads a patch

    (code changes)
    A code review tool
    (e.g., Gerrit)
    Invites via

    code review tool
    Reviewers

    View Slide

  17. 17
    Investigating the impact of human factors
    Impact of human factors

    on prediction models
    Influence of human factors
    on reviewer participation
    decision

    View Slide

  18. 18
    Extracting 12 metrics grouped along 3 dimensions
    Human Factors
    e.g., review participation rate,
    workload, familiarity
    Reviewer Experience
    e.g., reviewer code
    authoring exp.
    Patch Characteristics
    e.g., patch author code
    authoring exp., patch size

    View Slide

  19. 19
    To investigate the impact of human factors on
    prediction models, we construct 2 models
    Human
    Factors
    Our proposed model
    Patch
    Characteristics
    Reviewer
    Experience
    Baseline model
    Patch
    Characteristics
    Reviewer
    Experience

    View Slide

  20. The proposed models that include human factors
    outperform the baseline models
    AUC value 0.82-0.89 0.62-0.78
    Brier score

    (lower better)
    0.06-0.13 0.09-0.19
    Precision 0.68-0.78 0.60-0.77
    Recall 0.24-0.73 0.01-0.67
    F-measure 0.35-0.75 0.02-0.64
    Proposed Baseline

    View Slide

  21. 21
    Impact of human factors

    on prediction models
    Influence of human factors
    on reviewer participation
    decision
    Including human factors into
    the prediction models of review
    participation decision increases
    the performance of the models
    Investigating the impact of human factors

    View Slide

  22. 22
    Human factors and review experience metrics
    contribute large amount of explanatory power
    Human Factors
    Reviewer
    Experience
    Patch
    Characteristics
    1st 2nd 1st 1st
    1st
    3rd 4th 3rd 6th
    Review
    Participation
    Rate
    2nd 2nd 2nd
    Review
    Participation
    Rate
    Review
    Participation
    Rate
    Review
    Participation
    Rate
    Reviewer
    Code Authoring
    Experience
    Reviewer
    Reviewing
    Experience
    Reviewer
    Code Authoring
    Experience
    Reviewer
    Code Authoring
    Experience
    Patch Author
    Reviewing
    Experience
    Patch Author
    Reviewing
    Experience
    Patch Author
    Reviewing
    Experience
    Patch Author
    Reviewing
    Experience

    View Slide

  23. 23
    Influence of human factors
    on reviewer participation
    decision
    Human factors play an
    important role on reviewer
    participation decision in addition
    to reviewer experience
    Impact of human factors

    on prediction models
    Including human factors into
    the prediction models of review
    participation decision increases
    the performance of the models
    Investigating the impact of human factors

    View Slide

  24. 24
    While reviewer experience is well-known among
    practitioners, only few are aware of human factors
    of the patch authors invite reviewers 

    based on reviewer experience
    72%
    of the patch authors invite reviewers 

    based on responsiveness of the reviewers
    24%
    Example of online survey results

    View Slide

  25. Prediction models can help developers making a
    better decision at inviting reviewers
    A developer Reviewers
    List potential reviewers
    Invites only reviewers with

    high participation likelihood
    Prediction model
    Low participation likelihood
    Additionally reduces up to 31% of the unnecessary review invitations
    i.e., reviewers who may not 

    respond to the invitation

    View Slide

  26. 26
    The Impact of Human Factors on the Participation
    Decision of Reviewers in Modern Code Review
    Human factors should be considered in addition to reviewer
    experience and patch characteristics when inviting reviewers.
    Thus, leading to a better inviting decision and reviewer
    participation.
    S. Ruangwan, P. Thongtanunam, A. Ihara, and K. Matsumoto (2019) The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code
    Review. Empirical Software Engineering (EMSE) 24(2): 973-1016
    Shade Ruangwan
    [email protected]
    www.ruangwan.com

    View Slide

  27. 27

    View Slide