Review Dynamics and Their Impact on Software Quality

Video available at: https://youtu.be/DK9GqrrQBRU
Venue: IEEE Transaction on Software Engineering
Abstract:
Code review is a crucial activity for ensuring the quality of software products. Unlike the traditional code review process of the past where reviewers independently examine software artifacts, contemporary code review processes allow teams to collaboratively examine and discuss proposed patches. While the visibility of reviewing activities including review discussions in a contemporary code review tends to increase developer collaboration and openness, little is known whether such visible information influences the evaluation decision of a reviewer or not (i.e., knowing others’ feedback about the patch before providing ones own feedback). Therefore, in this work, we set out to investigate the review dynamics, i.e., a practice of providing a vote to accept a proposed patch, in a code review process. To do so, we first characterize the review dynamics by examining the relationship between the evaluation decision of a reviewer and the visible information about a patch under review (e.g., comments and votes that are provided by prior co-reviewers). We then investigate the association between the characterized review dynamics and the defect-proneness of a patch. Through a case study of 83,750 patches of the OpenStack and Qt projects, we observe that the amount of feedback (either votes and comments of prior reviewers) and the co-working frequency of a reviewer with the patch author are highly associated with the likelihood that the reviewer will provide a positive vote to accept a proposed patch. Furthermore, we find that the proportion of reviewers who provided a vote consistent with prior reviewers is significantly associated with the defect-proneness of a patch. However, the associations of these review dynamics are not as strong as the confounding factors (i.e., patch characteristics and overall reviewing activities). Our observations shed light on the implicit influence of the visible information about a patch under review on the evaluation decision of a reviewer. Our findings suggest that the code reviewing policies that are mindful of these practices may help teams improve code review effectiveness. Nonetheless, such review dynamics should not be too concerning in terms of software quality.

Transcript

1. patanamon.t@unimelb.edu.au @patanamon
   Review Dynamics and Their Impact on
   Software Quality
   - Journal First Presentation -
   Patanamon (Pick)
   Thongtanunam Ahmed Hassan
   

   View Slide

2. Reviewer Reviewer
   Code Review: A manual activity to examine and
   improve the overall quality of a new code change
   Author
   A patch
   A code review tool
   (Ex. Gerrit)
   2
   Code review tools often provide
   a transparent environment
   One can see and learn from others’
   code and review comments
   
   [Bacchelli and Bird, ICSE2013]
   Expertise can be perceived from visible
   history of code changes and reviews
   
   [Dabbish et. al., CSCW2012; 

   Thongtanunam et. al., ICSE2016]
   

   View Slide

3. Relationship with
   the patch author
   Reviewers may have subconscious biases due to
   the visible information in a code review tool
   Reviewer
   Author A code review tool
   (Ex. Gerrit)
   A patch
   3
   

   View Slide

4. Relationship with
   the patch author
   Reviewers may have subconscious biases due to
   the visible information in a code review tool
   Reviewer
   Author A code review tool
   (Ex. Gerrit)
   A patch
   3
   Pick usually writes
   good code
   

   View Slide

5. Relationship with
   the patch author
   Reviewers may have subconscious biases due to
   the visible information in a code review tool
   Reviewer
   Author A code review tool
   (Ex. Gerrit)
   A patch
   3
   Pick usually writes
   good code
   Looks good to me
   

   View Slide

6. Relationship with
   the patch author
   Reviewers may have subconscious biases due to
   the visible information in a code review tool
   Reviewer
   Author A code review tool
   (Ex. Gerrit)
   A patch
   3
   Pick usually writes
   good code
   Looks good to me
   Ingroup-Outgroup Bias
   

   View Slide

7. Relationship with
   the patch author
   Reviewers may have subconscious biases due to
   the visible information in a code review tool
   Reviewer
   Author A code review tool
   (Ex. Gerrit)
   A patch
   3
   Pick usually writes
   good code
   Looks good to me
   “Patches from a trustworthy author are
   more likely to require less reviewing effort”
   [Bosu et. al., TSE2017]
   Developers with a weak relationship (e.g.,
   newcomers) face negative impressions
   
   [Lee et. al., ICSE2017; Steinmacher et.al., CSCW2015]
   Ingroup-Outgroup Bias
   

   View Slide

8. Reviewers may have subconscious biases due to
   the visible information in a code review tool
   Reviewer
   Author A code review tool
   (Ex. Gerrit)
   A patch
   4
   Status or Reputation
   

   View Slide

9. Reviewers may have subconscious biases due to
   the visible information in a code review tool
   Reviewer
   Author A code review tool
   (Ex. Gerrit)
   A patch
   4
   Status or Reputation
   

   View Slide

10. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Reviewer
    Author A code review tool
    (Ex. Gerrit)
    A patch
    4
    Ahmed is a core
    developer
    Looks good to me
    Status or Reputation
    

    View Slide

11. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Reviewer
    Author A code review tool
    (Ex. Gerrit)
    A patch
    4
    Ahmed is a core
    developer
    Looks good to me
    Status or Reputation
    A cognitive bias in terms of
    social hierarchy
    

    View Slide

12. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Reviewer
    Author A code review tool
    (Ex. Gerrit)
    A patch
    4
    Ahmed is a core
    developer
    Looks good to me
    Developers’ reputation can influence the
    review decision of their patches
    
    [Bosu & Carver, ESEM2014]
    Status or Reputation
    A cognitive bias in terms of
    social hierarchy
    

    View Slide

13. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Reviewer
    Author A code review tool
    (Ex. Gerrit)
    A patch
    4
    Ahmed is a core
    developer
    Looks good to me
    Developers’ reputation can influence the
    review decision of their patches
    
    [Bosu & Carver, ESEM2014]
    Feedback of outsiders tends to receive less
    attention than that of the core members
    
    [Rigby & Storey, ICSE2011]
    Status or Reputation
    A cognitive bias in terms of
    social hierarchy
    

    View Slide

14. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    5
    Visible
    Feedback
    A code review tool
    (Ex. Gerrit)
    Reviewer
    Reviewer
    

    View Slide

15. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    5
    Visible
    Feedback
    A code review tool
    (Ex. Gerrit)
    Reviewer
    Reviewer
    Reviewer
    

    View Slide

16. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    5
    Looks like everyone
    is happy with it
    Visible
    Feedback
    A code review tool
    (Ex. Gerrit)
    Reviewer
    Reviewer
    Reviewer
    

    View Slide

17. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    5
    Looks like everyone
    is happy with it
    Visible
    Feedback
    A code review tool
    (Ex. Gerrit)
    Reviewer
    Reviewer
    Reviewer
    

    View Slide

18. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    5
    Looks like everyone
    is happy with it
    Visible
    Feedback
    A code review tool
    (Ex. Gerrit)
    Reviewer
    Reviewer
    Reviewer
    Social Influence
    

    View Slide

19. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    5
    Looks like everyone
    is happy with it
    “Given that (the prior reviewer) already
    +1’d this patch as well, I will +2 it”
    OpenStack Review ID 1411
    “-1 to this patch. Backing up (the first
    reviewer)’s comment on.....”
    OpenStack Review ID 1257
    Visible
    Feedback
    A code review tool
    (Ex. Gerrit)
    Reviewer
    Reviewer
    Reviewer
    Social Influence
    

    View Slide

20. Investigating the signals of visible information that
    are associated with the review decision of a reviewer
    6
    STEP 1: Capture visible information
    about a patch under review Use 8 metrics grouped into 3 dimensions
    Relationship dimension
    e.g., the co-working frequency of
    a reviewer with the patch author
    Status dimension
    e.g., is a patch author a
    core member
    Prior feedback dimension
    e.g., %prior positive votes
    

    View Slide

21. Investigating the signals of visible information that
    are associated with the review decision of a reviewer
    7
    STEP 2: Analyze the relationship with
    the likelihood of giving a positive vote Use a mixed-effects logistic regression model
    IsPositiveVote ∼ x1 + x2 + ….. + xn
    + (1 | ReviewerId )
    8 Studied metrics
    Relationship Status Prior Feedback
    Confounding factors:

    Patch Characteristics as
    e.g., #Added Lines
    

    View Slide

22. In addition to patch characteristics, other visible
    information is associated with the review decision
    8
    25K Patches, 694Reviewers
    (09/2013 - 07/2019)
    57K Patches, 3.6K Reviewers
    (11/2011 - 07/2019)
    A Case study
    Relationship Status
    Prior Feedback
    Patch
    Characteristics
    Explanatory Power
    (Log-likelihood ratio test)
    Association Direction
    Higher %Reviewed
    past patches for
    the patch author
    More likely to
    Higher %Prior
    positive votes
    Lower %Prior
    comments
    More likely to
    More likely to
    

    View Slide

23. In addition to patch characteristics, other visible
    information is associated with the review decision
    9
    Relationship Status
    Prior Feedback
    Patch
    Characteristics
    Explanatory Power
    (Log-likelihood ratio test)
    Association Direction
    Higher %Reviewed
    past patches for
    the patch author
    More likely to
    Higher %Prior
    positive votes
    Lower %Prior
    comments
    More likely to
    More likely to
    We can characterize review dynamics 

    (i.e., the practices of making a review decision)
    Visible information has a stronger association with
    the review decision than patch characteristics
    

    View Slide

24. Investigating the risk that review dynamics can
    have on software quality
    10
    If such review dynamics happen often, will it pose a risk of having defects in a patch?
    

    View Slide

25. Investigating the risk that review dynamics can
    have on software quality
    10
    If such review dynamics happen often, will it pose a risk of having defects in a patch?
    STEP 1: Compute review 

    dynamics metrics
    Higher %Prior
    positive votes
    More likely to
    Lower %Prior
    comments
    More likely to
    %Reviewers who provide a +vote when having
    a strong relationship with the patch author
    

    View Slide

26. Investigating the risk that review dynamics can
    have on software quality
    10
    If such review dynamics happen often, will it pose a risk of having defects in a patch?
    STEP 1: Compute review 

    dynamics metrics
    %Reviewers who provided a +vote when
    there are prior +votes
    Lower %Prior
    comments
    More likely to
    %Reviewers who provide a +vote when having
    a strong relationship with the patch author
    

    View Slide

27. Investigating the risk that review dynamics can
    have on software quality
    10
    If such review dynamics happen often, will it pose a risk of having defects in a patch?
    STEP 1: Compute review 

    dynamics metrics
    %Reviewers who provided a +vote when
    there are prior +votes
    %Reviewers who provide a +vote when
    no reviewer comments
    %Reviewers who provide a +vote when having
    a strong relationship with the patch author
    

    View Slide

28. Investigating the risk that review dynamics can
    have on software quality
    10
    If such review dynamics happen often, will it pose a risk of having defects in a patch?
    STEP 1: Compute review 

    dynamics metrics
    %Reviewers who provided a +vote when
    there are prior +votes
    %Reviewers who provide a +vote when
    no reviewer comments
    %Reviewers who provide a +vote when having
    a strong relationship with the patch author
    STEP 2: Analyze the relationship with
    the defect-proness of a patch
    

    View Slide

29. Investigating the risk that review dynamics can
    have on software quality
    10
    If such review dynamics happen often, will it pose a risk of having defects in a patch?
    STEP 1: Compute review 

    dynamics metrics
    %Reviewers who provided a +vote when
    there are prior +votes
    %Reviewers who provide a +vote when
    no reviewer comments
    %Reviewers who provide a +vote when having
    a strong relationship with the patch author
    STEP 2: Analyze the relationship with
    the defect-proness of a patch
    Use a simple logistic
    regression model
    Patch
    Characteristics
    Reviewing

    Activities
    Confounding factors:
    

    View Slide

30. Review dynamics are weakly associated with
    defect-proneness of a patch
    11
    Explanatory Power
    (Log-likelihood ratio test)
    Patch
    Characteristics
    1
    Reviewing

    Activities
    2
    Review
    Dynamics
    3
    Compare with other confounding factors,
    
    review dynamics have a relatively small
    impact on software quality
    

    View Slide

31. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Looks like everyone
    is happy with it
    Visible
    Feedback
    A code review tool
    (Ex. Gerrit)
    Reviewer
    Reviewer
    Reviewer
    In addition to patch characteristics, other visible
    information is associated with the review decision
    25K Patches, 694Reviewers
    (09/2013 - 07/2019)
    57K Patches, 3.6K Reviewers
    (11/2011 - 07/2019)
    A Case study
    Relationship Status
    Prior Feedback
    Patch
    Characteristics
    Explanatory Power
    (Log-likelihood ratio test)
    Association Direction
    Higher %Reviewed
    past patches for
    the patch author
    More likely to
    Higher %Prior
    positive votes
    Lower %Prior
    comments
    More likely to
    More likely to
    Our work sheds light on the implicit influence that the
    visible information may have on the review decision of
    a reviewer. However, such review dynamics should not
    be too concerning in terms of software quality
    patanamon.t@unimelb.edu.au
    @patanamon
    http://patanamon.com
    Review dynamics are weakly associated with
    defect-proneness of a patch
    11
    Explanatory Power
    (Log-likelihood ratio test)
    Patch
    Characteristics
    1
    Reviewing

    Activities
    2
    Review
    Dynamics
    3
    Compare with other confounding factors,
    
    review dynamics have a relatively small
    impact on software quality
    

    View Slide