$30 off During Our Annual Pro Sale. View Details »

Review Dynamics and Their Impact on Software Quality

Review Dynamics and Their Impact on Software Quality

Video available at: https://youtu.be/DK9GqrrQBRU
Venue: IEEE Transaction on Software Engineering
Abstract:
Code review is a crucial activity for ensuring the quality of software products. Unlike the traditional code review process of the past where reviewers independently examine software artifacts, contemporary code review processes allow teams to collaboratively examine and discuss proposed patches. While the visibility of reviewing activities including review discussions in a contemporary code review tends to increase developer collaboration and openness, little is known whether such visible information influences the evaluation decision of a reviewer or not (i.e., knowing others’ feedback about the patch before providing ones own feedback). Therefore, in this work, we set out to investigate the review dynamics, i.e., a practice of providing a vote to accept a proposed patch, in a code review process. To do so, we first characterize the review dynamics by examining the relationship between the evaluation decision of a reviewer and the visible information about a patch under review (e.g., comments and votes that are provided by prior co-reviewers). We then investigate the association between the characterized review dynamics and the defect-proneness of a patch. Through a case study of 83,750 patches of the OpenStack and Qt projects, we observe that the amount of feedback (either votes and comments of prior reviewers) and the co-working frequency of a reviewer with the patch author are highly associated with the likelihood that the reviewer will provide a positive vote to accept a proposed patch. Furthermore, we find that the proportion of reviewers who provided a vote consistent with prior reviewers is significantly associated with the defect-proneness of a patch. However, the associations of these review dynamics are not as strong as the confounding factors (i.e., patch characteristics and overall reviewing activities). Our observations shed light on the implicit influence of the visible information about a patch under review on the evaluation decision of a reviewer. Our findings suggest that the code reviewing policies that are mindful of these practices may help teams improve code review effectiveness. Nonetheless, such review dynamics should not be too concerning in terms of software quality.

More Decks by Patanamon (Pick) Thongtanunam

Other Decks in Research

Transcript

  1. [email protected] @patanamon
    Review Dynamics and Their Impact on
    Software Quality
    - Journal First Presentation -
    Patanamon (Pick)
    Thongtanunam Ahmed Hassan

    View Slide

  2. Reviewer Reviewer
    Code Review: A manual activity to examine and
    improve the overall quality of a new code change
    Author
    A patch
    A code review tool
    (Ex. Gerrit)
    2
    Code review tools often provide
    a transparent environment
    One can see and learn from others’
    code and review comments

    [Bacchelli and Bird, ICSE2013]
    Expertise can be perceived from visible
    history of code changes and reviews

    [Dabbish et. al., CSCW2012; 

    Thongtanunam et. al., ICSE2016]

    View Slide

  3. Relationship with
    the patch author
    Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Reviewer
    Author A code review tool
    (Ex. Gerrit)
    A patch
    3

    View Slide

  4. Relationship with
    the patch author
    Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Reviewer
    Author A code review tool
    (Ex. Gerrit)
    A patch
    3
    Pick usually writes
    good code

    View Slide

  5. Relationship with
    the patch author
    Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Reviewer
    Author A code review tool
    (Ex. Gerrit)
    A patch
    3
    Pick usually writes
    good code
    Looks good to me

    View Slide

  6. Relationship with
    the patch author
    Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Reviewer
    Author A code review tool
    (Ex. Gerrit)
    A patch
    3
    Pick usually writes
    good code
    Looks good to me
    Ingroup-Outgroup Bias

    View Slide

  7. Relationship with
    the patch author
    Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Reviewer
    Author A code review tool
    (Ex. Gerrit)
    A patch
    3
    Pick usually writes
    good code
    Looks good to me
    “Patches from a trustworthy author are
    more likely to require less reviewing effort”
    [Bosu et. al., TSE2017]
    Developers with a weak relationship (e.g.,
    newcomers) face negative impressions

    [Lee et. al., ICSE2017; Steinmacher et.al., CSCW2015]
    Ingroup-Outgroup Bias

    View Slide

  8. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Reviewer
    Author A code review tool
    (Ex. Gerrit)
    A patch
    4
    Status or Reputation

    View Slide

  9. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Reviewer
    Author A code review tool
    (Ex. Gerrit)
    A patch
    4
    Status or Reputation

    View Slide

  10. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Reviewer
    Author A code review tool
    (Ex. Gerrit)
    A patch
    4
    Ahmed is a core
    developer
    Looks good to me
    Status or Reputation

    View Slide

  11. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Reviewer
    Author A code review tool
    (Ex. Gerrit)
    A patch
    4
    Ahmed is a core
    developer
    Looks good to me
    Status or Reputation
    A cognitive bias in terms of
    social hierarchy

    View Slide

  12. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Reviewer
    Author A code review tool
    (Ex. Gerrit)
    A patch
    4
    Ahmed is a core
    developer
    Looks good to me
    Developers’ reputation can influence the
    review decision of their patches

    [Bosu & Carver, ESEM2014]
    Status or Reputation
    A cognitive bias in terms of
    social hierarchy

    View Slide

  13. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Reviewer
    Author A code review tool
    (Ex. Gerrit)
    A patch
    4
    Ahmed is a core
    developer
    Looks good to me
    Developers’ reputation can influence the
    review decision of their patches

    [Bosu & Carver, ESEM2014]
    Feedback of outsiders tends to receive less
    attention than that of the core members

    [Rigby & Storey, ICSE2011]
    Status or Reputation
    A cognitive bias in terms of
    social hierarchy

    View Slide

  14. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    5
    Visible
    Feedback
    A code review tool
    (Ex. Gerrit)
    Reviewer
    Reviewer

    View Slide

  15. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    5
    Visible
    Feedback
    A code review tool
    (Ex. Gerrit)
    Reviewer
    Reviewer
    Reviewer

    View Slide

  16. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    5
    Looks like everyone
    is happy with it
    Visible
    Feedback
    A code review tool
    (Ex. Gerrit)
    Reviewer
    Reviewer
    Reviewer

    View Slide

  17. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    5
    Looks like everyone
    is happy with it
    Visible
    Feedback
    A code review tool
    (Ex. Gerrit)
    Reviewer
    Reviewer
    Reviewer

    View Slide

  18. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    5
    Looks like everyone
    is happy with it
    Visible
    Feedback
    A code review tool
    (Ex. Gerrit)
    Reviewer
    Reviewer
    Reviewer
    Social Influence

    View Slide

  19. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    5
    Looks like everyone
    is happy with it
    “Given that (the prior reviewer) already
    +1’d this patch as well, I will +2 it”
    OpenStack Review ID 1411
    “-1 to this patch. Backing up (the first
    reviewer)’s comment on.....”
    OpenStack Review ID 1257
    Visible
    Feedback
    A code review tool
    (Ex. Gerrit)
    Reviewer
    Reviewer
    Reviewer
    Social Influence

    View Slide

  20. Investigating the signals of visible information that
    are associated with the review decision of a reviewer
    6
    STEP 1: Capture visible information
    about a patch under review Use 8 metrics grouped into 3 dimensions
    Relationship dimension
    e.g., the co-working frequency of
    a reviewer with the patch author
    Status dimension
    e.g., is a patch author a
    core member
    Prior feedback dimension
    e.g., %prior positive votes

    View Slide

  21. Investigating the signals of visible information that
    are associated with the review decision of a reviewer
    7
    STEP 2: Analyze the relationship with
    the likelihood of giving a positive vote Use a mixed-effects logistic regression model
    IsPositiveVote ∼ x1 + x2 + ….. + xn
    + (1 | ReviewerId )
    8 Studied metrics
    Relationship Status Prior Feedback
    Confounding factors:

    Patch Characteristics as
    e.g., #Added Lines

    View Slide

  22. In addition to patch characteristics, other visible
    information is associated with the review decision
    8
    25K Patches, 694Reviewers
    (09/2013 - 07/2019)
    57K Patches, 3.6K Reviewers
    (11/2011 - 07/2019)
    A Case study
    Relationship Status
    Prior Feedback
    Patch
    Characteristics
    Explanatory Power
    (Log-likelihood ratio test)
    Association Direction
    Higher %Reviewed
    past patches for
    the patch author
    More likely to
    Higher %Prior
    positive votes
    Lower %Prior
    comments
    More likely to
    More likely to

    View Slide

  23. In addition to patch characteristics, other visible
    information is associated with the review decision
    9
    Relationship Status
    Prior Feedback
    Patch
    Characteristics
    Explanatory Power
    (Log-likelihood ratio test)
    Association Direction
    Higher %Reviewed
    past patches for
    the patch author
    More likely to
    Higher %Prior
    positive votes
    Lower %Prior
    comments
    More likely to
    More likely to
    We can characterize review dynamics 

    (i.e., the practices of making a review decision)
    Visible information has a stronger association with
    the review decision than patch characteristics

    View Slide

  24. Investigating the risk that review dynamics can
    have on software quality
    10
    If such review dynamics happen often, will it pose a risk of having defects in a patch?

    View Slide

  25. Investigating the risk that review dynamics can
    have on software quality
    10
    If such review dynamics happen often, will it pose a risk of having defects in a patch?
    STEP 1: Compute review 

    dynamics metrics
    Higher %Prior
    positive votes
    More likely to
    Lower %Prior
    comments
    More likely to
    %Reviewers who provide a +vote when having
    a strong relationship with the patch author

    View Slide

  26. Investigating the risk that review dynamics can
    have on software quality
    10
    If such review dynamics happen often, will it pose a risk of having defects in a patch?
    STEP 1: Compute review 

    dynamics metrics
    %Reviewers who provided a +vote when
    there are prior +votes
    Lower %Prior
    comments
    More likely to
    %Reviewers who provide a +vote when having
    a strong relationship with the patch author

    View Slide

  27. Investigating the risk that review dynamics can
    have on software quality
    10
    If such review dynamics happen often, will it pose a risk of having defects in a patch?
    STEP 1: Compute review 

    dynamics metrics
    %Reviewers who provided a +vote when
    there are prior +votes
    %Reviewers who provide a +vote when
    no reviewer comments
    %Reviewers who provide a +vote when having
    a strong relationship with the patch author

    View Slide

  28. Investigating the risk that review dynamics can
    have on software quality
    10
    If such review dynamics happen often, will it pose a risk of having defects in a patch?
    STEP 1: Compute review 

    dynamics metrics
    %Reviewers who provided a +vote when
    there are prior +votes
    %Reviewers who provide a +vote when
    no reviewer comments
    %Reviewers who provide a +vote when having
    a strong relationship with the patch author
    STEP 2: Analyze the relationship with
    the defect-proness of a patch

    View Slide

  29. Investigating the risk that review dynamics can
    have on software quality
    10
    If such review dynamics happen often, will it pose a risk of having defects in a patch?
    STEP 1: Compute review 

    dynamics metrics
    %Reviewers who provided a +vote when
    there are prior +votes
    %Reviewers who provide a +vote when
    no reviewer comments
    %Reviewers who provide a +vote when having
    a strong relationship with the patch author
    STEP 2: Analyze the relationship with
    the defect-proness of a patch
    Use a simple logistic
    regression model
    Patch
    Characteristics
    Reviewing

    Activities
    Confounding factors:

    View Slide

  30. Review dynamics are weakly associated with
    defect-proneness of a patch
    11
    Explanatory Power
    (Log-likelihood ratio test)
    Patch
    Characteristics
    1
    Reviewing

    Activities
    2
    Review
    Dynamics
    3
    Compare with other confounding factors,

    review dynamics have a relatively small
    impact on software quality

    View Slide

  31. Reviewers may have subconscious biases due to
    the visible information in a code review tool
    Looks like everyone
    is happy with it
    Visible
    Feedback
    A code review tool
    (Ex. Gerrit)
    Reviewer
    Reviewer
    Reviewer
    In addition to patch characteristics, other visible
    information is associated with the review decision
    25K Patches, 694Reviewers
    (09/2013 - 07/2019)
    57K Patches, 3.6K Reviewers
    (11/2011 - 07/2019)
    A Case study
    Relationship Status
    Prior Feedback
    Patch
    Characteristics
    Explanatory Power
    (Log-likelihood ratio test)
    Association Direction
    Higher %Reviewed
    past patches for
    the patch author
    More likely to
    Higher %Prior
    positive votes
    Lower %Prior
    comments
    More likely to
    More likely to
    Our work sheds light on the implicit influence that the
    visible information may have on the review decision of
    a reviewer. However, such review dynamics should not
    be too concerning in terms of software quality
    [email protected]
    @patanamon
    http://patanamon.com
    Review dynamics are weakly associated with
    defect-proneness of a patch
    11
    Explanatory Power
    (Log-likelihood ratio test)
    Patch
    Characteristics
    1
    Reviewing

    Activities
    2
    Review
    Dynamics
    3
    Compare with other confounding factors,

    review dynamics have a relatively small
    impact on software quality

    View Slide