Review Dynamics and Their Impact on Software Quality

Video available at: https://youtu.be/DK9GqrrQBRU
Venue: IEEE Transaction on Software Engineering
Abstract:
Code review is a crucial activity for ensuring the quality of software products. Unlike the traditional code review process of the past where reviewers independently examine software artifacts, contemporary code review processes allow teams to collaboratively examine and discuss proposed patches. While the visibility of reviewing activities including review discussions in a contemporary code review tends to increase developer collaboration and openness, little is known whether such visible information influences the evaluation decision of a reviewer or not (i.e., knowing others’ feedback about the patch before providing ones own feedback). Therefore, in this work, we set out to investigate the review dynamics, i.e., a practice of providing a vote to accept a proposed patch, in a code review process. To do so, we first characterize the review dynamics by examining the relationship between the evaluation decision of a reviewer and the visible information about a patch under review (e.g., comments and votes that are provided by prior co-reviewers). We then investigate the association between the characterized review dynamics and the defect-proneness of a patch. Through a case study of 83,750 patches of the OpenStack and Qt projects, we observe that the amount of feedback (either votes and comments of prior reviewers) and the co-working frequency of a reviewer with the patch author are highly associated with the likelihood that the reviewer will provide a positive vote to accept a proposed patch. Furthermore, we find that the proportion of reviewers who provided a vote consistent with prior reviewers is significantly associated with the defect-proneness of a patch. However, the associations of these review dynamics are not as strong as the confounding factors (i.e., patch characteristics and overall reviewing activities). Our observations shed light on the implicit influence of the visible information about a patch under review on the evaluation decision of a reviewer. Our findings suggest that the code reviewing policies that are mindful of these practices may help teams improve code review effectiveness. Nonetheless, such review dynamics should not be too concerning in terms of software quality.

Transcript

1. patanamon.t@unimelb.edu.au @patanamon Review Dynamics and Their Impact on Software Quality - Journal First Presentation - Patanamon (Pick) Thongtanunam Ahmed Hassan
2. Reviewer Reviewer Code Review: A manual activity to examine and improve the overall quality of a new code change Author A patch A code review tool (Ex. Gerrit) 2 Code review tools often provide a transparent environment One can see and learn from others’ code and review comments [Bacchelli and Bird, ICSE2013] Expertise can be perceived from visible history of code changes and reviews [Dabbish et. al., CSCW2012; 
 Thongtanunam et. al., ICSE2016]
3. Relationship with the patch author Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 3
4. Relationship with the patch author Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 3 Pick usually writes good code
5. Relationship with the patch author Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 3 Pick usually writes good code Looks good to me
6. Relationship with the patch author Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 3 Pick usually writes good code Looks good to me Ingroup-Outgroup Bias
7. Relationship with the patch author Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 3 Pick usually writes good code Looks good to me “Patches from a trustworthy author are more likely to require less reviewing effort” [Bosu et. al., TSE2017] Developers with a weak relationship (e.g., newcomers) face negative impressions [Lee et. al., ICSE2017; Steinmacher et.al., CSCW2015] Ingroup-Outgroup Bias
8. Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 4 Status or Reputation
9. Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 4 Status or Reputation
10. Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 4 Ahmed is a core developer Looks good to me Status or Reputation
11. Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 4 Ahmed is a core developer Looks good to me Status or Reputation A cognitive bias in terms of social hierarchy
12. Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 4 Ahmed is a core developer Looks good to me Developers’ reputation can influence the review decision of their patches [Bosu & Carver, ESEM2014] Status or Reputation A cognitive bias in terms of social hierarchy
13. Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 4 Ahmed is a core developer Looks good to me Developers’ reputation can influence the review decision of their patches [Bosu & Carver, ESEM2014] Feedback of outsiders tends to receive less attention than that of the core members [Rigby & Storey, ICSE2011] Status or Reputation A cognitive bias in terms of social hierarchy
14. Reviewers may have subconscious biases due to the visible information in a code review tool 5 Visible Feedback A code review tool (Ex. Gerrit) Reviewer Reviewer
15. Reviewers may have subconscious biases due to the visible information in a code review tool 5 Visible Feedback A code review tool (Ex. Gerrit) Reviewer Reviewer Reviewer
16. Reviewers may have subconscious biases due to the visible information in a code review tool 5 Looks like everyone is happy with it Visible Feedback A code review tool (Ex. Gerrit) Reviewer Reviewer Reviewer
17. Reviewers may have subconscious biases due to the visible information in a code review tool 5 Looks like everyone is happy with it Visible Feedback A code review tool (Ex. Gerrit) Reviewer Reviewer Reviewer
18. Reviewers may have subconscious biases due to the visible information in a code review tool 5 Looks like everyone is happy with it Visible Feedback A code review tool (Ex. Gerrit) Reviewer Reviewer Reviewer Social Influence
19. Reviewers may have subconscious biases due to the visible information in a code review tool 5 Looks like everyone is happy with it “Given that (the prior reviewer) already +1’d this patch as well, I will +2 it” OpenStack Review ID 1411 “-1 to this patch. Backing up (the first reviewer)’s comment on.....” OpenStack Review ID 1257 Visible Feedback A code review tool (Ex. Gerrit) Reviewer Reviewer Reviewer Social Influence
20. Investigating the signals of visible information that are associated with the review decision of a reviewer 6 STEP 1: Capture visible information about a patch under review Use 8 metrics grouped into 3 dimensions Relationship dimension e.g., the co-working frequency of a reviewer with the patch author Status dimension e.g., is a patch author a core member Prior feedback dimension e.g., %prior positive votes
21. Investigating the signals of visible information that are associated with the review decision of a reviewer 7 STEP 2: Analyze the relationship with the likelihood of giving a positive vote Use a mixed-effects logistic regression model IsPositiveVote ∼ x1 + x2 + ….. + xn + (1 | ReviewerId ) 8 Studied metrics Relationship Status Prior Feedback Confounding factors:
 Patch Characteristics as e.g., #Added Lines
22. In addition to patch characteristics, other visible information is associated with the review decision 8 25K Patches, 694Reviewers (09/2013 - 07/2019) 57K Patches, 3.6K Reviewers (11/2011 - 07/2019) A Case study Relationship Status Prior Feedback Patch Characteristics Explanatory Power (Log-likelihood ratio test) Association Direction Higher %Reviewed past patches for the patch author More likely to Higher %Prior positive votes Lower %Prior comments More likely to More likely to
23. In addition to patch characteristics, other visible information is associated with the review decision 9 Relationship Status Prior Feedback Patch Characteristics Explanatory Power (Log-likelihood ratio test) Association Direction Higher %Reviewed past patches for the patch author More likely to Higher %Prior positive votes Lower %Prior comments More likely to More likely to We can characterize review dynamics 
 (i.e., the practices of making a review decision) Visible information has a stronger association with the review decision than patch characteristics
24. Investigating the risk that review dynamics can have on software quality 10 If such review dynamics happen often, will it pose a risk of having defects in a patch?
25. Investigating the risk that review dynamics can have on software quality 10 If such review dynamics happen often, will it pose a risk of having defects in a patch? STEP 1: Compute review 
 dynamics metrics Higher %Prior positive votes More likely to Lower %Prior comments More likely to %Reviewers who provide a +vote when having a strong relationship with the patch author
26. Investigating the risk that review dynamics can have on software quality 10 If such review dynamics happen often, will it pose a risk of having defects in a patch? STEP 1: Compute review 
 dynamics metrics %Reviewers who provided a +vote when there are prior +votes Lower %Prior comments More likely to %Reviewers who provide a +vote when having a strong relationship with the patch author
27. Investigating the risk that review dynamics can have on software quality 10 If such review dynamics happen often, will it pose a risk of having defects in a patch? STEP 1: Compute review 
 dynamics metrics %Reviewers who provided a +vote when there are prior +votes %Reviewers who provide a +vote when no reviewer comments %Reviewers who provide a +vote when having a strong relationship with the patch author
28. Investigating the risk that review dynamics can have on software quality 10 If such review dynamics happen often, will it pose a risk of having defects in a patch? STEP 1: Compute review 
 dynamics metrics %Reviewers who provided a +vote when there are prior +votes %Reviewers who provide a +vote when no reviewer comments %Reviewers who provide a +vote when having a strong relationship with the patch author STEP 2: Analyze the relationship with the defect-proness of a patch
29. Investigating the risk that review dynamics can have on software quality 10 If such review dynamics happen often, will it pose a risk of having defects in a patch? STEP 1: Compute review 
 dynamics metrics %Reviewers who provided a +vote when there are prior +votes %Reviewers who provide a +vote when no reviewer comments %Reviewers who provide a +vote when having a strong relationship with the patch author STEP 2: Analyze the relationship with the defect-proness of a patch Use a simple logistic regression model Patch Characteristics Reviewing
 Activities Confounding factors:
30. Review dynamics are weakly associated with defect-proneness of a patch 11 Explanatory Power (Log-likelihood ratio test) Patch Characteristics 1 Reviewing
 Activities 2 Review Dynamics 3 Compare with other confounding factors, review dynamics have a relatively small impact on software quality
31. Reviewers may have subconscious biases due to the visible information in a code review tool Looks like everyone is happy with it Visible Feedback A code review tool (Ex. Gerrit) Reviewer Reviewer Reviewer In addition to patch characteristics, other visible information is associated with the review decision 25K Patches, 694Reviewers (09/2013 - 07/2019) 57K Patches, 3.6K Reviewers (11/2011 - 07/2019) A Case study Relationship Status Prior Feedback Patch Characteristics Explanatory Power (Log-likelihood ratio test) Association Direction Higher %Reviewed past patches for the patch author More likely to Higher %Prior positive votes Lower %Prior comments More likely to More likely to Our work sheds light on the implicit influence that the visible information may have on the review decision of a reviewer. However, such review dynamics should not be too concerning in terms of software quality patanamon.t@unimelb.edu.au @patanamon http://patanamon.com Review dynamics are weakly associated with defect-proneness of a patch 11 Explanatory Power (Log-likelihood ratio test) Patch Characteristics 1 Reviewing
 Activities 2 Review Dynamics 3 Compare with other confounding factors, review dynamics have a relatively small impact on software quality