Review Dynamics and Their Impact on Software Quality

Slide 1

Slide 1 text

[email protected] @patanamon Review Dynamics and Their Impact on Software Quality - Journal First Presentation - Patanamon (Pick) Thongtanunam Ahmed Hassan

Slide 2

Slide 2 text

Reviewer Reviewer Code Review: A manual activity to examine and improve the overall quality of a new code change Author A patch A code review tool (Ex. Gerrit) 2 Code review tools often provide a transparent environment One can see and learn from others’ code and review comments [Bacchelli and Bird, ICSE2013] Expertise can be perceived from visible history of code changes and reviews [Dabbish et. al., CSCW2012;   Thongtanunam et. al., ICSE2016]

Slide 3

Slide 3 text

Relationship with the patch author Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 3

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Relationship with the patch author Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 3 Pick usually writes good code Looks good to me “Patches from a trustworthy author are more likely to require less reviewing eﬀort” [Bosu et. al., TSE2017] Developers with a weak relationship (e.g., newcomers) face negative impressions [Lee et. al., ICSE2017; Steinmacher et.al., CSCW2015] Ingroup-Outgroup Bias

Slide 8

Slide 8 text

Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 4 Status or Reputation

Slide 9

Slide 9 text

Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 4 Status or Reputation

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 4 Ahmed is a core developer Looks good to me Developers’ reputation can inﬂuence the review decision of their patches [Bosu & Carver, ESEM2014] Status or Reputation A cognitive bias in terms of social hierarchy

Slide 13

Slide 13 text

Reviewers may have subconscious biases due to the visible information in a code review tool Reviewer Author A code review tool (Ex. Gerrit) A patch 4 Ahmed is a core developer Looks good to me Developers’ reputation can inﬂuence the review decision of their patches [Bosu & Carver, ESEM2014] Feedback of outsiders tends to receive less attention than that of the core members [Rigby & Storey, ICSE2011] Status or Reputation A cognitive bias in terms of social hierarchy

Slide 14

Slide 14 text

Reviewers may have subconscious biases due to the visible information in a code review tool 5 Visible Feedback A code review tool (Ex. Gerrit) Reviewer Reviewer

Slide 15

Slide 15 text

Reviewers may have subconscious biases due to the visible information in a code review tool 5 Visible Feedback A code review tool (Ex. Gerrit) Reviewer Reviewer Reviewer

Slide 16

Slide 16 text

Reviewers may have subconscious biases due to the visible information in a code review tool 5 Looks like everyone is happy with it Visible Feedback A code review tool (Ex. Gerrit) Reviewer Reviewer Reviewer

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Reviewers may have subconscious biases due to the visible information in a code review tool 5 Looks like everyone is happy with it “Given that (the prior reviewer) already +1’d this patch as well, I will +2 it” OpenStack Review ID 1411 “-1 to this patch. Backing up (the ﬁrst reviewer)’s comment on.....” OpenStack Review ID 1257 Visible Feedback A code review tool (Ex. Gerrit) Reviewer Reviewer Reviewer Social Inﬂuence

Slide 20

Slide 20 text

Investigating the signals of visible information that are associated with the review decision of a reviewer 6 STEP 1: Capture visible information about a patch under review Use 8 metrics grouped into 3 dimensions Relationship dimension e.g., the co-working frequency of a reviewer with the patch author Status dimension e.g., is a patch author a core member Prior feedback dimension e.g., %prior positive votes

Slide 21

Slide 21 text

Investigating the signals of visible information that are associated with the review decision of a reviewer 7 STEP 2: Analyze the relationship with the likelihood of giving a positive vote Use a mixed-eﬀects logistic regression model IsPositiveVote ∼ x1 + x2 + ….. + xn + (1 | ReviewerId ) 8 Studied metrics Relationship Status Prior Feedback Confounding factors:  Patch Characteristics as e.g., #Added Lines

Slide 22

Slide 22 text

In addition to patch characteristics, other visible information is associated with the review decision 8 25K Patches, 694Reviewers (09/2013 - 07/2019) 57K Patches, 3.6K Reviewers (11/2011 - 07/2019) A Case study Relationship Status Prior Feedback Patch Characteristics Explanatory Power (Log-likelihood ratio test) Association Direction Higher %Reviewed past patches for the patch author More likely to Higher %Prior positive votes Lower %Prior comments More likely to More likely to

Slide 23

Slide 23 text

In addition to patch characteristics, other visible information is associated with the review decision 9 Relationship Status Prior Feedback Patch Characteristics Explanatory Power (Log-likelihood ratio test) Association Direction Higher %Reviewed past patches for the patch author More likely to Higher %Prior positive votes Lower %Prior comments More likely to More likely to We can characterize review dynamics   (i.e., the practices of making a review decision) Visible information has a stronger association with the review decision than patch characteristics

Slide 24

Slide 24 text

Investigating the risk that review dynamics can have on software quality 10 If such review dynamics happen often, will it pose a risk of having defects in a patch?

Slide 25

Slide 25 text

Investigating the risk that review dynamics can have on software quality 10 If such review dynamics happen often, will it pose a risk of having defects in a patch? STEP 1: Compute review   dynamics metrics Higher %Prior positive votes More likely to Lower %Prior comments More likely to %Reviewers who provide a +vote when having a strong relationship with the patch author

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Investigating the risk that review dynamics can have on software quality 10 If such review dynamics happen often, will it pose a risk of having defects in a patch? STEP 1: Compute review   dynamics metrics %Reviewers who provided a +vote when there are prior +votes %Reviewers who provide a +vote when no reviewer comments %Reviewers who provide a +vote when having a strong relationship with the patch author

Slide 28

Slide 28 text

Investigating the risk that review dynamics can have on software quality 10 If such review dynamics happen often, will it pose a risk of having defects in a patch? STEP 1: Compute review   dynamics metrics %Reviewers who provided a +vote when there are prior +votes %Reviewers who provide a +vote when no reviewer comments %Reviewers who provide a +vote when having a strong relationship with the patch author STEP 2: Analyze the relationship with the defect-proness of a patch

Slide 29

Slide 29 text

Investigating the risk that review dynamics can have on software quality 10 If such review dynamics happen often, will it pose a risk of having defects in a patch? STEP 1: Compute review   dynamics metrics %Reviewers who provided a +vote when there are prior +votes %Reviewers who provide a +vote when no reviewer comments %Reviewers who provide a +vote when having a strong relationship with the patch author STEP 2: Analyze the relationship with the defect-proness of a patch Use a simple logistic regression model Patch Characteristics Reviewing  Activities Confounding factors:

Slide 30

Slide 30 text

Review dynamics are weakly associated with defect-proneness of a patch 11 Explanatory Power (Log-likelihood ratio test) Patch Characteristics 1 Reviewing  Activities 2 Review Dynamics 3 Compare with other confounding factors, review dynamics have a relatively small impact on software quality

Slide 31

Slide 31 text

Reviewers may have subconscious biases due to the visible information in a code review tool Looks like everyone is happy with it Visible Feedback A code review tool (Ex. Gerrit) Reviewer Reviewer Reviewer In addition to patch characteristics, other visible information is associated with the review decision 25K Patches, 694Reviewers (09/2013 - 07/2019) 57K Patches, 3.6K Reviewers (11/2011 - 07/2019) A Case study Relationship Status Prior Feedback Patch Characteristics Explanatory Power (Log-likelihood ratio test) Association Direction Higher %Reviewed past patches for the patch author More likely to Higher %Prior positive votes Lower %Prior comments More likely to More likely to Our work sheds light on the implicit influence that the visible information may have on the review decision of a reviewer. However, such review dynamics should not be too concerning in terms of software quality [email protected] @patanamon http://patanamon.com Review dynamics are weakly associated with defect-proneness of a patch 11 Explanatory Power (Log-likelihood ratio test) Patch Characteristics 1 Reviewing  Activities 2 Review Dynamics 3 Compare with other confounding factors, review dynamics have a relatively small impact on software quality