Mining Gerrit Reviews for Contentious Reviews and Community Evolution

View Slide

You ask me for a contribution
Well, you know
We’re all doing what we can

View Slide

Shane
McIntosh
Assistant
Professor
[email protected]
@SoftwareREBELs
rebels.ece.mcgill.ca
RE
Software
s
EL
B
Mining Gerrit
for Contentious Reviews
& Community Evolution

View Slide

RE
Software
s
EL
B
RE
Software
s
EL
B

View Slide

RE
Software
s
EL
B
RE
Software
s
EL
B
Repository
Excavation

View Slide

Development data is mainly
kept for archival purposes
4

View Slide

Development data is mainly
kept for archival purposes
4
What if your software
repositories could talk?

View Slide

My students and I mine through historical
development data in search of gold
5

View Slide

5
In which modules
are future defects
likely to appear?
Baljinder
Ghotra
Chakkrit
(Kla) T.

View Slide

5
In which modules
are future defects
likely to appear?
Baljinder
Ghotra
Chakkrit
(Kla) T.
Can risky code
be predicted
at creation time?
Takafumi
Fukushima
Jacob G.
Barnett

View Slide

RE
Software
s
EL
B
RE
Software
s
EL
B
Repository
Excavation

View Slide

RE
Software
s
EL
B
RE
Software
s
EL
B
report
deploy
build
commit
Build
Engineering
RE
S B
Repository
Excavation

View Slide

Release Engineering:
Supporting release pipelines
7

View Slide

Patch
+
+
-
-
7

View Slide

1. Integration
Patch
+
+
-
-
7

View Slide

1. Integration
Patch
+
+
-
-
2. Build
7

View Slide

1. Integration
Patch
+
+
-
-
New!
2. Build
3. Deployment
7

View Slide

1. Integration
Patch
+
+
-
-
New!
2. Build
3. Deployment
7
Decision support for
modern release pipelines

View Slide

1. Integration
Patch
2. Build
3. Deployment
+
+
-
-
New!
8

View Slide

1. Integration
Patch
2. Build
3. Deployment
+
+
-
-
New!
8
Can we improve
support for modern
code review?
Toshiki
Hirao
Ray
Wen

View Slide

1. Integration
Patch
2. Build
3. Deployment
+
+
-
-
New!
8
How can we
optimize available
CI/CD resources?
Keheliya
Gallaba
Can we improve
support for modern
code review?
Toshiki
Hirao
Ray
Wen

View Slide

RE
Software
s
EL
B
report
deploy
build
commit
Build
Engineering
Repository
Excavation

View Slide

RE
Software
s
EL
B
report
deploy
build
commit
Build
Engineering
Repository
Excavation
We are looking to form
industrial partnerships!

View Slide

10
An overview of the modern
code review process

View Slide

Author
Module
A
Code change
10
code review process

View Slide

A code review tool
(e.g., Gerrit)
Author
Module
A
Code change
10
code review process

View Slide

Module
A
Qt Review #27977
11
A code review tool
(e.g., Gerrit)
Author
Reviewers may identify problems or
suggest alternative approaches

View Slide

Module
A
Qt Review #27977
11
A code review tool
(e.g., Gerrit)
Reviewer
Author

View Slide

Module
A
Qt Review #27977
11
A code review tool
(e.g., Gerrit)
Reviewer
Shouldn't console.log() call the
toString() method (where
appropriate) on objects?
Identifying a defect
Author

View Slide

Module
A
Qt Review #27977
11
A code review tool
(e.g., Gerrit)
Reviewer
Reviewer
I think it’s better to do
var s = "{}"
console.log(s)
Suggesting a solution
Author

View Slide

Module
A
Qt Review #27977
11
A code review tool
(e.g., Gerrit)
Reviewer
Reviewer
I think it’s better to do
var s = "{}"
console.log(s)
Suggesting a solution
Author
Update
v2

View Slide

Author
Module
A
Qt Review #35360
Providing updates to
the code change
Note that running qutlook will probably
crash. I will push a patch set to ﬁx the issue.
Reviewer
12
A code review tool
(e.g., Gerrit)
Some reviewers even provide potential
solutions to the issues they identify

View Slide

Author
Module
A
Qt Review #35360
Providing updates to
the code change
Note that running qutlook will probably
crash. I will push a patch set to ﬁx the issue.
Reviewer
12
A code review tool
(e.g., Gerrit)
v2
Some reviewers even provide potential
solutions to the issues they identify

View Slide

Modern code review:
Changes are guilty until proven innocent

View Slide

(1)
Upload change
revision(s)
(2)
Execute sanity
tests
(3)
Solicit
peer feedback
(4)
Initiate integration
request
(5)
Execute
integration tests
(6)
Final integration
Version
Control
System
Pass
Fail
Legend
Modern code review:
Changes are guilty until proven innocent

View Slide

Code review takes a lot of developer time
and is not always useful
14

View Slide

Impact of Peer Code Review on Peer
Impression Formation: A Survey
[Bosu and Carver, ESEM 2013]
Devs spend
6 hours per week
reviewing code
14

View Slide

Impact of Peer Code Review on Peer
Impression Formation: A Survey
[Bosu and Carver, ESEM 2013]
Devs spend
6 hours per week
reviewing code
Characteristics of Useful Code Reviews:
An Empirical Study at Microsoft
[Bosu et al., MSR 2015]
35% of review
comments are
“not useful”
14

View Slide

Developers do not like to
spend time on process

View Slide

Mining Gerrit reviews to
study reviewing processes

View Slide

Contentious
reviews
Toshiki
Hirao
PhD
Student

View Slide

Community
evolution
Ray
Wen
Master’s
Student
Contentious
reviews
Toshiki
Hirao
PhD
Student

View Slide

View Slide

And when the broken hearted people,
Living in the world agree,
There will be an answer,
Let it be

View Slide

And when the broken hearted people,
Living in the world agree,
There will be an answer,
Let it be
What happens when
reviewers disagree?

View Slide

Reviewer opinions about a
patch may differ
Author

View Slide

patch may differ
Reviewer 1
Author
+2

View Slide

patch may differ
Reviewer 1
Author
Reviewer 2
+2
–2

View Slide

patch may differ
Reviewer 1
Author
Reviewer 2
+2
–2
How do we
proceed?

View Slide

Understanding code review
in contentious patches

View Slide

How often?

View Slide

How often? Trend over time?

View Slide

What drives
abandonment?
Start

View Slide

What drives
abandonment?
Start
How are concerns
resolved?

View Slide

Selecting subject systems for analysis
or: why did we study OpenStack?
0 22,500 45,000 67,500 90,000
Multi-reviewer patches
Solo-reviewer patches
67%
21%
26%
17%
6%

View Slide

What drives
abandonment?
Start
How are concerns
resolved?

View Slide

Classifying contentious reviews
Include SP (+2)
Include SN (-2) Include SN (-2)
Yes No
Yes No
Yes No
+2
+1
-2
-1
+2
+1 -1 +1
-2
-1 +1 -1
SP-SN SP-WN WP-SN WP-WN

View Slide

Handling reviews with several revisions
Contentious Pattern Non-contentious Pattern
Revision 1
Revision 2
-1
B
+2
A
+1
A
+1
C
Revision 1
Revision 2
-2
A
+1
B
+1
C

View Slide

Contentious reviews account for 26% of all
OpenStack reviews
0 2,000 4,000 6,000 8,000
6,378
558
7,753
684 (1%)
(13%)
(1%)
(11%)
+2, -2
+2, -1
+1, -2
+1, -1
Number (and percentage) or reviews in each category

View Slide

Contentious reviews with equal scores
have a tendency to be integrated
0% 22.5% 45% 67.5% 90%
76%
29%
88%
54%
+2, -2
+2, -1
+1, -2
+1, -1
Rate at which patches are eventually integrated

View Slide

What drives
abandonment?
Start
How are concerns
resolved?
Contentious reviews
are not rare
Contention is not a
show stopper

View Slide

Contentious reviews are growing
over time in absolute terms
1
10
100
1000
10000
1 3 5 7 9 11 15 19 23 27 31 35
●
●
●
●
● ●
●
● ●
● ● ●
● ● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
● ● ●
1
10
100
1000
10000
●
SP−SN SP−WN WP−SN WP−WN
11
011 11 12 12
012 12
012 12 13 13
013 13
013 13 14 14
014

View Slide

Contentious reviews are also growing over
time in comparative terms
0%
5%
10%
15%
20%
25%
30%
1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4
Weak Nega;ve (SP-WN, WP-WN) Strong Nega;ve (SP-SN, WP-SN)

View Slide

Integration rates of contentious reviews
have roughly stabilized in recent periods
0.0
0.2
0.4
0.6
0.8
1.0
●
●
●
●
●
● ●
● ● ● ● ● ● ● ● ● ●
● ● ●
● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
0%
20%
40%
60%
80%
100%
●
SP−SN SP−WN WP−SN WP−WN

View Slide

What drives
abandonment?
Start
How are concerns
resolved?
Contentious reviews
are not rare
Contention is not a
show stopper
Contentious reviews
have been growing
over time
Integration rates of
contentious reviews
have stabilized

View Slide

Qualitative analysis:
Selecting a representative sample
684 reviews
with +2 and -2
Sample of 362
reviews
Random
sampling for
95% CL
5% CI

View Slide

Manual classiﬁcation of reviews
Sample of 362
reviews

View Slide

Sample of 362
reviews
Looks like a test
coverage issue

View Slide

Sample of 362
reviews
Looks like a test
coverage issue
Agreed!

View Slide

Sample of 362
reviews
Looks like a test
coverage issue
Agreed!
Test
coverage

View Slide

Lifting tags to cohesive subhierarchies
Test
coverage
Already
ﬁxed
Side
effects
Alternative
solution

View Slide

Test
coverage
Already
ﬁxed
Side
effects
Alternative
solution
Design Unnecessary
contribution
QA

View Slide

Test
coverage
Already
ﬁxed
Side
effects
Alternative
solution
Design Unnecessary
contribution
QA
Internal
concerns
External
concerns

View Slide

Abandonment is often due to
external concerns!
Abandon
External
Concerns
Internal
Concerns
Integration Policy
Compliance
Unnecessary Fix
Integration Planning
Lack of Interest
Testing
Implementation
Design
Unclear Intention
Not an Issue
Already Fixed
Blueprint
Release Schedule
Patch Dependency
Lost by a reviewer
Lost by an author
Side Effect
Shallow Fix
Flawed Changes
Alternative Solution
Backward Compatibility
Patch Size
Test Failure
Test Coverage
59%
(102/173)
21
12
24
6
1
1
32
17
5
6
14
6
1
173
10
41%
(71/173)
58% (59/105)
7% (7/105)
87% (62/71)
10% (7/71)
3% (2/71)
25% (25/105)
5
Branch Placement
Squashing Commits 6
5
11% (11/105)
1

View Slide

What drives
abandonment?
Start
How are concerns
resolved?
Contentious reviews
are not rare
Contention is not a
show stopper
Contentious reviews
have been growing
over time
contentious reviews
have stabilized
Concerns beyond
patch scope are more
strongly linked with
abandonment than
concerns within
patch scope

View Slide

Contention is more often resolved without
altering the patch!
Integrate
Directly
Addressing
Indirectly
Addressing
Legal Problems
Unnecessary Fix
Design
Implementation
Testing
Integration Policy
Compliance
Alternative Solution
Side Effect
Shallow Fix
Flawed Changes
Backward Compatibility
Test Coverage
Test Failure
Squashing Commits
Branch Placement
44%
(83/189)
2
1
3
9
21
7
4
65
12
3
3
3
70% (58/83)
189
Withdrawal of
Negative Score
Integration
Planning
Persuasion
Self-change
Release Schedule
Blueprint
Patch Dependency
10
56%
(106/189)
4% (3/83)
7% (6/83)
5% (4/83)
28% (30/106)
72% (76/106)
Legal Issues
Unclear Intention
12% (10/83)
2% (2/83)
4
4
38

View Slide

What drives
abandonment?
Start
How are concerns
resolved?
Contentious reviews
are not rare
Contention is not a
show stopper
Contentious reviews
have been growing
over time
contentious reviews
have stabilized
Concerns beyond
patch scope are more
strongly linked with
abandonment than
concerns within
patch scope
Concerns are often
addressed through
negotiation or scheduling

View Slide

View Slide

Life is very short,
and there’s no time,
for fussing and ﬁghting, my friends

View Slide

Community
evolution
Ray
Wen
Master’s
Student
Contentious
reviews
Toshiki
Hirao
PhD
Student

View Slide

View Slide

And in the end
The love you take
Is equal to
The love you make

View Slide

And in the end
The love you take
Is equal to
The love you make
You get what you put
in(to code review)

View Slide

into code review
Author

View Slide

into code review
Author
Looks like
code to me!
+2
Rookie
Reviewer

View Slide

into code review
Author
Looks like
code to me!
+2
Rookie
Reviewer
Seasoned
Reviewer
Did you consider
alternative designs?
-1

View Slide

How does reviewing feedback evolve with
respect to…

View Slide

respect to…
Community
aging?

View Slide

respect to…
Community
aging?
Reviewer
experience?

View Slide

respect to…
Community
aging?
Reviewer
experience?
Reviewer
workload?

View Slide

Automatically analyzing topics that are
discussed in code reviews

View Slide

Get inline comments
506,950 inline
comments

View Slide

Get inline comments
506,950 inline
comments
Apply topic analysis Topic
1
Topic
2
Topic
3
Topic
1
Topic
2
Topic
3

View Slide

Get inline comments
506,950 inline
comments
Apply topic analysis Topic
1
Topic
2
Topic
3
Topic
1
Topic
2
Topic
3
Topic score analysis

View Slide

LDA automatically groups words into topics
based on co-occurrence in comments
How Does Code Reviewing Feedback Evolve? 11
Table 1: The labelled topics with their corresponding words and topic share score
CATEGORY TOPIC NAME SELECTED MOST RELEVANT
WORDS
TOPIC
SHARE (%)
Operations
Logging log, debug, warn, little, anywhere,
bit, level, dashboard
4.8
Release Engineering comment, version, package, install,
repo, option, config, copyright
4.4
Performance big, core, timeout, performance,
time, team, choice
5.5
Testing
Testing test, miss, unit, sentence, character,
functional, period
4.1
Improvement Request need, line, blank, pep, ref, alarm,
insert, codeblock
4.1
Social Communication
General Social Comm. ok, release, fix, yeah, liberty, thank,
rid, mitaka, ah
6.8
Confirmation ditto, token, context, label, auth,
align, hint, keystone
2.1
Server-related Discussion
User and Roles service, role, admin, user, rule, ten-
ant, security, snapshot
4.5
Network Issues port, network, ip, agent, router, ad-
dress, neutron, subnet
2.8
Node Issues node, task, state, migration, cluster,
disk, step, lock
4.1
Disk Image Issues image, class, glance, function, 3.3

View Slide

We read comments with high topic scores
and name the topics
WORDS
TOPIC
SHARE (%)
Operations
4.8
4.4
time, team, choice
5.5
Testing
functional, period
4.1
insert, codeblock
4.1
rid, mitaka, ah
6.8
2.1
4.5
2.8
disk, step, lock
4.1
ging features are used to aid in debugging and understanding system behaviour. To
ensure that the logs contain useful information, logging concerns are often raised
during the code review process.
How to enable TRACE log level in conﬁg ﬁle? Enabling the trace would enable
debug also?
Comment #136066, topic score(logging) = 0.94
The development of OPENSTACK involves contributions from several large soft-

View Slide

Then we group related topics into
categories
WORDS
TOPIC
SHARE (%)
Operations
4.8
4.4
time, team, choice
5.5
Testing
functional, period
4.1
insert, codeblock
4.1
rid, mitaka, ah
6.8
2.1
4.5
2.8
disk, step, lock
4.1

View Slide

PC1
PC2
Marginal topic distribution
2%
5%
10%
1
2
3
4
5
6
7
8
9
10
11
12
13
14 15
16
17
20
19
18
Communication
Testing
Software Design
Operations
Server-oriented
Version Control
File System
Documentation
Formatting
1 - Documentation and Testing
2 - Social Communication
3 - Testing
4 - Key Errors
5 - Logging
6 - Default Parameters
7 - Object names ?
8 - Release Engineering & Ops
9 - Time and Performance
10 - User & Roles
11 - Exception Handling
12 - Image Issues
13 - Version Control
14 - File System
15 - Network Issues
16 - Node Issues
17 - Conﬁrmation
18 - Requests
19 - Function Header
20 - Formatting
Legends

View Slide

As the community has aged, design-
oriented feedback has declined
Node Issues
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
*
Disk Image Issues ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
Software Design
Key Errors +
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● +
Exception Han-
dling
+
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Object Names
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
*
Function Headers +
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● *
Default and Pa-
rameters
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●

View Slide

On the other hand, discussion of networking
and patch formatting have grown
Social
Communication
General Social
Comm. ●
●
●
●
●
● ●
●
*
Conﬁrmation
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
Server-related
Discussion
User and Roles
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● +
Network Issues * ●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Node Issues ●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
*
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
Software Design
Key Errors +
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● +
●
● ●
●
●
●
●
● ●
Function Headers +
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● *
Default and Pa-
rameters
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
- Version Control
●
●
●
●
●
● ●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
- File System +
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
+
- Formatting * ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
+
- Documentation
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

View Slide

respect to…
Community
aging?
Reviewer
experience?
Reviewer
workload?
As OpenStack has aged,
the community has focused
less on patch design and
more on networking and
patch formatting

View Slide

System-oriented topics increase as reviewers
accrue experience with OpenStack
CATEGORY TOPIC NAME
TIME
TREND
TOPIC IMPACT
(OVER TIME)
EXP.
TREND
Operations
Logging
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
*
Release Engineer-
ing
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
+
Performance ●
●
●
●
●
● ● ●
●
●
●
●
●
● ●
● ●
●
●
●
●
*
Quality
Assurance
Testing ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Improvement Re-
quest
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
E
TIME
TREND
TOPIC IMPACT
(OVER TIME)
EXP.
TREND
TOPIC EXPERIENCE
IMPACT
WORKL
TREND
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
*
●
●
●
●
●
●●●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
*
ineer- ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
+
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●●
●●●
●●
●
+
●
●
●
●
●
● ● ●
●
●
●
●
●
● ●
● ●
●
●
●
●
* ●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
t Re-
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
● ●
TREND (OVER TIME) TREND
Operations
Logging
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
*
Release Engineer-
ing
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
+
Performance ●
●
●
●
●
● ● ●
●
●
●
●
●
● ●
● ●
●
●
●
●
*
Quality
Assurance
Testing ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Improvement Re-
quest
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
Social
Communication
General Social
Comm.
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
*
●
● ●
●
●
●
TREND (OVER TIME) TREND IMPACT TREND
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
*
●
●
●
●
●
●●●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
*
ineer- ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
+
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●●
●●●
●●
●
+
e ●
●
●
●
●
● ● ●
●
●
●
●
●
● ●
● ●
●
●
●
●
* ●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
nt Re-
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
Social ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
*
●
●●●
●
●●●●
●
●●●
●●
●●
●
●
●
●
●
●●
●
●●●
●
●
*
●
● ●
●
●
●
●
●
●
●
●
●●
●
●●●
●●
●
●
●
●
●
●
●●●
●
●
●
●
Communication Comm.
●
Conﬁrmation
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
Server-related
Discussion
User and Roles
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● +
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Node Issues ●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
*
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
Software Design
Key Errors +
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● +
Exception Han- +
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
n
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●●
●
●
●
●
●
●
●●●
●
●
●
●
oles
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● +
●
●
●
●
●●
●
●
●
●●●●
●
●
●
●●●●●●
●
●
●
●
●
●
●
●
+
ues * ●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
* ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
+
Issues ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
+
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● +
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
Han- +
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
*
Improvement Re-
quest
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
Social
Communication
General Social
Comm.
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
*
Conﬁrmation
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
Server-related
Discussion
User and Roles
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● +
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Node Issues ●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
*
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
CATEGORY TOPIC NAME
TIME
TREND
TOPIC IMPACT
(OVER TIME)
EXP.
TREND
Operations
Logging
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
*
Release Engineer-
ing
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
+
Performance ●
●
●
●
●
● ● ●
●
●
●
●
●
● ●
● ●
●
●
●
●
*
Quality
Assurance
Testing ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Improvement Re-
quest
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
Social
Communication
General Social
Comm.
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
*
Server-related
Discussion
User and Roles
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● +
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Node Issues ●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
*
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
Software Design
Key Errors +
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● +
Exception Han-
dling
+
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Object Names
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
*
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
* ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
+
Issues ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
+
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● +
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
Han- +
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
*
es
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
*
●
●
●
●●●
●
●
●
●●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
*
aders +
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● *
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
*
d Pa- ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●● ● ● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
* ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
+
Issues ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
+
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● +
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
Han- +
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
*
es
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
*
●
●
●
●●●
●
●
●
●●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
*
aders +
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● *
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
*
d Pa- ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●● ●
● ●
●
●●
●●
●●
● ● ●
●
●
●
●
●
Node Issues ●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
*
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
Software Design
Key Errors +
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● +
Exception Han-
dling
+
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Object Names
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
*
Function Headers +
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● *
Default and Pa- ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
E
TIME
TREND
TOPIC IMPACT
(OVER TIME)
EXP.
TREND
TOPIC EXPERIENCE
IMPACT
WORKL
TREND
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
*
●
●
●
●
●
●●●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
*
ineer- ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
+
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●●
●●●
●●
●
+
e ●
●
●
●
●
● ● ●
●
●
●
●
●
● ●
● ●
●
●
●
●
* ●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
t Re-
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
Social ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
*
●
●●●
●
●●●●
●
●●●
●●
●●
●
●
●
●
●
●●
●
●●●
●
●
*
●

View Slide

Low ROI topics like Formatting tend to
decrease as reviewers accrue experience
rameters ●
●
●
●
●
●
●
- Version Control
●
●
●
●
●
● ●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
- File System +
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
+
- Formatting * ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
+
- Documentation
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
where D(m) represents the set of review comm
other words, the topic impact metric measures
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
Control
●
●
●
●
●
● ●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
em +
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
+
●●
●●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
+
ng * ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
+ ●
●
●
●
●●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
+
tation
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
esents the set of review comments that are written
topic impact metric measures the proportion of r

View Slide

respect to…
Community
aging?
Reviewer
experience?
Reviewer
workload?
As OpenStack reviewers
accrue experience, they
tend to comment more on
system-oriented topics and
less on low ROI topics
patch formatting

View Slide

Reviewers with high experience scores
tend to also have a heavy workload score
( )
,
=0.68

View Slide

respect to…
Community
aging?
Reviewer
experience?
Reviewer
workload?
Similar observations
as were reported
for experience
patch formatting
As OpenStack reviewers
accrue experience, they
tend to comment more on
system-oriented topics and
less on low ROI topics

View Slide

View Slide

I wanna hold your hand!

View Slide

Community
evolution
Ray
Wen
Master’s
Student
Contentious
reviews
Toshiki
Hirao
PhD
Student

View Slide

View Slide

Mining Gerrit Reviews for Contentious Reviews and Community Evolution

Mining Gerrit Reviews for Contentious Reviews and Community Evolution

Shane McIntosh

Other Decks in Programming

Featured

Transcript