Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

You ask me for a contribution Well, you know We’re all doing what we can

Slide 3

Slide 3 text

Shane McIntosh Assistant Professor shane.mcintosh@mcgill.ca @SoftwareREBELs rebels.ece.mcgill.ca RE Software s EL B Mining Gerrit for Contentious Reviews & Community Evolution

Slide 4

Slide 4 text

RE Software s EL B RE Software s EL B

Slide 5

Slide 5 text

RE Software s EL B RE Software s EL B Repository Excavation

Slide 6

Slide 6 text

Development data is mainly kept for archival purposes 4

Slide 7

Slide 7 text

Development data is mainly kept for archival purposes 4 What if your software repositories could talk?

Slide 8

Slide 8 text

My students and I mine through historical development data in search of gold 5

Slide 9

Slide 9 text

My students and I mine through historical development data in search of gold 5 In which modules are future defects likely to appear? Baljinder Ghotra Chakkrit (Kla) T.

Slide 10

Slide 10 text

My students and I mine through historical development data in search of gold 5 In which modules are future defects likely to appear? Baljinder Ghotra Chakkrit (Kla) T. Can risky code be predicted at creation time? Takafumi Fukushima Jacob G. Barnett

Slide 11

Slide 11 text

RE Software s EL B RE Software s EL B Repository Excavation

Slide 12

Slide 12 text

RE Software s EL B RE Software s EL B report deploy build commit Build Engineering RE S B Repository Excavation

Slide 13

Slide 13 text

Release Engineering: Supporting release pipelines 7

Slide 14

Slide 14 text

Release Engineering: Supporting release pipelines Patch + + - - 7

Slide 15

Slide 15 text

Release Engineering: Supporting release pipelines 1. Integration Patch + + - - 7

Slide 16

Slide 16 text

Release Engineering: Supporting release pipelines 1. Integration Patch + + - - 2. Build 7

Slide 17

Slide 17 text

Release Engineering: Supporting release pipelines 1. Integration Patch + + - - New! 2. Build 3. Deployment 7

Slide 18

Slide 18 text

Release Engineering: Supporting release pipelines 1. Integration Patch + + - - New! 2. Build 3. Deployment 7 Decision support for modern release pipelines

Slide 19

Slide 19 text

1. Integration Patch 2. Build 3. Deployment Release Engineering: Supporting release pipelines + + - - New! 8

Slide 20

Slide 20 text

1. Integration Patch 2. Build 3. Deployment Release Engineering: Supporting release pipelines + + - - New! 8 Can we improve support for modern code review? Toshiki Hirao Ray Wen

Slide 21

Slide 21 text

1. Integration Patch 2. Build 3. Deployment Release Engineering: Supporting release pipelines + + - - New! 8 How can we optimize available CI/CD resources? Keheliya Gallaba Can we improve support for modern code review? Toshiki Hirao Ray Wen

Slide 22

Slide 22 text

RE Software s EL B report deploy build commit Build Engineering Repository Excavation

Slide 23

Slide 23 text

RE Software s EL B report deploy build commit Build Engineering Repository Excavation We are looking to form industrial partnerships!

Slide 24

Slide 24 text

10 An overview of the modern code review process

Slide 25

Slide 25 text

Author Module A Code change 10 An overview of the modern code review process

Slide 26

Slide 26 text

A code review tool (e.g., Gerrit) Author Module A Code change 10 An overview of the modern code review process

Slide 27

Slide 27 text

A code review tool (e.g., Gerrit) Author Module A Code change 10 An overview of the modern code review process

Slide 28

Slide 28 text

Module A Qt Review #27977 11 A code review tool (e.g., Gerrit) Author Reviewers may identify problems or suggest alternative approaches

Slide 29

Slide 29 text

Module A Qt Review #27977 11 A code review tool (e.g., Gerrit) Reviewer Author Reviewers may identify problems or suggest alternative approaches

Slide 30

Slide 30 text

Module A Qt Review #27977 11 A code review tool (e.g., Gerrit) Reviewer Shouldn't console.log() call the toString() method (where appropriate) on objects? Identifying a defect Author Reviewers may identify problems or suggest alternative approaches

Slide 31

Slide 31 text

Module A Qt Review #27977 11 A code review tool (e.g., Gerrit) Reviewer Shouldn't console.log() call the toString() method (where appropriate) on objects? Identifying a defect Author Reviewers may identify problems or suggest alternative approaches

Slide 32

Slide 32 text

Module A Qt Review #27977 11 A code review tool (e.g., Gerrit) Reviewer Shouldn't console.log() call the toString() method (where appropriate) on objects? Identifying a defect Reviewer I think it’s better to do var s = "{}" console.log(s) Suggesting a solution Author Reviewers may identify problems or suggest alternative approaches

Slide 33

Slide 33 text

Module A Qt Review #27977 11 A code review tool (e.g., Gerrit) Reviewer Shouldn't console.log() call the toString() method (where appropriate) on objects? Identifying a defect Reviewer I think it’s better to do var s = "{}" console.log(s) Suggesting a solution Author Update v2 Reviewers may identify problems or suggest alternative approaches

Slide 34

Slide 34 text

Author Module A Qt Review #35360 Providing updates to the code change Note that running qutlook will probably crash. I will push a patch set to fix the issue. Reviewer 12 A code review tool (e.g., Gerrit) Some reviewers even provide potential solutions to the issues they identify

Slide 35

Slide 35 text

Author Module A Qt Review #35360 Providing updates to the code change Note that running qutlook will probably crash. I will push a patch set to fix the issue. Reviewer 12 A code review tool (e.g., Gerrit) v2 Some reviewers even provide potential solutions to the issues they identify

Slide 36

Slide 36 text

Modern code review: Changes are guilty until proven innocent

Slide 37

Slide 37 text

(1) Upload change revision(s) (2) Execute sanity tests (3) Solicit peer feedback (4) Initiate integration request (5) Execute integration tests (6) Final integration Version Control System Pass Fail Legend Modern code review: Changes are guilty until proven innocent

Slide 38

Slide 38 text

Code review takes a lot of developer time and is not always useful 14

Slide 39

Slide 39 text

Code review takes a lot of developer time and is not always useful Impact of Peer Code Review on Peer Impression Formation: A Survey [Bosu and Carver, ESEM 2013] Devs spend 6 hours per week reviewing code 14

Slide 40

Slide 40 text

Code review takes a lot of developer time and is not always useful Impact of Peer Code Review on Peer Impression Formation: A Survey [Bosu and Carver, ESEM 2013] Devs spend 6 hours per week reviewing code Characteristics of Useful Code Reviews: An Empirical Study at Microsoft [Bosu et al., MSR 2015] 35% of review comments are “not useful” 14

Slide 41

Slide 41 text

Developers do not like to spend time on process

Slide 42

Slide 42 text

Mining Gerrit reviews to study reviewing processes

Slide 43

Slide 43 text

Mining Gerrit reviews to study reviewing processes Contentious reviews Toshiki Hirao PhD Student

Slide 44

Slide 44 text

Mining Gerrit reviews to study reviewing processes Community evolution Ray Wen Master’s Student Contentious reviews Toshiki Hirao PhD Student

Slide 45

Slide 45 text

Mining Gerrit reviews to study reviewing processes Community evolution Ray Wen Master’s Student Contentious reviews Toshiki Hirao PhD Student

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

And when the broken hearted people, Living in the world agree, There will be an answer, Let it be

Slide 48

Slide 48 text

And when the broken hearted people, Living in the world agree, There will be an answer, Let it be What happens when reviewers disagree?

Slide 49

Slide 49 text

Reviewer opinions about a patch may differ Author

Slide 50

Slide 50 text

Reviewer opinions about a patch may differ Author

Slide 51

Slide 51 text

Reviewer opinions about a patch may differ Reviewer 1 Author +2

Slide 52

Slide 52 text

Reviewer opinions about a patch may differ Reviewer 1 Author Reviewer 2 +2 –2

Slide 53

Slide 53 text

Reviewer opinions about a patch may differ Reviewer 1 Author Reviewer 2 +2 –2 How do we proceed?

Slide 54

Slide 54 text

Understanding code review in contentious patches

Slide 55

Slide 55 text

Understanding code review in contentious patches

Slide 56

Slide 56 text

Understanding code review in contentious patches How often?

Slide 57

Slide 57 text

Understanding code review in contentious patches How often? Trend over time?

Slide 58

Slide 58 text

Understanding code review in contentious patches How often? Trend over time? What drives abandonment? Start

Slide 59

Slide 59 text

Understanding code review in contentious patches How often? Trend over time? What drives abandonment? Start How are concerns resolved?

Slide 60

Slide 60 text

Selecting subject systems for analysis or: why did we study OpenStack? 0 22,500 45,000 67,500 90,000 Multi-reviewer patches Solo-reviewer patches 67% 21% 26% 17% 6%

Slide 61

Slide 61 text

Selecting subject systems for analysis or: why did we study OpenStack? 0 22,500 45,000 67,500 90,000 Multi-reviewer patches Solo-reviewer patches 67% 21% 26% 17% 6%

Slide 62

Slide 62 text

Understanding code review in contentious patches How often? Trend over time? What drives abandonment? Start How are concerns resolved?

Slide 63

Slide 63 text

Understanding code review in contentious patches How often? Trend over time? What drives abandonment? Start How are concerns resolved?

Slide 64

Slide 64 text

Classifying contentious reviews Include SP (+2) Include SN (-2) Include SN (-2) Yes No Yes No Yes No +2 +1 -2 -1 +2 +1 -1 +1 -2 -1 +1 -1 SP-SN SP-WN WP-SN WP-WN

Slide 65

Slide 65 text

Handling reviews with several revisions Contentious Pattern Non-contentious Pattern Revision 1 Revision 2 -1 B +2 A +1 A +1 C Revision 1 Revision 2 -2 A +1 B +1 C

Slide 66

Slide 66 text

Contentious reviews account for 26% of all OpenStack reviews 0 2,000 4,000 6,000 8,000 6,378 558 7,753 684 (1%) (13%) (1%) (11%) +2, -2 +2, -1 +1, -2 +1, -1 Number (and percentage) or reviews in each category

Slide 67

Slide 67 text

Contentious reviews with equal scores have a tendency to be integrated 0% 22.5% 45% 67.5% 90% 76% 29% 88% 54% +2, -2 +2, -1 +1, -2 +1, -1 Rate at which patches are eventually integrated

Slide 68

Slide 68 text

Understanding code review in contentious patches How often? Trend over time? What drives abandonment? Start How are concerns resolved? Contentious reviews are not rare Contention is not a show stopper

Slide 69

Slide 69 text

Understanding code review in contentious patches How often? Trend over time? What drives abandonment? Start How are concerns resolved? Contentious reviews are not rare Contention is not a show stopper

Slide 70

Slide 70 text

Contentious reviews are growing over time in absolute terms 1 10 100 1000 10000 1 3 5 7 9 11 15 19 23 27 31 35 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1 10 100 1000 10000 ● SP−SN SP−WN WP−SN WP−WN 11 011 11 12 12 012 12 012 12 13 13 013 13 013 13 14 14 014

Slide 71

Slide 71 text

Contentious reviews are also growing over time in comparative terms 0% 5% 10% 15% 20% 25% 30% 1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 Weak Nega;ve (SP-WN, WP-WN) Strong Nega;ve (SP-SN, WP-SN)

Slide 72

Slide 72 text

Integration rates of contentious reviews have roughly stabilized in recent periods 0.0 0.2 0.4 0.6 0.8 1.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0% 20% 40% 60% 80% 100% ● SP−SN SP−WN WP−SN WP−WN

Slide 73

Slide 73 text

Understanding code review in contentious patches How often? Trend over time? What drives abandonment? Start How are concerns resolved? Contentious reviews are not rare Contention is not a show stopper Contentious reviews have been growing over time Integration rates of contentious reviews have stabilized

Slide 74

Slide 74 text

Understanding code review in contentious patches How often? Trend over time? What drives abandonment? Start How are concerns resolved? Contentious reviews are not rare Contention is not a show stopper Contentious reviews have been growing over time Integration rates of contentious reviews have stabilized

Slide 75

Slide 75 text

Qualitative analysis: Selecting a representative sample 684 reviews with +2 and -2 Sample of 362 reviews Random sampling for 95% CL 5% CI

Slide 76

Slide 76 text

Qualitative analysis: Selecting a representative sample 684 reviews with +2 and -2 Sample of 362 reviews Random sampling for 95% CL 5% CI

Slide 77

Slide 77 text

Qualitative analysis: Manual classification of reviews Sample of 362 reviews

Slide 78

Slide 78 text

Qualitative analysis: Manual classification of reviews Sample of 362 reviews Looks like a test coverage issue

Slide 79

Slide 79 text

Qualitative analysis: Manual classification of reviews Sample of 362 reviews Looks like a test coverage issue Agreed!

Slide 80

Slide 80 text

Qualitative analysis: Manual classification of reviews Sample of 362 reviews Looks like a test coverage issue Agreed! Test coverage

Slide 81

Slide 81 text

Qualitative analysis: Lifting tags to cohesive subhierarchies Test coverage Already fixed Side effects Alternative solution

Slide 82

Slide 82 text

Qualitative analysis: Lifting tags to cohesive subhierarchies Test coverage Already fixed Side effects Alternative solution Design Unnecessary contribution QA

Slide 83

Slide 83 text

Qualitative analysis: Lifting tags to cohesive subhierarchies Test coverage Already fixed Side effects Alternative solution Design Unnecessary contribution QA Internal concerns External concerns

Slide 84

Slide 84 text

Abandonment is often due to external concerns! Abandon External Concerns Internal Concerns Integration Policy Compliance Unnecessary Fix Integration Planning Lack of Interest Testing Implementation Design Unclear Intention Not an Issue Already Fixed Blueprint Release Schedule Patch Dependency Lost by a reviewer Lost by an author Side Effect Shallow Fix Flawed Changes Alternative Solution Backward Compatibility Patch Size Test Failure Test Coverage 59% (102/173) 21 12 24 6 1 1 32 17 5 6 14 6 1 173 10 41% (71/173) 58% (59/105) 7% (7/105) 87% (62/71) 10% (7/71) 3% (2/71) 25% (25/105) 5 Branch Placement Squashing Commits 6 5 11% (11/105) 1

Slide 85

Slide 85 text

Abandonment is often due to external concerns! Abandon External Concerns Internal Concerns Integration Policy Compliance Unnecessary Fix Integration Planning Lack of Interest Testing Implementation Design Unclear Intention Not an Issue Already Fixed Blueprint Release Schedule Patch Dependency Lost by a reviewer Lost by an author Side Effect Shallow Fix Flawed Changes Alternative Solution Backward Compatibility Patch Size Test Failure Test Coverage 59% (102/173) 21 12 24 6 1 1 32 17 5 6 14 6 1 173 10 41% (71/173) 58% (59/105) 7% (7/105) 87% (62/71) 10% (7/71) 3% (2/71) 25% (25/105) 5 Branch Placement Squashing Commits 6 5 11% (11/105) 1

Slide 86

Slide 86 text

Understanding code review in contentious patches How often? Trend over time? What drives abandonment? Start How are concerns resolved? Contentious reviews are not rare Contention is not a show stopper Contentious reviews have been growing over time Integration rates of contentious reviews have stabilized Concerns beyond patch scope are more strongly linked with abandonment than concerns within patch scope

Slide 87

Slide 87 text

Understanding code review in contentious patches How often? Trend over time? What drives abandonment? Start How are concerns resolved? Contentious reviews are not rare Contention is not a show stopper Contentious reviews have been growing over time Integration rates of contentious reviews have stabilized Concerns beyond patch scope are more strongly linked with abandonment than concerns within patch scope

Slide 88

Slide 88 text

Contention is more often resolved without altering the patch! Integrate Directly Addressing Indirectly Addressing Legal Problems Unnecessary Fix Design Implementation Testing Integration Policy Compliance Alternative Solution Side Effect Shallow Fix Flawed Changes Backward Compatibility Test Coverage Test Failure Squashing Commits Branch Placement 44% (83/189) 2 1 3 9 21 7 4 65 12 3 3 3 70% (58/83) 189 Withdrawal of Negative Score Integration Planning Persuasion Self-change Release Schedule Blueprint Patch Dependency 10 56% (106/189) 4% (3/83) 7% (6/83) 5% (4/83) 28% (30/106) 72% (76/106) Legal Issues Unclear Intention 12% (10/83) 2% (2/83) 4 4 38

Slide 89

Slide 89 text

Contention is more often resolved without altering the patch! Integrate Directly Addressing Indirectly Addressing Legal Problems Unnecessary Fix Design Implementation Testing Integration Policy Compliance Alternative Solution Side Effect Shallow Fix Flawed Changes Backward Compatibility Test Coverage Test Failure Squashing Commits Branch Placement 44% (83/189) 2 1 3 9 21 7 4 65 12 3 3 3 70% (58/83) 189 Withdrawal of Negative Score Integration Planning Persuasion Self-change Release Schedule Blueprint Patch Dependency 10 56% (106/189) 4% (3/83) 7% (6/83) 5% (4/83) 28% (30/106) 72% (76/106) Legal Issues Unclear Intention 12% (10/83) 2% (2/83) 4 4 38

Slide 90

Slide 90 text

Understanding code review in contentious patches How often? Trend over time? What drives abandonment? Start How are concerns resolved? Contentious reviews are not rare Contention is not a show stopper Contentious reviews have been growing over time Integration rates of contentious reviews have stabilized Concerns beyond patch scope are more strongly linked with abandonment than concerns within patch scope Concerns are often addressed through negotiation or scheduling

Slide 91

Slide 91 text

No content

Slide 92

Slide 92 text

Life is very short, and there’s no time, for fussing and fighting, my friends

Slide 93

Slide 93 text

Mining Gerrit reviews to study reviewing processes Community evolution Ray Wen Master’s Student Contentious reviews Toshiki Hirao PhD Student

Slide 94

Slide 94 text

Mining Gerrit reviews to study reviewing processes Community evolution Ray Wen Master’s Student Contentious reviews Toshiki Hirao PhD Student

Slide 95

Slide 95 text

No content

Slide 96

Slide 96 text

And in the end The love you take Is equal to The love you make

Slide 97

Slide 97 text

And in the end The love you take Is equal to The love you make You get what you put in(to code review)

Slide 98

Slide 98 text

You get what you put into code review Author

Slide 99

Slide 99 text

You get what you put into code review Author

Slide 100

Slide 100 text

You get what you put into code review Author Looks like code to me! +2 Rookie Reviewer

Slide 101

Slide 101 text

You get what you put into code review Author Looks like code to me! +2 Rookie Reviewer Seasoned Reviewer Did you consider alternative designs? -1

Slide 102

Slide 102 text

How does reviewing feedback evolve with respect to…

Slide 103

Slide 103 text

How does reviewing feedback evolve with respect to… Community aging?

Slide 104

Slide 104 text

How does reviewing feedback evolve with respect to… Community aging? Reviewer experience?

Slide 105

Slide 105 text

How does reviewing feedback evolve with respect to… Community aging? Reviewer experience? Reviewer workload?

Slide 106

Slide 106 text

How does reviewing feedback evolve with respect to… Community aging? Reviewer experience? Reviewer workload?

Slide 107

Slide 107 text

How does reviewing feedback evolve with respect to… Community aging? Reviewer experience? Reviewer workload?

Slide 108

Slide 108 text

Automatically analyzing topics that are discussed in code reviews

Slide 109

Slide 109 text

Automatically analyzing topics that are discussed in code reviews Get inline comments 506,950 inline comments

Slide 110

Slide 110 text

Automatically analyzing topics that are discussed in code reviews Get inline comments 506,950 inline comments Apply topic analysis Topic 1 Topic 2 Topic 3 Topic 1 Topic 2 Topic 3

Slide 111

Slide 111 text

Automatically analyzing topics that are discussed in code reviews Get inline comments 506,950 inline comments Apply topic analysis Topic 1 Topic 2 Topic 3 Topic 1 Topic 2 Topic 3 Topic score analysis

Slide 112

Slide 112 text

LDA automatically groups words into topics based on co-occurrence in comments How Does Code Reviewing Feedback Evolve? 11 Table 1: The labelled topics with their corresponding words and topic share score CATEGORY TOPIC NAME SELECTED MOST RELEVANT WORDS TOPIC SHARE (%) Operations Logging log, debug, warn, little, anywhere, bit, level, dashboard 4.8 Release Engineering comment, version, package, install, repo, option, config, copyright 4.4 Performance big, core, timeout, performance, time, team, choice 5.5 Testing Testing test, miss, unit, sentence, character, functional, period 4.1 Improvement Request need, line, blank, pep, ref, alarm, insert, codeblock 4.1 Social Communication General Social Comm. ok, release, fix, yeah, liberty, thank, rid, mitaka, ah 6.8 Confirmation ditto, token, context, label, auth, align, hint, keystone 2.1 Server-related Discussion User and Roles service, role, admin, user, rule, ten- ant, security, snapshot 4.5 Network Issues port, network, ip, agent, router, ad- dress, neutron, subnet 2.8 Node Issues node, task, state, migration, cluster, disk, step, lock 4.1 Disk Image Issues image, class, glance, function, 3.3

Slide 113

Slide 113 text

We read comments with high topic scores and name the topics How Does Code Reviewing Feedback Evolve? 11 Table 1: The labelled topics with their corresponding words and topic share score CATEGORY TOPIC NAME SELECTED MOST RELEVANT WORDS TOPIC SHARE (%) Operations Logging log, debug, warn, little, anywhere, bit, level, dashboard 4.8 Release Engineering comment, version, package, install, repo, option, config, copyright 4.4 Performance big, core, timeout, performance, time, team, choice 5.5 Testing Testing test, miss, unit, sentence, character, functional, period 4.1 Improvement Request need, line, blank, pep, ref, alarm, insert, codeblock 4.1 Social Communication General Social Comm. ok, release, fix, yeah, liberty, thank, rid, mitaka, ah 6.8 Confirmation ditto, token, context, label, auth, align, hint, keystone 2.1 Server-related Discussion User and Roles service, role, admin, user, rule, ten- ant, security, snapshot 4.5 Network Issues port, network, ip, agent, router, ad- dress, neutron, subnet 2.8 Node Issues node, task, state, migration, cluster, disk, step, lock 4.1 Disk Image Issues image, class, glance, function, 3.3 How Does Code Reviewing Feedback Evolve? 13 ging features are used to aid in debugging and understanding system behaviour. To ensure that the logs contain useful information, logging concerns are often raised during the code review process. How to enable TRACE log level in config file? Enabling the trace would enable debug also? Comment #136066, topic score(logging) = 0.94 The development of OPENSTACK involves contributions from several large soft-

Slide 114

Slide 114 text

Then we group related topics into categories How Does Code Reviewing Feedback Evolve? 11 Table 1: The labelled topics with their corresponding words and topic share score CATEGORY TOPIC NAME SELECTED MOST RELEVANT WORDS TOPIC SHARE (%) Operations Logging log, debug, warn, little, anywhere, bit, level, dashboard 4.8 Release Engineering comment, version, package, install, repo, option, config, copyright 4.4 Performance big, core, timeout, performance, time, team, choice 5.5 Testing Testing test, miss, unit, sentence, character, functional, period 4.1 Improvement Request need, line, blank, pep, ref, alarm, insert, codeblock 4.1 Social Communication General Social Comm. ok, release, fix, yeah, liberty, thank, rid, mitaka, ah 6.8 Confirmation ditto, token, context, label, auth, align, hint, keystone 2.1 Server-related Discussion User and Roles service, role, admin, user, rule, ten- ant, security, snapshot 4.5 Network Issues port, network, ip, agent, router, ad- dress, neutron, subnet 2.8 Node Issues node, task, state, migration, cluster, disk, step, lock 4.1 Disk Image Issues image, class, glance, function, 3.3

Slide 115

Slide 115 text

PC1 PC2 Marginal topic distribution 2% 5% 10% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 20 19 18 Communication Testing Software Design Operations Server-oriented Version Control File System Documentation Formatting 1 - Documentation and Testing 2 - Social Communication 3 - Testing 4 - Key Errors 5 - Logging 6 - Default Parameters 7 - Object names ? 8 - Release Engineering & Ops 9 - Time and Performance 10 - User & Roles 11 - Exception Handling 12 - Image Issues 13 - Version Control 14 - File System 15 - Network Issues 16 - Node Issues 17 - Confirmation 18 - Requests 19 - Function Header 20 - Formatting Legends

Slide 116

Slide 116 text

As the community has aged, design- oriented feedback has declined Node Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Disk Image Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Software Design Key Errors + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Exception Han- dling + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Object Names ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Function Headers + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Default and Pa- rameters ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

Slide 117

Slide 117 text

On the other hand, discussion of networking and patch formatting have grown Social Communication General Social Comm. ● ● ● ● ● ● ● ● * Confirmation ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Server-related Discussion User and Roles ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Network Issues * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Node Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Disk Image Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Software Design Key Errors + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ● ● ● ● ● ● ● ● ● Function Headers + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Default and Pa- rameters ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● - Version Control ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● - File System + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + - Formatting * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + - Documentation ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

Slide 118

Slide 118 text

How does reviewing feedback evolve with respect to… Community aging? Reviewer experience? Reviewer workload? As OpenStack has aged, the community has focused less on patch design and more on networking and patch formatting

Slide 119

Slide 119 text

How does reviewing feedback evolve with respect to… Community aging? Reviewer experience? Reviewer workload? As OpenStack has aged, the community has focused less on patch design and more on networking and patch formatting

Slide 120

Slide 120 text

System-oriented topics increase as reviewers accrue experience with OpenStack CATEGORY TOPIC NAME TIME TREND TOPIC IMPACT (OVER TIME) EXP. TREND Operations Logging ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Release Engineer- ing ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Performance ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Quality Assurance Testing ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Improvement Re- quest ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● E TIME TREND TOPIC IMPACT (OVER TIME) EXP. TREND TOPIC EXPERIENCE IMPACT WORKL TREND ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ● ●●●●●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● * ineer- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ●● ●●● ●● ● + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● t Re- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● TREND (OVER TIME) TREND Operations Logging ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Release Engineer- ing ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Performance ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Quality Assurance Testing ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Improvement Re- quest ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Social Communication General Social Comm. ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ● ● TREND (OVER TIME) TREND IMPACT TREND ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ● ●●●●●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● * ineer- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ●● ●●● ●● ● + e ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● nt Re- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● Social ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ●●● ● ●●●● ● ●●● ●● ●● ● ● ● ● ● ●● ● ●●● ● ● * ● ● ● ● ● ● ● ● ● ● ● ●● ● ●●● ●● ● ● ● ● ● ● ●●● ● ● ● ● Communication Comm. ● Confirmation ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Server-related Discussion User and Roles ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Network Issues * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Node Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Disk Image Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Software Design Key Errors + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Exception Han- + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● n ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●●● ●● ● ● ● ● ● ● ●●● ● ● ● ● oles ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ● ● ● ● ●● ● ● ● ●●●● ● ● ● ●●●●●● ● ● ● ● ● ● ● ● + ues * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ● ●● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● Han- + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● * Improvement Re- quest ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Social Communication General Social Comm. ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Confirmation ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Server-related Discussion User and Roles ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Network Issues * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Node Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Disk Image Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● CATEGORY TOPIC NAME TIME TREND TOPIC IMPACT (OVER TIME) EXP. TREND Operations Logging ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Release Engineer- ing ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Performance ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Quality Assurance Testing ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Improvement Re- quest ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Social Communication General Social Comm. ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Server-related Discussion User and Roles ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Network Issues * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Node Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Disk Image Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Software Design Key Errors + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Exception Han- dling + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Object Names ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ● ●● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● Han- + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● * es ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ●●● ● ● ● ●●●●● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● * aders + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ●●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● * d Pa- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ● ●● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● Han- + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● * es ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ●●● ● ● ● ●●●●● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● * aders + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ●●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● * d Pa- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ●● ●● ● ● ● ● ● ● ● ● Node Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Disk Image Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Software Design Key Errors + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Exception Han- dling + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Object Names ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Function Headers + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Default and Pa- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● E TIME TREND TOPIC IMPACT (OVER TIME) EXP. TREND TOPIC EXPERIENCE IMPACT WORKL TREND ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ● ●●●●●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● * ineer- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ●● ●●● ●● ● + e ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● t Re- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● Social ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ●●● ● ●●●● ● ●●● ●● ●● ● ● ● ● ● ●● ● ●●● ● ● * ●

Slide 121

Slide 121 text

System-oriented topics increase as reviewers accrue experience with OpenStack CATEGORY TOPIC NAME TIME TREND TOPIC IMPACT (OVER TIME) EXP. TREND Operations Logging ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Release Engineer- ing ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Performance ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Quality Assurance Testing ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Improvement Re- quest ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● E TIME TREND TOPIC IMPACT (OVER TIME) EXP. TREND TOPIC EXPERIENCE IMPACT WORKL TREND ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ● ●●●●●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● * ineer- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ●● ●●● ●● ● + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● t Re- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● TREND (OVER TIME) TREND Operations Logging ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Release Engineer- ing ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Performance ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Quality Assurance Testing ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Improvement Re- quest ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Social Communication General Social Comm. ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ● ● TREND (OVER TIME) TREND IMPACT TREND ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ● ●●●●●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● * ineer- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ●● ●●● ●● ● + e ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● nt Re- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● Social ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ●●● ● ●●●● ● ●●● ●● ●● ● ● ● ● ● ●● ● ●●● ● ● * ● ● ● ● ● ● ● ● ● ● ● ●● ● ●●● ●● ● ● ● ● ● ● ●●● ● ● ● ● Communication Comm. ● Confirmation ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Server-related Discussion User and Roles ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Network Issues * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Node Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Disk Image Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Software Design Key Errors + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Exception Han- + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● n ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●●● ●● ● ● ● ● ● ● ●●● ● ● ● ● oles ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ● ● ● ● ●● ● ● ● ●●●● ● ● ● ●●●●●● ● ● ● ● ● ● ● ● + ues * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ● ●● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● Han- + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● * Improvement Re- quest ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Social Communication General Social Comm. ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Confirmation ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Server-related Discussion User and Roles ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Network Issues * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Node Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Disk Image Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● CATEGORY TOPIC NAME TIME TREND TOPIC IMPACT (OVER TIME) EXP. TREND Operations Logging ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Release Engineer- ing ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Performance ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Quality Assurance Testing ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Improvement Re- quest ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Social Communication General Social Comm. ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Server-related Discussion User and Roles ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Network Issues * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Node Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Disk Image Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Software Design Key Errors + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Exception Han- dling + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Object Names ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ● ●● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● Han- + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● * es ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ●●● ● ● ● ●●●●● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● * aders + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ●●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● * d Pa- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ● ●● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● Han- + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● * es ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ●●● ● ● ● ●●●●● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● * aders + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ●●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● * d Pa- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ●● ●● ● ● ● ● ● ● ● ● Node Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Disk Image Issues ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Software Design Key Errors + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + Exception Han- dling + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Object Names ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Function Headers + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * Default and Pa- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● E TIME TREND TOPIC IMPACT (OVER TIME) EXP. TREND TOPIC EXPERIENCE IMPACT WORKL TREND ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ● ● ●●●●●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● * ineer- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ●● ●●● ●● ● + e ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● t Re- ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● Social ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● * ● ●●● ● ●●●● ● ●●● ●● ●● ● ● ● ● ● ●● ● ●●● ● ● * ●

Slide 122

Slide 122 text

Low ROI topics like Formatting tend to decrease as reviewers accrue experience rameters ● ● ● ● ● ● ● - Version Control ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● - File System + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + - Formatting * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + - Documentation ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● where D(m) represents the set of review comm other words, the topic impact metric measures ● ● ● ● ● ● ● ● ● ●●●●● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● Control ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● em + ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ●● ●●● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● + ng * ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● + ● ● ● ● ●● ● ● ● ●● ●● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● + tation ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● esents the set of review comments that are written topic impact metric measures the proportion of r

Slide 123

Slide 123 text

How does reviewing feedback evolve with respect to… Community aging? Reviewer experience? Reviewer workload? As OpenStack reviewers accrue experience, they tend to comment more on system-oriented topics and less on low ROI topics As OpenStack has aged, the community has focused less on patch design and more on networking and patch formatting

Slide 124

Slide 124 text

How does reviewing feedback evolve with respect to… Community aging? Reviewer experience? Reviewer workload? As OpenStack reviewers accrue experience, they tend to comment more on system-oriented topics and less on low ROI topics As OpenStack has aged, the community has focused less on patch design and more on networking and patch formatting

Slide 125

Slide 125 text

Reviewers with high experience scores tend to also have a heavy workload score ( ) , =0.68

Slide 126

Slide 126 text

How does reviewing feedback evolve with respect to… Community aging? Reviewer experience? Reviewer workload? Similar observations as were reported for experience As OpenStack has aged, the community has focused less on patch design and more on networking and patch formatting As OpenStack reviewers accrue experience, they tend to comment more on system-oriented topics and less on low ROI topics

Slide 127

Slide 127 text

No content

Slide 128

Slide 128 text

I wanna hold your hand!

Slide 129

Slide 129 text

Mining Gerrit reviews to study reviewing processes Community evolution Ray Wen Master’s Student Contentious reviews Toshiki Hirao PhD Student

Slide 130

Slide 130 text

Mining Gerrit reviews to study reviewing processes Community evolution Ray Wen Master’s Student Contentious reviews Toshiki Hirao PhD Student

Slide 131

Slide 131 text

No content

Slide 132

Slide 132 text

No content

Slide 133

Slide 133 text

No content

Slide 134

Slide 134 text

No content

Slide 135

Slide 135 text

No content

Slide 136

Slide 136 text

No content

Slide 137

Slide 137 text

No content