Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mining Gerrit Reviews for Contentious Reviews and Community Evolution

Mining Gerrit Reviews for Contentious Reviews and Community Evolution

Software teams that use Gerrit are quickly accumulating data about their code reviewing processes. The Software REBELs—a research lab at McGill University—mine code review data to study topics like the impact that code review practices have on software release and design quality. Our more recent work mines code review data to study the reviewing process itself. In this talk, I will describe the results of two empirical studies of data that we collected from the Gerrit instances of the OpenStack project. The first study aims to understand the reviews where reviewers disagree about a patch. The second study follows how the concerns that reviewers raise evolve as the OpenStack community ages and individual reviews accrue experience.

D9463576a8b69ad67bac941fbf5cceb4?s=128

Shane McIntosh

October 03, 2017
Tweet

Transcript

  1. None
  2. You ask me for a contribution Well, you know We’re

    all doing what we can
  3. Shane McIntosh Assistant Professor shane.mcintosh@mcgill.ca @SoftwareREBELs rebels.ece.mcgill.ca RE Software s

    EL B Mining Gerrit for Contentious Reviews & Community Evolution
  4. RE Software s EL B RE Software s EL B

  5. RE Software s EL B RE Software s EL B

    Repository Excavation
  6. Development data is mainly kept for archival purposes 4

  7. Development data is mainly kept for archival purposes 4 What

    if your software repositories could talk?
  8. My students and I mine through historical development data in

    search of gold 5
  9. My students and I mine through historical development data in

    search of gold 5 In which modules are future defects likely to appear? Baljinder Ghotra Chakkrit (Kla) T.
  10. My students and I mine through historical development data in

    search of gold 5 In which modules are future defects likely to appear? Baljinder Ghotra Chakkrit (Kla) T. Can risky code be predicted at creation time? Takafumi Fukushima Jacob G. Barnett
  11. RE Software s EL B RE Software s EL B

    Repository Excavation
  12. RE Software s EL B RE Software s EL B

    report deploy build commit Build Engineering RE S B Repository Excavation
  13. Release Engineering: Supporting release pipelines 7

  14. Release Engineering: Supporting release pipelines Patch + + - -

    7
  15. Release Engineering: Supporting release pipelines 1. Integration Patch + +

    - - 7
  16. Release Engineering: Supporting release pipelines 1. Integration Patch + +

    - - 2. Build 7
  17. Release Engineering: Supporting release pipelines 1. Integration Patch + +

    - - New! 2. Build 3. Deployment 7
  18. Release Engineering: Supporting release pipelines 1. Integration Patch + +

    - - New! 2. Build 3. Deployment 7 Decision support for modern release pipelines
  19. 1. Integration Patch 2. Build 3. Deployment Release Engineering: Supporting

    release pipelines + + - - New! 8
  20. 1. Integration Patch 2. Build 3. Deployment Release Engineering: Supporting

    release pipelines + + - - New! 8 Can we improve support for modern code review? Toshiki Hirao Ray Wen
  21. 1. Integration Patch 2. Build 3. Deployment Release Engineering: Supporting

    release pipelines + + - - New! 8 How can we optimize available CI/CD resources? Keheliya Gallaba Can we improve support for modern code review? Toshiki Hirao Ray Wen
  22. RE Software s EL B report deploy build commit Build

    Engineering Repository Excavation
  23. RE Software s EL B report deploy build commit Build

    Engineering Repository Excavation We are looking to form industrial partnerships!
  24. 10 An overview of the modern code review process

  25. Author Module A Code change 10 An overview of the

    modern code review process
  26. A code review tool (e.g., Gerrit) Author Module A Code

    change 10 An overview of the modern code review process
  27. A code review tool (e.g., Gerrit) Author Module A Code

    change 10 An overview of the modern code review process
  28. Module A Qt Review #27977 11 A code review tool

    (e.g., Gerrit) Author Reviewers may identify problems or suggest alternative approaches
  29. Module A Qt Review #27977 11 A code review tool

    (e.g., Gerrit) Reviewer Author Reviewers may identify problems or suggest alternative approaches
  30. Module A Qt Review #27977 11 A code review tool

    (e.g., Gerrit) Reviewer Shouldn't console.log() call the toString() method (where appropriate) on objects? Identifying a defect Author Reviewers may identify problems or suggest alternative approaches
  31. Module A Qt Review #27977 11 A code review tool

    (e.g., Gerrit) Reviewer Shouldn't console.log() call the toString() method (where appropriate) on objects? Identifying a defect Author Reviewers may identify problems or suggest alternative approaches
  32. Module A Qt Review #27977 11 A code review tool

    (e.g., Gerrit) Reviewer Shouldn't console.log() call the toString() method (where appropriate) on objects? Identifying a defect Reviewer I think it’s better to do var s = "{}" console.log(s) Suggesting a solution Author Reviewers may identify problems or suggest alternative approaches
  33. Module A Qt Review #27977 11 A code review tool

    (e.g., Gerrit) Reviewer Shouldn't console.log() call the toString() method (where appropriate) on objects? Identifying a defect Reviewer I think it’s better to do var s = "{}" console.log(s) Suggesting a solution Author Update v2 Reviewers may identify problems or suggest alternative approaches
  34. Author Module A Qt Review #35360 Providing updates to the

    code change Note that running qutlook will probably crash. I will push a patch set to fix the issue. Reviewer 12 A code review tool (e.g., Gerrit) Some reviewers even provide potential solutions to the issues they identify
  35. Author Module A Qt Review #35360 Providing updates to the

    code change Note that running qutlook will probably crash. I will push a patch set to fix the issue. Reviewer 12 A code review tool (e.g., Gerrit) v2 Some reviewers even provide potential solutions to the issues they identify
  36. Modern code review: Changes are guilty until proven innocent

  37. (1) Upload change revision(s) (2) Execute sanity tests (3) Solicit

    peer feedback (4) Initiate integration request (5) Execute integration tests (6) Final integration Version Control System Pass Fail Legend Modern code review: Changes are guilty until proven innocent
  38. Code review takes a lot of developer time and is

    not always useful 14
  39. Code review takes a lot of developer time and is

    not always useful Impact of Peer Code Review on Peer Impression Formation: A Survey [Bosu and Carver, ESEM 2013] Devs spend 6 hours per week reviewing code 14
  40. Code review takes a lot of developer time and is

    not always useful Impact of Peer Code Review on Peer Impression Formation: A Survey [Bosu and Carver, ESEM 2013] Devs spend 6 hours per week reviewing code Characteristics of Useful Code Reviews: An Empirical Study at Microsoft [Bosu et al., MSR 2015] 35% of review comments are “not useful” 14
  41. Developers do not like to spend time on process

  42. Mining Gerrit reviews to study reviewing processes

  43. Mining Gerrit reviews to study reviewing processes Contentious reviews Toshiki

    Hirao PhD Student
  44. Mining Gerrit reviews to study reviewing processes Community evolution Ray

    Wen Master’s Student Contentious reviews Toshiki Hirao PhD Student
  45. Mining Gerrit reviews to study reviewing processes Community evolution Ray

    Wen Master’s Student Contentious reviews Toshiki Hirao PhD Student
  46. None
  47. And when the broken hearted people, Living in the world

    agree, There will be an answer, Let it be
  48. And when the broken hearted people, Living in the world

    agree, There will be an answer, Let it be What happens when reviewers disagree?
  49. Reviewer opinions about a patch may differ Author

  50. Reviewer opinions about a patch may differ Author

  51. Reviewer opinions about a patch may differ Reviewer 1 Author

    +2
  52. Reviewer opinions about a patch may differ Reviewer 1 Author

    Reviewer 2 +2 –2
  53. Reviewer opinions about a patch may differ Reviewer 1 Author

    Reviewer 2 +2 –2 How do we proceed?
  54. Understanding code review in contentious patches

  55. Understanding code review in contentious patches

  56. Understanding code review in contentious patches How often?

  57. Understanding code review in contentious patches How often? Trend over

    time?
  58. Understanding code review in contentious patches How often? Trend over

    time? What drives abandonment? Start
  59. Understanding code review in contentious patches How often? Trend over

    time? What drives abandonment? Start How are concerns resolved?
  60. Selecting subject systems for analysis or: why did we study

    OpenStack? 0 22,500 45,000 67,500 90,000 Multi-reviewer patches Solo-reviewer patches 67% 21% 26% 17% 6%
  61. Selecting subject systems for analysis or: why did we study

    OpenStack? 0 22,500 45,000 67,500 90,000 Multi-reviewer patches Solo-reviewer patches 67% 21% 26% 17% 6%
  62. Understanding code review in contentious patches How often? Trend over

    time? What drives abandonment? Start How are concerns resolved?
  63. Understanding code review in contentious patches How often? Trend over

    time? What drives abandonment? Start How are concerns resolved?
  64. Classifying contentious reviews Include SP (+2) Include SN (-2) Include

    SN (-2) Yes No Yes No Yes No +2 +1 -2 -1 +2 +1 -1 +1 -2 -1 +1 -1 SP-SN SP-WN WP-SN WP-WN
  65. Handling reviews with several revisions Contentious Pattern Non-contentious Pattern Revision

    1 Revision 2 -1 B +2 A +1 A +1 C Revision 1 Revision 2 -2 A +1 B +1 C
  66. Contentious reviews account for 26% of all OpenStack reviews 0

    2,000 4,000 6,000 8,000 6,378 558 7,753 684 (1%) (13%) (1%) (11%) +2, -2 +2, -1 +1, -2 +1, -1 Number (and percentage) or reviews in each category
  67. Contentious reviews with equal scores have a tendency to be

    integrated 0% 22.5% 45% 67.5% 90% 76% 29% 88% 54% +2, -2 +2, -1 +1, -2 +1, -1 Rate at which patches are eventually integrated
  68. Understanding code review in contentious patches How often? Trend over

    time? What drives abandonment? Start How are concerns resolved? Contentious reviews are not rare Contention is not a show stopper
  69. Understanding code review in contentious patches How often? Trend over

    time? What drives abandonment? Start How are concerns resolved? Contentious reviews are not rare Contention is not a show stopper
  70. Contentious reviews are growing over time in absolute terms 1

    10 100 1000 10000 1 3 5 7 9 11 15 19 23 27 31 35 • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • 1 10 100 1000 10000 • SP−SN SP−WN WP−SN WP−WN 11 011 11 12 12 012 12 012 12 13 13 013 13 013 13 14 14 014
  71. Contentious reviews are also growing over time in comparative terms

    0% 5% 10% 15% 20% 25% 30% 1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 Weak Nega;ve (SP-WN, WP-WN) Strong Nega;ve (SP-SN, WP-SN)
  72. Integration rates of contentious reviews have roughly stabilized in recent

    periods 0.0 0.2 0.4 0.6 0.8 1.0 • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • 0% 20% 40% 60% 80% 100% • SP−SN SP−WN WP−SN WP−WN
  73. Understanding code review in contentious patches How often? Trend over

    time? What drives abandonment? Start How are concerns resolved? Contentious reviews are not rare Contention is not a show stopper Contentious reviews have been growing over time Integration rates of contentious reviews have stabilized
  74. Understanding code review in contentious patches How often? Trend over

    time? What drives abandonment? Start How are concerns resolved? Contentious reviews are not rare Contention is not a show stopper Contentious reviews have been growing over time Integration rates of contentious reviews have stabilized
  75. Qualitative analysis: Selecting a representative sample 684 reviews with +2

    and -2 Sample of 362 reviews Random sampling for 95% CL 5% CI
  76. Qualitative analysis: Selecting a representative sample 684 reviews with +2

    and -2 Sample of 362 reviews Random sampling for 95% CL 5% CI
  77. Qualitative analysis: Manual classification of reviews Sample of 362 reviews

  78. Qualitative analysis: Manual classification of reviews Sample of 362 reviews

    Looks like a test coverage issue
  79. Qualitative analysis: Manual classification of reviews Sample of 362 reviews

    Looks like a test coverage issue Agreed!
  80. Qualitative analysis: Manual classification of reviews Sample of 362 reviews

    Looks like a test coverage issue Agreed! Test coverage
  81. Qualitative analysis: Lifting tags to cohesive subhierarchies Test coverage Already

    fixed Side effects Alternative solution
  82. Qualitative analysis: Lifting tags to cohesive subhierarchies Test coverage Already

    fixed Side effects Alternative solution Design Unnecessary contribution QA
  83. Qualitative analysis: Lifting tags to cohesive subhierarchies Test coverage Already

    fixed Side effects Alternative solution Design Unnecessary contribution QA Internal concerns External concerns
  84. Abandonment is often due to external concerns! Abandon External Concerns

    Internal Concerns Integration Policy Compliance Unnecessary Fix Integration Planning Lack of Interest Testing Implementation Design Unclear Intention Not an Issue Already Fixed Blueprint Release Schedule Patch Dependency Lost by a reviewer Lost by an author Side Effect Shallow Fix Flawed Changes Alternative Solution Backward Compatibility Patch Size Test Failure Test Coverage 59% (102/173) 21 12 24 6 1 1 32 17 5 6 14 6 1 173 10 41% (71/173) 58% (59/105) 7% (7/105) 87% (62/71) 10% (7/71) 3% (2/71) 25% (25/105) 5 Branch Placement Squashing Commits 6 5 11% (11/105) 1
  85. Abandonment is often due to external concerns! Abandon External Concerns

    Internal Concerns Integration Policy Compliance Unnecessary Fix Integration Planning Lack of Interest Testing Implementation Design Unclear Intention Not an Issue Already Fixed Blueprint Release Schedule Patch Dependency Lost by a reviewer Lost by an author Side Effect Shallow Fix Flawed Changes Alternative Solution Backward Compatibility Patch Size Test Failure Test Coverage 59% (102/173) 21 12 24 6 1 1 32 17 5 6 14 6 1 173 10 41% (71/173) 58% (59/105) 7% (7/105) 87% (62/71) 10% (7/71) 3% (2/71) 25% (25/105) 5 Branch Placement Squashing Commits 6 5 11% (11/105) 1
  86. Understanding code review in contentious patches How often? Trend over

    time? What drives abandonment? Start How are concerns resolved? Contentious reviews are not rare Contention is not a show stopper Contentious reviews have been growing over time Integration rates of contentious reviews have stabilized Concerns beyond patch scope are more strongly linked with abandonment than concerns within patch scope
  87. Understanding code review in contentious patches How often? Trend over

    time? What drives abandonment? Start How are concerns resolved? Contentious reviews are not rare Contention is not a show stopper Contentious reviews have been growing over time Integration rates of contentious reviews have stabilized Concerns beyond patch scope are more strongly linked with abandonment than concerns within patch scope
  88. Contention is more often resolved without altering the patch! Integrate

    Directly Addressing Indirectly Addressing Legal Problems Unnecessary Fix Design Implementation Testing Integration Policy Compliance Alternative Solution Side Effect Shallow Fix Flawed Changes Backward Compatibility Test Coverage Test Failure Squashing Commits Branch Placement 44% (83/189) 2 1 3 9 21 7 4 65 12 3 3 3 70% (58/83) 189 Withdrawal of Negative Score Integration Planning Persuasion Self-change Release Schedule Blueprint Patch Dependency 10 56% (106/189) 4% (3/83) 7% (6/83) 5% (4/83) 28% (30/106) 72% (76/106) Legal Issues Unclear Intention 12% (10/83) 2% (2/83) 4 4 38
  89. Contention is more often resolved without altering the patch! Integrate

    Directly Addressing Indirectly Addressing Legal Problems Unnecessary Fix Design Implementation Testing Integration Policy Compliance Alternative Solution Side Effect Shallow Fix Flawed Changes Backward Compatibility Test Coverage Test Failure Squashing Commits Branch Placement 44% (83/189) 2 1 3 9 21 7 4 65 12 3 3 3 70% (58/83) 189 Withdrawal of Negative Score Integration Planning Persuasion Self-change Release Schedule Blueprint Patch Dependency 10 56% (106/189) 4% (3/83) 7% (6/83) 5% (4/83) 28% (30/106) 72% (76/106) Legal Issues Unclear Intention 12% (10/83) 2% (2/83) 4 4 38
  90. Understanding code review in contentious patches How often? Trend over

    time? What drives abandonment? Start How are concerns resolved? Contentious reviews are not rare Contention is not a show stopper Contentious reviews have been growing over time Integration rates of contentious reviews have stabilized Concerns beyond patch scope are more strongly linked with abandonment than concerns within patch scope Concerns are often addressed through negotiation or scheduling
  91. None
  92. Life is very short, and there’s no time, for fussing

    and fighting, my friends
  93. Mining Gerrit reviews to study reviewing processes Community evolution Ray

    Wen Master’s Student Contentious reviews Toshiki Hirao PhD Student
  94. Mining Gerrit reviews to study reviewing processes Community evolution Ray

    Wen Master’s Student Contentious reviews Toshiki Hirao PhD Student
  95. None
  96. And in the end The love you take Is equal

    to The love you make
  97. And in the end The love you take Is equal

    to The love you make You get what you put in(to code review)
  98. You get what you put into code review Author

  99. You get what you put into code review Author

  100. You get what you put into code review Author Looks

    like code to me! +2 Rookie Reviewer
  101. You get what you put into code review Author Looks

    like code to me! +2 Rookie Reviewer Seasoned Reviewer Did you consider alternative designs? -1
  102. How does reviewing feedback evolve with respect to…

  103. How does reviewing feedback evolve with respect to… Community aging?

  104. How does reviewing feedback evolve with respect to… Community aging?

    Reviewer experience?
  105. How does reviewing feedback evolve with respect to… Community aging?

    Reviewer experience? Reviewer workload?
  106. How does reviewing feedback evolve with respect to… Community aging?

    Reviewer experience? Reviewer workload?
  107. How does reviewing feedback evolve with respect to… Community aging?

    Reviewer experience? Reviewer workload?
  108. Automatically analyzing topics that are discussed in code reviews

  109. Automatically analyzing topics that are discussed in code reviews Get

    inline comments 506,950 inline comments
  110. Automatically analyzing topics that are discussed in code reviews Get

    inline comments 506,950 inline comments Apply topic analysis Topic 1 Topic 2 Topic 3 Topic 1 Topic 2 Topic 3
  111. Automatically analyzing topics that are discussed in code reviews Get

    inline comments 506,950 inline comments Apply topic analysis Topic 1 Topic 2 Topic 3 Topic 1 Topic 2 Topic 3 Topic score analysis
  112. LDA automatically groups words into topics based on co-occurrence in

    comments How Does Code Reviewing Feedback Evolve? 11 Table 1: The labelled topics with their corresponding words and topic share score CATEGORY TOPIC NAME SELECTED MOST RELEVANT WORDS TOPIC SHARE (%) Operations Logging log, debug, warn, little, anywhere, bit, level, dashboard 4.8 Release Engineering comment, version, package, install, repo, option, config, copyright 4.4 Performance big, core, timeout, performance, time, team, choice 5.5 Testing Testing test, miss, unit, sentence, character, functional, period 4.1 Improvement Request need, line, blank, pep, ref, alarm, insert, codeblock 4.1 Social Communication General Social Comm. ok, release, fix, yeah, liberty, thank, rid, mitaka, ah 6.8 Confirmation ditto, token, context, label, auth, align, hint, keystone 2.1 Server-related Discussion User and Roles service, role, admin, user, rule, ten- ant, security, snapshot 4.5 Network Issues port, network, ip, agent, router, ad- dress, neutron, subnet 2.8 Node Issues node, task, state, migration, cluster, disk, step, lock 4.1 Disk Image Issues image, class, glance, function, 3.3
  113. We read comments with high topic scores and name the

    topics How Does Code Reviewing Feedback Evolve? 11 Table 1: The labelled topics with their corresponding words and topic share score CATEGORY TOPIC NAME SELECTED MOST RELEVANT WORDS TOPIC SHARE (%) Operations Logging log, debug, warn, little, anywhere, bit, level, dashboard 4.8 Release Engineering comment, version, package, install, repo, option, config, copyright 4.4 Performance big, core, timeout, performance, time, team, choice 5.5 Testing Testing test, miss, unit, sentence, character, functional, period 4.1 Improvement Request need, line, blank, pep, ref, alarm, insert, codeblock 4.1 Social Communication General Social Comm. ok, release, fix, yeah, liberty, thank, rid, mitaka, ah 6.8 Confirmation ditto, token, context, label, auth, align, hint, keystone 2.1 Server-related Discussion User and Roles service, role, admin, user, rule, ten- ant, security, snapshot 4.5 Network Issues port, network, ip, agent, router, ad- dress, neutron, subnet 2.8 Node Issues node, task, state, migration, cluster, disk, step, lock 4.1 Disk Image Issues image, class, glance, function, 3.3 How Does Code Reviewing Feedback Evolve? 13 ging features are used to aid in debugging and understanding system behaviour. To ensure that the logs contain useful information, logging concerns are often raised during the code review process. How to enable TRACE log level in config file? Enabling the trace would enable debug also? Comment #136066, topic score(logging) = 0.94 The development of OPENSTACK involves contributions from several large soft-
  114. Then we group related topics into categories How Does Code

    Reviewing Feedback Evolve? 11 Table 1: The labelled topics with their corresponding words and topic share score CATEGORY TOPIC NAME SELECTED MOST RELEVANT WORDS TOPIC SHARE (%) Operations Logging log, debug, warn, little, anywhere, bit, level, dashboard 4.8 Release Engineering comment, version, package, install, repo, option, config, copyright 4.4 Performance big, core, timeout, performance, time, team, choice 5.5 Testing Testing test, miss, unit, sentence, character, functional, period 4.1 Improvement Request need, line, blank, pep, ref, alarm, insert, codeblock 4.1 Social Communication General Social Comm. ok, release, fix, yeah, liberty, thank, rid, mitaka, ah 6.8 Confirmation ditto, token, context, label, auth, align, hint, keystone 2.1 Server-related Discussion User and Roles service, role, admin, user, rule, ten- ant, security, snapshot 4.5 Network Issues port, network, ip, agent, router, ad- dress, neutron, subnet 2.8 Node Issues node, task, state, migration, cluster, disk, step, lock 4.1 Disk Image Issues image, class, glance, function, 3.3
  115. PC1 PC2 Marginal topic distribution 2% 5% 10% 1 2

    3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 20 19 18 Communication Testing Software Design Operations Server-oriented Version Control File System Documentation Formatting 1 - Documentation and Testing 2 - Social Communication 3 - Testing 4 - Key Errors 5 - Logging 6 - Default Parameters 7 - Object names ? 8 - Release Engineering & Ops 9 - Time and Performance 10 - User & Roles 11 - Exception Handling 12 - Image Issues 13 - Version Control 14 - File System 15 - Network Issues 16 - Node Issues 17 - Confirmation 18 - Requests 19 - Function Header 20 - Formatting Legends
  116. As the community has aged, design- oriented feedback has declined

    Node Issues • • • • • • • • • • • • • • • * Disk Image Issues • • • • • • • • • • • • • • • • • • • • • Software Design Key Errors + • • • • • • • • • • • • • • • • • • • • • + Exception Han- dling + • • • • • • • • • • • • • • • • • • • • • Object Names • • • • • • • • • • • • • • • • • • • • • * Function Headers + • • • • • • • • • • • • • • • • • • • • • * Default and Pa- rameters • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •
  117. On the other hand, discussion of networking and patch formatting

    have grown Social Communication General Social Comm. • • • • • • • • * Confirmation • • • • • • • • • • • • • • • • • • • • • Server-related Discussion User and Roles • • • • • • • • • • • • • • • • • • • • • + Network Issues * • • • • • • • • • • • • • • • • • • • • • Node Issues • • • • • • • • • • • • • • • • • • • • • * Disk Image Issues • • • • • • • • • • • • • • • • • • • • • Software Design Key Errors + • • • • • • • • • • • • • • • • • • • • • + • • • • • • • • • Function Headers + • • • • • • • • • • • • • • • • • • • • • * Default and Pa- rameters • • • • • • • • • • • • • • • • • • • • • - Version Control • • • • • • • • • • • • • • • • • • • • • - File System + • • • • • • • • • • • • • • • • • • • • • + - Formatting * • • • • • • • • • • • • • • • • • • • • • + - Documentation • • • • • • • • • • • • • • • • • • • •
  118. How does reviewing feedback evolve with respect to… Community aging?

    Reviewer experience? Reviewer workload? As OpenStack has aged, the community has focused less on patch design and more on networking and patch formatting
  119. How does reviewing feedback evolve with respect to… Community aging?

    Reviewer experience? Reviewer workload? As OpenStack has aged, the community has focused less on patch design and more on networking and patch formatting
  120. System-oriented topics increase as reviewers accrue experience with OpenStack CATEGORY

    TOPIC NAME TIME TREND TOPIC IMPACT (OVER TIME) EXP. TREND Operations Logging • • • • • • • • • • • • • • • • • • • • • * Release Engineer- ing • • • • • • • • • • • • • • • • • • • • • + Performance • • • • • • • • • • • • • • • • • • • • • * Quality Assurance Testing • • • • • • • • • • • • • • • • • • • • • Improvement Re- quest • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • E TIME TREND TOPIC IMPACT (OVER TIME) EXP. TREND TOPIC EXPERIENCE IMPACT WORKL TREND • • • • • • • • • • • • • • • • • • • • • * • • • • • •••••• • • • • • • •• • • • • • • • • • •• * ineer- • • • • • • • • • • • • • • • • • • • • • + • • • • • • • • • • •• • • •• •• • • • • •• ••• •• • + • • • • • • • • • • • • • • • • • • • • • * • • • •• • • • • • • •• • • • • • ••• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• •• • • • • • • ••• • • • • • t Re- • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • • ••• • • • • • • • • • • • • • • • • • • • • • TREND (OVER TIME) TREND Operations Logging • • • • • • • • • • • • • • • • • • • • • * Release Engineer- ing • • • • • • • • • • • • • • • • • • • • • + Performance • • • • • • • • • • • • • • • • • • • • • * Quality Assurance Testing • • • • • • • • • • • • • • • • • • • • • Improvement Re- quest • • • • • • • • • • • • • • • • • • • • • Social Communication General Social Comm. • • • • • • • • • • • • • • • • • • • • • * • • • • • • TREND (OVER TIME) TREND IMPACT TREND • • • • • • • • • • • • • • • • • • • • • * • • • • • •••••• • • • • • • •• • • • • • • • • • •• * ineer- • • • • • • • • • • • • • • • • • • • • • + • • • • • • • • • • •• • • •• •• • • • • •• ••• •• • + e • • • • • • • • • • • • • • • • • • • • • * • • • •• • • • • • • •• • • • • • ••• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• •• • • • • • • ••• • • • • • nt Re- • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • • ••• • • • • • • • • • • Social • • • • • • • • • • • • • • • • • • • • • * • ••• • •••• • ••• •• •• • • • • • •• • ••• • • * • • • • • • • • • • • •• • ••• •• • • • • • • ••• • • • • Communication Comm. • Confirmation • • • • • • • • • • • • • • • • • • • • • Server-related Discussion User and Roles • • • • • • • • • • • • • • • • • • • • • + Network Issues * • • • • • • • • • • • • • • • • • • • • • Node Issues • • • • • • • • • • • • • • • • • • • • • * Disk Image Issues • • • • • • • • • • • • • • • • • • • • • Software Design Key Errors + • • • • • • • • • • • • • • • • • • • • • + Exception Han- + • • • • • • • • • • • • • • • • • • • • • • n • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • ••• •• • • • • • • ••• • • • • oles • • • • • • • • • • • • • • • • • • • • • + • • • • •• • • • •••• • • • •••••• • • • • • • • • + ues * • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • • • ••• • • • • •• • • • • • • • • • • • • • • • • • • • • • • • • * • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • + Issues • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • •• • • • • • • • • • • ••• • • • • • • + • • • • • • • • • • • • • • • • • • • • • + • •• • • • • •• •• • • • • • • • • • • ••• •• • • • • Han- + • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • ••• • • * Improvement Re- quest • • • • • • • • • • • • • • • • • • • • Social Communication General Social Comm. • • • • • • • • • • • • • • • • • • • • • * Confirmation • • • • • • • • • • • • • • • • • • • • • Server-related Discussion User and Roles • • • • • • • • • • • • • • • • • • • • • + Network Issues * • • • • • • • • • • • • • • • • • • • • • Node Issues • • • • • • • • • • • • • • • • • • • • • * Disk Image Issues • • • • • • • • • • • • • • • • • • • • • CATEGORY TOPIC NAME TIME TREND TOPIC IMPACT (OVER TIME) EXP. TREND Operations Logging • • • • • • • • • • • • • • • • • • • • • * Release Engineer- ing • • • • • • • • • • • • • • • • • • • • • + Performance • • • • • • • • • • • • • • • • • • • • • * Quality Assurance Testing • • • • • • • • • • • • • • • • • • • • • Improvement Re- quest • • • • • • • • • • • • • • • • • • • • • Social Communication General Social Comm. • • • • • • • • • • • • • • • • • • • • • * Server-related Discussion User and Roles • • • • • • • • • • • • • • • • • • • • + Network Issues * • • • • • • • • • • • • • • • • • • • • • Node Issues • • • • • • • • • • • • • • • • • • • • • * Disk Image Issues • • • • • • • • • • • • • • • • • • • • • Software Design Key Errors + • • • • • • • • • • • • • • • • • • • • • + Exception Han- dling + • • • • • • • • • • • • • • • • • • • • • Object Names • • • • • • • • • • • • • • • • • • • • • * • • • • • • • • • • • • • • • • • • • • • * • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • + Issues • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • •• • • • • • • • • • • ••• • • • • • • + • • • • • • • • • • • • • • • • • • • • • + • •• • • • • •• •• • • • • • • • • • • ••• •• • • • • Han- + • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • ••• • • • * es • • • • • • • • • • • • • • • • • • • • • * • • • ••• • • • ••••• • •• • •• •• • • • • • • • • * aders + • • • • • • • • • • • • • • • • • • • • • * • • • • ••• • •• • • • • • • • • • • • ••• • •• • • • * d Pa- • • • • • • • • • • • • • • • • • • • • •• •• •• • • • • • • • • • • • • • • • • • • • • • • • • • • • * • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • + Issues • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • •• • • • • • • • • • • ••• • • • • • • + • • • • • • • • • • • • • • • • • • • • • + • •• • • • • •• •• • • • • • • • • • • ••• •• • • • • Han- + • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • ••• • • • * es • • • • • • • • • • • • • • • • • • • • • * • • • ••• • • • ••••• • •• • •• •• • • • • • • • • * aders + • • • • • • • • • • • • • • • • • • • • • * • • • • ••• • •• • • • • • • • • • • • ••• • •• • • • * d Pa- • • • • • • • • • • • • • • • • • • • • • • ••• • • • • •• •• •• • • • • • • • • Node Issues • • • • • • • • • • • • • • • • • • • • • * Disk Image Issues • • • • • • • • • • • • • • • • • • • • • Software Design Key Errors + • • • • • • • • • • • • • • • • • • • • • + Exception Han- dling + • • • • • • • • • • • • • • • • • • • • • Object Names • • • • • • • • • • • • • • • • • • • • • * Function Headers + • • • • • • • • • • • • • • • • • • • • • * Default and Pa- • • • • • • • • • • • • • • • • • • E TIME TREND TOPIC IMPACT (OVER TIME) EXP. TREND TOPIC EXPERIENCE IMPACT WORKL TREND • • • • • • • • • • • • • • • • • • • • • * • • • • • •••••• • • • • • • •• • • • • • • • • • •• * ineer- • • • • • • • • • • • • • • • • • • • • • + • • • • • • • • • • •• • • •• •• • • • • •• ••• •• • + e • • • • • • • • • • • • • • • • • • • • • * • • • •• • • • • • • •• • • • • • ••• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• •• • • • • • • ••• • • • • • t Re- • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • • ••• • • • • • • • • • • Social • • • • • • • • • • • • • • • • • • • • • * • ••• • •••• • ••• •• •• • • • • • •• • ••• • • * •
  121. System-oriented topics increase as reviewers accrue experience with OpenStack CATEGORY

    TOPIC NAME TIME TREND TOPIC IMPACT (OVER TIME) EXP. TREND Operations Logging • • • • • • • • • • • • • • • • • • • • • * Release Engineer- ing • • • • • • • • • • • • • • • • • • • • • + Performance • • • • • • • • • • • • • • • • • • • • • * Quality Assurance Testing • • • • • • • • • • • • • • • • • • • • • Improvement Re- quest • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • E TIME TREND TOPIC IMPACT (OVER TIME) EXP. TREND TOPIC EXPERIENCE IMPACT WORKL TREND • • • • • • • • • • • • • • • • • • • • • * • • • • • •••••• • • • • • • •• • • • • • • • • • •• * ineer- • • • • • • • • • • • • • • • • • • • • • + • • • • • • • • • • •• • • •• •• • • • • •• ••• •• • + • • • • • • • • • • • • • • • • • • • • • * • • • •• • • • • • • •• • • • • • ••• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• •• • • • • • • ••• • • • • • t Re- • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • • ••• • • • • • • • • • • • • • • • • • • • • • TREND (OVER TIME) TREND Operations Logging • • • • • • • • • • • • • • • • • • • • • * Release Engineer- ing • • • • • • • • • • • • • • • • • • • • • + Performance • • • • • • • • • • • • • • • • • • • • • * Quality Assurance Testing • • • • • • • • • • • • • • • • • • • • • Improvement Re- quest • • • • • • • • • • • • • • • • • • • • • Social Communication General Social Comm. • • • • • • • • • • • • • • • • • • • • • * • • • • • • TREND (OVER TIME) TREND IMPACT TREND • • • • • • • • • • • • • • • • • • • • • * • • • • • •••••• • • • • • • •• • • • • • • • • • •• * ineer- • • • • • • • • • • • • • • • • • • • • • + • • • • • • • • • • •• • • •• •• • • • • •• ••• •• • + e • • • • • • • • • • • • • • • • • • • • • * • • • •• • • • • • • •• • • • • • ••• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• •• • • • • • • ••• • • • • • nt Re- • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • • ••• • • • • • • • • • • Social • • • • • • • • • • • • • • • • • • • • • * • ••• • •••• • ••• •• •• • • • • • •• • ••• • • * • • • • • • • • • • • •• • ••• •• • • • • • • ••• • • • • Communication Comm. • Confirmation • • • • • • • • • • • • • • • • • • • • • Server-related Discussion User and Roles • • • • • • • • • • • • • • • • • • • • • + Network Issues * • • • • • • • • • • • • • • • • • • • • • Node Issues • • • • • • • • • • • • • • • • • • • • • * Disk Image Issues • • • • • • • • • • • • • • • • • • • • • Software Design Key Errors + • • • • • • • • • • • • • • • • • • • • • + Exception Han- + • • • • • • • • • • • • • • • • • • • • • • n • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • ••• •• • • • • • • ••• • • • • oles • • • • • • • • • • • • • • • • • • • • • + • • • • •• • • • •••• • • • •••••• • • • • • • • • + ues * • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • • • ••• • • • • •• • • • • • • • • • • • • • • • • • • • • • • • • * • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • + Issues • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • •• • • • • • • • • • • ••• • • • • • • + • • • • • • • • • • • • • • • • • • • • • + • •• • • • • •• •• • • • • • • • • • • ••• •• • • • • Han- + • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • ••• • • * Improvement Re- quest • • • • • • • • • • • • • • • • • • • • Social Communication General Social Comm. • • • • • • • • • • • • • • • • • • • • • * Confirmation • • • • • • • • • • • • • • • • • • • • • Server-related Discussion User and Roles • • • • • • • • • • • • • • • • • • • • • + Network Issues * • • • • • • • • • • • • • • • • • • • • • Node Issues • • • • • • • • • • • • • • • • • • • • • * Disk Image Issues • • • • • • • • • • • • • • • • • • • • • CATEGORY TOPIC NAME TIME TREND TOPIC IMPACT (OVER TIME) EXP. TREND Operations Logging • • • • • • • • • • • • • • • • • • • • • * Release Engineer- ing • • • • • • • • • • • • • • • • • • • • • + Performance • • • • • • • • • • • • • • • • • • • • • * Quality Assurance Testing • • • • • • • • • • • • • • • • • • • • • Improvement Re- quest • • • • • • • • • • • • • • • • • • • • • Social Communication General Social Comm. • • • • • • • • • • • • • • • • • • • • • * Server-related Discussion User and Roles • • • • • • • • • • • • • • • • • • • • + Network Issues * • • • • • • • • • • • • • • • • • • • • • Node Issues • • • • • • • • • • • • • • • • • • • • • * Disk Image Issues • • • • • • • • • • • • • • • • • • • • • Software Design Key Errors + • • • • • • • • • • • • • • • • • • • • • + Exception Han- dling + • • • • • • • • • • • • • • • • • • • • • Object Names • • • • • • • • • • • • • • • • • • • • • * • • • • • • • • • • • • • • • • • • • • • * • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • + Issues • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • •• • • • • • • • • • • ••• • • • • • • + • • • • • • • • • • • • • • • • • • • • • + • •• • • • • •• •• • • • • • • • • • • ••• •• • • • • Han- + • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • ••• • • • * es • • • • • • • • • • • • • • • • • • • • • * • • • ••• • • • ••••• • •• • •• •• • • • • • • • • * aders + • • • • • • • • • • • • • • • • • • • • • * • • • • ••• • •• • • • • • • • • • • • ••• • •• • • • * d Pa- • • • • • • • • • • • • • • • • • • • • •• •• •• • • • • • • • • • • • • • • • • • • • • • • • • • • • * • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • + Issues • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • •• • • • • • • • • • • ••• • • • • • • + • • • • • • • • • • • • • • • • • • • • • + • •• • • • • •• •• • • • • • • • • • • ••• •• • • • • Han- + • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • ••• • • • * es • • • • • • • • • • • • • • • • • • • • • * • • • ••• • • • ••••• • •• • •• •• • • • • • • • • * aders + • • • • • • • • • • • • • • • • • • • • • * • • • • ••• • •• • • • • • • • • • • • ••• • •• • • • * d Pa- • • • • • • • • • • • • • • • • • • • • • • ••• • • • • •• •• •• • • • • • • • • Node Issues • • • • • • • • • • • • • • • • • • • • • * Disk Image Issues • • • • • • • • • • • • • • • • • • • • • Software Design Key Errors + • • • • • • • • • • • • • • • • • • • • • + Exception Han- dling + • • • • • • • • • • • • • • • • • • • • • Object Names • • • • • • • • • • • • • • • • • • • • • * Function Headers + • • • • • • • • • • • • • • • • • • • • • * Default and Pa- • • • • • • • • • • • • • • • • • • E TIME TREND TOPIC IMPACT (OVER TIME) EXP. TREND TOPIC EXPERIENCE IMPACT WORKL TREND • • • • • • • • • • • • • • • • • • • • • * • • • • • •••••• • • • • • • •• • • • • • • • • • •• * ineer- • • • • • • • • • • • • • • • • • • • • • + • • • • • • • • • • •• • • •• •• • • • • •• ••• •• • + e • • • • • • • • • • • • • • • • • • • • • * • • • •• • • • • • • •• • • • • • ••• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• •• • • • • • • ••• • • • • • t Re- • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • • ••• • • • • • • • • • • Social • • • • • • • • • • • • • • • • • • • • • * • ••• • •••• • ••• •• •• • • • • • •• • ••• • • * •
  122. Low ROI topics like Formatting tend to decrease as reviewers

    accrue experience rameters • • • • • • • - Version Control • • • • • • • • • • • • • • • • • • • • • - File System + • • • • • • • • • • • • • • • • • • • • • + - Formatting * • • • • • • • • • • • • • • • • • • • • • + - Documentation • • • • • • • • • • • • • • • • • • • • where D(m) represents the set of review comm other words, the topic impact metric measures • • • • • • • • • ••••• • • • • •• • •• •• • • • • • • • • • • Control • • • • • • • • • • • • • • • • • • • • • • • • •• • • • • • •• • •• • • • • • • •• • • • • • • • em + • • • • • • • • • • • • • • • • • • • • • + •• ••• •• • • • • • •• • • • • • • • • •• • • • • • • + ng * • • • • • • • • • • • • • • • • • • • • • + • • • • •• • • • •• •• •• • • • • • • •• • • • • • • • + tation • • • • • • • • • • • • • • • • • • • • • • • ••• • • • • •• • • • • • •• • • • • • • • • • • • esents the set of review comments that are written topic impact metric measures the proportion of r
  123. How does reviewing feedback evolve with respect to… Community aging?

    Reviewer experience? Reviewer workload? As OpenStack reviewers accrue experience, they tend to comment more on system-oriented topics and less on low ROI topics As OpenStack has aged, the community has focused less on patch design and more on networking and patch formatting
  124. How does reviewing feedback evolve with respect to… Community aging?

    Reviewer experience? Reviewer workload? As OpenStack reviewers accrue experience, they tend to comment more on system-oriented topics and less on low ROI topics As OpenStack has aged, the community has focused less on patch design and more on networking and patch formatting
  125. Reviewers with high experience scores tend to also have a

    heavy workload score ( ) , =0.68
  126. How does reviewing feedback evolve with respect to… Community aging?

    Reviewer experience? Reviewer workload? Similar observations as were reported for experience As OpenStack has aged, the community has focused less on patch design and more on networking and patch formatting As OpenStack reviewers accrue experience, they tend to comment more on system-oriented topics and less on low ROI topics
  127. None
  128. I wanna hold your hand!

  129. Mining Gerrit reviews to study reviewing processes Community evolution Ray

    Wen Master’s Student Contentious reviews Toshiki Hirao PhD Student
  130. Mining Gerrit reviews to study reviewing processes Community evolution Ray

    Wen Master’s Student Contentious reviews Toshiki Hirao PhD Student
  131. None
  132. None
  133. None
  134. None
  135. None
  136. None
  137. None