Shadow PC Patanamon (Pick) Thongtanunam Ayushi Rastogi Mining Software Repositories (MSR) 2021 

Manuscript Manuscript Manuscript What is Shadow PC? An initiative program that allows early-career researchers (ECRs) to gain the first-hand experience about the academic peer-review process Train the next generation of PC members Expose ECRs to the review process of the technical track Provide an opportunity for ECRs to be embedded in the community 2 

Shadow PC Recruitment Open for self-nomination during Nov 1 - Dec 10, 2020 Targeting PhD students, postdocs, new faculty members, and industry practitioners who never served as a PC member of the premier SE venues Publicize via various social network platforms, SEWorld, reach out to various SE communities & under-representative countries Received 162 applicants from a wide array of backgrounds from 36 countries 3 

Shadow PC members 106 Shadow PC members Selection based on research exp. & motivation while ensuring the diversity 32 Countries Gender M F 64% 30% 6% Occupation PhD Students 75% Faculty members 9% 3% Industry practitioners Researchers 4% Post-docs 10% 13 were invited as a MSR21 PC member Remaining is encouraged to re-apply 48 Post-PhD applicants were shared with other PC chairs Unselected applicants 4 

Shadow PC Reviewing Process Shadow reviews are conducted in a separate HotCRP instance from the MSR main track Review Each Shadow PC member was assigned with a load of 2 papers 20 Jan - 12 Feb Advisor check & Discussion An advisor check the quality of reviews. Shadow PCs discuss to reach consensus 13-25 Feb Notify authors The reviews are shared with the authors of the papers 9 Mar Select Papers 40 out of 59 papers that opt-in Shadow PC were selected 15 Jan Declare COI & Bidding Shadow PC declared conflicts of interest & bidded the papers 11-18 Jan 5 

Support for Shadow PCs Reviewing Guidelines & Tips Q&A with peers & Shadow PC chairs 6 

Shadow PC Reviews 5-6 reviews per paper A team of 26 advisors (5-10 reviews per advisor) 680 messages (408 Shadow PC messages) Paper Acceptance: 19 40 by Shadow PC 14 40 by MSR PC by MSR PC Accepted Rejected by Shadow PC Accepted Rejected 12 19 7 2 7 

Feedback from Shadow PC members Survey responses from 73 shadow PC members 89% Better understand the academic peer review process 94% Gain a new perspective of how actual pc member read/review the manuscript 83% Received useful feedback from my advisor on how to write a good review 98% Will recommend Shadow PC program to others “Great to read feedback on not only my reviews but also on other shadow PC members’ review” “It was my first time seeing the work behind the scenes that goes into discussing the papers and finalizing the decision.” “PC is to accept papers instead of rejecting papers.” “there is not often an opportunity to learn this (except if you are lucky and have a good supervisor/colleagues during your phd studies).” 8 

Feedback from Authors Survey responses from 27 authors 78% Shadow PC reviews are clear and concise 77% Shadow PC reviews are constructive and useful. They can improve my manuscript 88% The language and tone used in Shadow PC reviews are appropriate 82% Overall, I’m satisfied with the Shadow PC reviews “Receiving additional feedback and allowing new community members to become acquainted with the inner workings of a conference.” “to see if both PC's come to similar conclusions and also to give the young people in the shadow PC a chance to have a real PC experience” “We got better and more actionable feedback than from the shadow PC than from the main PC.” “I used some reviews in the camera-ready” 9 

Reflection ● Benefits of Shadow PC program ○ SE community: a pool of diverse reviewers ○ Authors: additional feedback ○ Shadow PC: learn to review, engage, understand process ● Areas for improvement ○ Advisor: clear expectations, actual reviewer ○ Shadow PC: consistent experiences, training workshop ○ Timeline: shorter feedback loop ○ Program: rebuttal, distinguished reviewer, promote to main PC ● Future of reviewing ○ Shadow + experienced PC reviews (reduced workload) 10 96% Shadow PC 82% Authors “MSR should hold the Shadow PC program again in the future” 

Special Thanks Kelly Blincoe MSR PC Co-Chair Mei Nagappan MSR PC Co-Chair Serge Demeyer Shadow PC Advisor Co-Chair Foutse Khomh Shadow PC Advisor Co-Chair 11 

Huge Thanks to Shadow PC Advisors 12 

Thanks to the Shadow PC members! 13