ICSE 2021 Reviewing Process

Slide deck used to discuss the reviewing process of the ACM/IEEE International Conference on Software Engineering (ICSE) with the full program committee (PC).

The slide deck covers the PC roles, responsibilities, composition, the time line, and details about the reviewing and discussion phase.

The slide formed the basis for a video recorded by the program chairs (Tao Xie and Arie van Deursen), available on YouTube at https://www.youtube.com/watch?v=q5gnvMA5FdA.

The video was discussed at six sessions in three different time bands, in which the over 150 PC members would get to know each other and discuss the proposed reviewing process.

The actual reviewing process may deviate in details, but will be quite similar to what's described here. For more info on ICSE see https://conf.researchr.org/home/icse-2021.

Transcript

1. ICSE 2021 Reviewing Process Arie van Deursen, TU Delft, The Netherlands Tao Xie, Peking University, China 1
2. Objectives • Accept high quality papers in any topic in software engineering • Give clear feedback to papers of insufficient quality • Judge different papers against the same bar • Ensure safe and transparent process • Ensure decisions are reached with at least 5 people involved (10 eyes) • Have a process that scales up to 600+ papers • Ensure people involved have manageable load 2
3. ICSE 2021 PC Roles • Reviewer: ◦ Writes high quality review ◦ Participates in discussion • Moderator: ◦ Moderates discussion among three reviewers ◦ Builds consensus where possible ◦ Proposes a decision ◦ Write meta-review, explaining decision • Area chair: ◦ Ensures consistency for papers within area, and across areas ◦ Confirms (challenges) proposed decisions ◦ Ensures all steps are taken in a timely manner ◦ Basically act as “proxy chair” in final discussions ◦ (also suggest PC members during PC formation) In contrast to ICSE 2020: • Every PC member has reviewer and moderation roles • No separate “program board” • New “Area Chairs”: moderation role replaced with area chair role 3
4. Area Chairs Help further scaling of ICSE Main topics and chairs for 2021: • AI and software engineering: Dongmei Zhang (Microsoft Research) • Testing and analysis: Anders Møller (Aarhus University) • Empirical software engineering: Tom Zimmermann (Microsoft Research) • Software evolution: Lori Pollock (U. Delaware) • Social aspects of software engineering: Daniela Damian (U. Victoria) • Requirements, modeling and design: Zhi Jin (Peking University) • Dependability; Eric Bodden (Paderborn University) 4
5. Seven Key Areas and their 40 Sub-Topics 5
6. Who is a Paper’s Area Chair? • Areas have (unavoidable) overlap • PC members indicate their expertise on 40 sub-topics • Authors mark their paper as relevant to multiple sub-topics • For each paper calculate fit with each area • Papers in principle assigned to best fitting area chair • If area size is too unbalanced, we’ll move papers to other (fitting) areas 6
7. PC Formation • 158 program committee members accepted, of which 7 area chairs • Male/female: 61/39% • North America / EU / Asia / Other: 40/30/20/10% • Academia / Industry: 91/9% • Reviewing / Moderation / No preference: 47/24/29% 7
8. Expected Reviewer & Moderation Load Assuming 600 papers (similar to 2020): 8
9. HotCrp Conflict Of Interest Declarations • List all potential authors you have a conflict with! • Copy from earlier hotcrp installations • Ensure they are up to date by August 15 Different from EasyChair Please do this in advance 9
10. Paper Bidding Ensure your paper topics in hotcrp are up to date Place positive bids on papers that you have the expertise to review If you think you’re one of the few on the PC with expertise on a paper make sure to place a high bid! Avoid placing bids on papers you’d like to learn about, but where you have no expertise. Place at least 45 positive bids 10
11. Reviewing Phases I: Review Writing • Bidding [ 3 working days ] • Review round 1 (50% of papers) [ 4 weeks ] • Review round 2 (100% of papers) [ 3 weeks ] • Quality gate [ 1 week ] ◦ Moderators give feedback on reviews ◦ Moderators identify papers needing replacement reviewer • Review round 3 (±1 emergency paper) [ 1 week ] • Discussion 1 (pre-rebuttal) [ earlier weeks, plus 4 days ] • Rebuttal phase (by authors) [ 3 working days ] 11
12. Reviewing Phases II: Discussions (1) • Discussion 1: Pre-rebuttal [ 4 days ] ◦ Can start as soon as three reviews are in (not intended to build consensus for decision making) ◦ Goal: Identify what extra information from authors could support decision making ◦ Result: Questions for authors (no decisions!) rebuttal period 12
13. Reviewing Phases II: Discussions (2) • Discussion 2: Clear cases, after rebuttal [ 5 days] ◦ After rebuttal ◦ Identify papers with only rejects or only accepts (estimate: 50%) ◦ Result: Decision + meta-review for first half of papers • Discussion 3: Mixed cases [ 5 + 5 days ] ◦ Make decision for papers with mixed score where consensus can be built ◦ Result 1: Decision + meta-review for another 25% of the papers. ◦ Result 2: Identification of papers without consensus • Discussions 4: No-consensus [ 5 + 5 + 5 days ] ◦ Moderator is in charge ◦ Active involvement from area chair ◦ Reach decision for final 25% 5 days 5 days rebuttal period 5 days 13
14. Reviewing Phases III: Final Decisions • Chair Meeting: [ 2 days ] ◦ Cover all remaining abnormal cases ◦ (Area) chairs finalize all decisions. ◦ PC members stand-by for fast responses on papers that were still discussed in phase 4. 14
15. Full Timeline 15
16. Review Guidelines • Write a summary in your own words • Collect feedback • Give thoughts on 5 ICSE criteria: ◦ Soundness, significance, novelty, verifiability, presentation • Identify the decisive factors resulting in acceptance • Identify clarification questions that can help in the decision making • Be polite, constructive, informative, … • Seek for reasons for acceptance (accept unless you see a blocker) Commit to the deadlines Start as early as possible Submit your review as soon as it is done 16
17. Discussion Guidelines • Be polite and friendly • Be willing to move • Seek for reasons for acceptance • Update reviews where necessary • Focus on the key decision factors • Let final meta-review reflect key factors resulting in the decision. Be responsive Stick to the deadlines Answer as early as possible 17
18. Conditional accepts • Conditional accept will be possible • Requires a clear and easily checkable explicit acceptance condition ◦ Easily checkable = even the chair(s) can check it! • One PC member should be willing to check the condition. • Conditions most likely about presentation 18
19. Open Science and Artefacts • ICSE follows (new!) SIGSOFT Open Science policies • Authors can upload supplementary material • Reviewers can look at material, but don’t have to • Once accepted, authors can submit material to the artifact evaluation track ◦ For reusable, available, replicated or reproduced artifacts 19
20. Double Blind • Authors should do best effort to blind • Reviewers should do best effort not to search / discover • Clear violations will lead to (desk) reject ◦ Checked in first week by chairs ◦ If later discovered by reviewers instead, later desk reject still possible • Reviewers can continue reviewing if they discover identity by accident • Authors can publish preprint on e.g., Arxiv • Further details on FAQ 20
21. ICSE 2021 Reviewing Process Arie van Deursen, TU Delft, The Netherlands Tao Xie, Peking University, China 21