ICSE 2021 Reviewing Process

Slide deck used to discuss the reviewing process of the ACM/IEEE International Conference on Software Engineering (ICSE) with the full program committee (PC).

The slide deck covers the PC roles, responsibilities, composition, the time line, and details about the reviewing and discussion phase.

The slide formed the basis for a video recorded by the program chairs (Tao Xie and Arie van Deursen), available on YouTube at https://www.youtube.com/watch?v=q5gnvMA5FdA.

The video was discussed at six sessions in three different time bands, in which the over 150 PC members would get to know each other and discuss the proposed reviewing process.

The actual reviewing process may deviate in details, but will be quite similar to what's described here. For more info on ICSE see https://conf.researchr.org/home/icse-2021.

Transcript

1. ICSE 2021
   Reviewing Process
   Arie van Deursen, TU Delft, The Netherlands
   Tao Xie, Peking University, China
   1
   

   View Slide

2. Objectives
   ● Accept high quality papers in any topic in software engineering
   ● Give clear feedback to papers of insufficient quality
   ● Judge different papers against the same bar
   ● Ensure safe and transparent process
   ● Ensure decisions are reached with at least 5 people involved (10 eyes)
   ● Have a process that scales up to 600+ papers
   ● Ensure people involved have manageable load
   2
   

   View Slide

3. ICSE 2021 PC Roles
   ● Reviewer:
   ○ Writes high quality review
   ○ Participates in discussion
   ● Moderator:
   ○ Moderates discussion among three reviewers
   ○ Builds consensus where possible
   ○ Proposes a decision
   ○ Write meta-review, explaining decision
   ● Area chair:
   ○ Ensures consistency for papers within area, and across areas
   ○ Confirms (challenges) proposed decisions
   ○ Ensures all steps are taken in a timely manner
   ○ Basically act as “proxy chair” in final discussions
   ○ (also suggest PC members during PC formation)
   In contrast to ICSE 2020:
   ● Every PC member
   has reviewer and
   moderation roles
   ● No separate
   “program board”
   ● New “Area Chairs”:
   moderation role
   replaced with area
   chair role
   3
   

   View Slide

4. Area Chairs
   Help further scaling of ICSE
   Main topics and chairs for 2021:
   ● AI and software engineering: Dongmei Zhang (Microsoft Research)
   ● Testing and analysis: Anders Møller (Aarhus University)
   ● Empirical software engineering: Tom Zimmermann (Microsoft Research)
   ● Software evolution: Lori Pollock (U. Delaware)
   ● Social aspects of software engineering: Daniela Damian (U. Victoria)
   ● Requirements, modeling and design: Zhi Jin (Peking University)
   ● Dependability; Eric Bodden (Paderborn University)
   4
   

   View Slide

5. Seven Key Areas and their 40 Sub-Topics
   5
   

   View Slide

6. Who is a Paper’s Area Chair?
   ● Areas have (unavoidable) overlap
   ● PC members indicate their expertise on 40 sub-topics
   ● Authors mark their paper as relevant to multiple sub-topics
   ● For each paper calculate fit with each area
   ● Papers in principle assigned to best fitting area chair
   ● If area size is too unbalanced, we’ll move papers to other (fitting) areas
   6
   

   View Slide

7. PC Formation
   ● 158 program committee members accepted, of which 7 area chairs
   ● Male/female: 61/39%
   ● North America / EU / Asia / Other: 40/30/20/10%
   ● Academia / Industry: 91/9%
   ● Reviewing / Moderation / No preference: 47/24/29%
   7
   

   View Slide

8. Expected Reviewer & Moderation Load
   Assuming 600 papers (similar to 2020):
   8
   

   View Slide

9. HotCrp Conflict Of Interest Declarations
   ● List all potential authors you have a conflict with!
   ● Copy from earlier hotcrp installations
   ● Ensure they are up to date by August 15
   Different from
   EasyChair
   Please do this
   in advance
   9
   

   View Slide

10. Paper Bidding
    Ensure your paper topics in hotcrp are up to date
    Place positive bids on papers that you have the expertise to review
    If you think you’re one of the few on the PC with expertise on a paper make sure
    to place a high bid!
    Avoid placing bids on papers you’d like to learn about, but where you have no
    expertise.
    Place at least 45 positive bids
    10
    

    View Slide

11. Reviewing Phases I: Review Writing
    ● Bidding [ 3 working days ]
    ● Review round 1 (50% of papers) [ 4 weeks ]
    ● Review round 2 (100% of papers) [ 3 weeks ]
    ● Quality gate [ 1 week ]
    ○ Moderators give feedback on reviews
    ○ Moderators identify papers needing replacement reviewer
    ● Review round 3 (±1 emergency paper) [ 1 week ]
    ● Discussion 1 (pre-rebuttal) [ earlier weeks, plus 4 days ]
    ● Rebuttal phase (by authors) [ 3 working days ]
    11
    

    View Slide

12. Reviewing Phases II: Discussions (1)
    ● Discussion 1: Pre-rebuttal [ 4 days ]
    ○ Can start as soon as three reviews are in (not intended to build consensus for decision making)
    ○ Goal: Identify what extra information from authors could support decision making
    ○ Result: Questions for authors (no decisions!)
    rebuttal period
    12
    

    View Slide

13. Reviewing Phases II: Discussions (2)
    ● Discussion 2: Clear cases, after rebuttal [ 5 days]
    ○ After rebuttal
    ○ Identify papers with only rejects or only accepts (estimate: 50%)
    ○ Result: Decision + meta-review for first half of papers
    ● Discussion 3: Mixed cases [ 5 + 5 days ]
    ○ Make decision for papers with mixed score where consensus can be built
    ○ Result 1: Decision + meta-review for another 25% of the papers.
    ○ Result 2: Identification of papers without consensus
    ● Discussions 4: No-consensus [ 5 + 5 + 5 days ]
    ○ Moderator is in charge
    ○ Active involvement from area chair
    ○ Reach decision for final 25%
    5 days
    5 days
    rebuttal period
    5 days
    13
    

    View Slide

14. Reviewing Phases III: Final Decisions
    ● Chair Meeting: [ 2 days ]
    ○ Cover all remaining abnormal cases
    ○ (Area) chairs finalize all decisions.
    ○ PC members stand-by for fast responses on papers that were still discussed in phase 4.
    14
    

    View Slide

15. Full Timeline
    15
    

    View Slide

16. Review Guidelines
    ● Write a summary in your own words
    ● Collect feedback
    ● Give thoughts on 5 ICSE criteria:
    ○ Soundness, significance, novelty, verifiability, presentation
    ● Identify the decisive factors resulting in acceptance
    ● Identify clarification questions that can help in the decision making
    ● Be polite, constructive, informative, …
    ● Seek for reasons for acceptance (accept unless you see a blocker)
    Commit to the deadlines
    Start as early as possible
    Submit your review as
    soon as it is done
    16
    

    View Slide

17. Discussion Guidelines
    ● Be polite and friendly
    ● Be willing to move
    ● Seek for reasons for acceptance
    ● Update reviews where necessary
    ● Focus on the key decision factors
    ● Let final meta-review reflect key factors resulting in the decision.
    Be responsive
    Stick to the deadlines
    Answer as early as
    possible
    17
    

    View Slide

18. Conditional accepts
    ● Conditional accept will be possible
    ● Requires a clear and easily checkable explicit acceptance condition
    ○ Easily checkable = even the chair(s) can check it!
    ● One PC member should be willing to check the condition.
    ● Conditions most likely about presentation
    18
    

    View Slide

19. Open Science and Artefacts
    ● ICSE follows (new!) SIGSOFT Open Science policies
    ● Authors can upload supplementary material
    ● Reviewers can look at material, but don’t have to
    ● Once accepted, authors can submit material to the artifact evaluation track
    ○ For reusable, available, replicated or reproduced artifacts
    19
    

    View Slide

20. Double Blind
    ● Authors should do best effort to blind
    ● Reviewers should do best effort not to search / discover
    ● Clear violations will lead to (desk) reject
    ○ Checked in first week by chairs
    ○ If later discovered by reviewers instead, later desk reject still possible
    ● Reviewers can continue reviewing if they discover identity by accident
    ● Authors can publish preprint on e.g., Arxiv
    ● Further details on FAQ
    20
    

    View Slide

21. ICSE 2021
    Reviewing Process
    Arie van Deursen, TU Delft, The Netherlands
    Tao Xie, Peking University, China
    21
    

    View Slide