Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ICSE 2021 Reviewing Process

ICSE 2021 Reviewing Process

Slide deck used to discuss the reviewing process of the ACM/IEEE International Conference on Software Engineering (ICSE) with the full program committee (PC).

The slide deck covers the PC roles, responsibilities, composition, the time line, and details about the reviewing and discussion phase.

The slide formed the basis for a video recorded by the program chairs (Tao Xie and Arie van Deursen), available on YouTube at https://www.youtube.com/watch?v=q5gnvMA5FdA.

The video was discussed at six sessions in three different time bands, in which the over 150 PC members would get to know each other and discuss the proposed reviewing process.

The actual reviewing process may deviate in details, but will be quite similar to what's described here. For more info on ICSE see https://conf.researchr.org/home/icse-2021.

Arie van Deursen

August 06, 2020
Tweet

More Decks by Arie van Deursen

Other Decks in Research

Transcript

  1. ICSE 2021
    Reviewing Process
    Arie van Deursen, TU Delft, The Netherlands
    Tao Xie, Peking University, China
    1

    View Slide

  2. Objectives
    ● Accept high quality papers in any topic in software engineering
    ● Give clear feedback to papers of insufficient quality
    ● Judge different papers against the same bar
    ● Ensure safe and transparent process
    ● Ensure decisions are reached with at least 5 people involved (10 eyes)
    ● Have a process that scales up to 600+ papers
    ● Ensure people involved have manageable load
    2

    View Slide

  3. ICSE 2021 PC Roles
    ● Reviewer:
    ○ Writes high quality review
    ○ Participates in discussion
    ● Moderator:
    ○ Moderates discussion among three reviewers
    ○ Builds consensus where possible
    ○ Proposes a decision
    ○ Write meta-review, explaining decision
    ● Area chair:
    ○ Ensures consistency for papers within area, and across areas
    ○ Confirms (challenges) proposed decisions
    ○ Ensures all steps are taken in a timely manner
    ○ Basically act as “proxy chair” in final discussions
    ○ (also suggest PC members during PC formation)
    In contrast to ICSE 2020:
    ● Every PC member
    has reviewer and
    moderation roles
    ● No separate
    “program board”
    ● New “Area Chairs”:
    moderation role
    replaced with area
    chair role
    3

    View Slide

  4. Area Chairs
    Help further scaling of ICSE
    Main topics and chairs for 2021:
    ● AI and software engineering: Dongmei Zhang (Microsoft Research)
    ● Testing and analysis: Anders Møller (Aarhus University)
    ● Empirical software engineering: Tom Zimmermann (Microsoft Research)
    ● Software evolution: Lori Pollock (U. Delaware)
    ● Social aspects of software engineering: Daniela Damian (U. Victoria)
    ● Requirements, modeling and design: Zhi Jin (Peking University)
    ● Dependability; Eric Bodden (Paderborn University)
    4

    View Slide

  5. Seven Key Areas and their 40 Sub-Topics
    5

    View Slide

  6. Who is a Paper’s Area Chair?
    ● Areas have (unavoidable) overlap
    ● PC members indicate their expertise on 40 sub-topics
    ● Authors mark their paper as relevant to multiple sub-topics
    ● For each paper calculate fit with each area
    ● Papers in principle assigned to best fitting area chair
    ● If area size is too unbalanced, we’ll move papers to other (fitting) areas
    6

    View Slide

  7. PC Formation
    ● 158 program committee members accepted, of which 7 area chairs
    ● Male/female: 61/39%
    ● North America / EU / Asia / Other: 40/30/20/10%
    ● Academia / Industry: 91/9%
    ● Reviewing / Moderation / No preference: 47/24/29%
    7

    View Slide

  8. Expected Reviewer & Moderation Load
    Assuming 600 papers (similar to 2020):
    8

    View Slide

  9. HotCrp Conflict Of Interest Declarations
    ● List all potential authors you have a conflict with!
    ● Copy from earlier hotcrp installations
    ● Ensure they are up to date by August 15
    Different from
    EasyChair
    Please do this
    in advance
    9

    View Slide

  10. Paper Bidding
    Ensure your paper topics in hotcrp are up to date
    Place positive bids on papers that you have the expertise to review
    If you think you’re one of the few on the PC with expertise on a paper make sure
    to place a high bid!
    Avoid placing bids on papers you’d like to learn about, but where you have no
    expertise.
    Place at least 45 positive bids
    10

    View Slide

  11. Reviewing Phases I: Review Writing
    ● Bidding [ 3 working days ]
    ● Review round 1 (50% of papers) [ 4 weeks ]
    ● Review round 2 (100% of papers) [ 3 weeks ]
    ● Quality gate [ 1 week ]
    ○ Moderators give feedback on reviews
    ○ Moderators identify papers needing replacement reviewer
    ● Review round 3 (±1 emergency paper) [ 1 week ]
    ● Discussion 1 (pre-rebuttal) [ earlier weeks, plus 4 days ]
    ● Rebuttal phase (by authors) [ 3 working days ]
    11

    View Slide

  12. Reviewing Phases II: Discussions (1)
    ● Discussion 1: Pre-rebuttal [ 4 days ]
    ○ Can start as soon as three reviews are in (not intended to build consensus for decision making)
    ○ Goal: Identify what extra information from authors could support decision making
    ○ Result: Questions for authors (no decisions!)
    rebuttal period
    12

    View Slide

  13. Reviewing Phases II: Discussions (2)
    ● Discussion 2: Clear cases, after rebuttal [ 5 days]
    ○ After rebuttal
    ○ Identify papers with only rejects or only accepts (estimate: 50%)
    ○ Result: Decision + meta-review for first half of papers
    ● Discussion 3: Mixed cases [ 5 + 5 days ]
    ○ Make decision for papers with mixed score where consensus can be built
    ○ Result 1: Decision + meta-review for another 25% of the papers.
    ○ Result 2: Identification of papers without consensus
    ● Discussions 4: No-consensus [ 5 + 5 + 5 days ]
    ○ Moderator is in charge
    ○ Active involvement from area chair
    ○ Reach decision for final 25%
    5 days
    5 days
    rebuttal period
    5 days
    13

    View Slide

  14. Reviewing Phases III: Final Decisions
    ● Chair Meeting: [ 2 days ]
    ○ Cover all remaining abnormal cases
    ○ (Area) chairs finalize all decisions.
    ○ PC members stand-by for fast responses on papers that were still discussed in phase 4.
    14

    View Slide

  15. Full Timeline
    15

    View Slide

  16. Review Guidelines
    ● Write a summary in your own words
    ● Collect feedback
    ● Give thoughts on 5 ICSE criteria:
    ○ Soundness, significance, novelty, verifiability, presentation
    ● Identify the decisive factors resulting in acceptance
    ● Identify clarification questions that can help in the decision making
    ● Be polite, constructive, informative, …
    ● Seek for reasons for acceptance (accept unless you see a blocker)
    Commit to the deadlines
    Start as early as possible
    Submit your review as
    soon as it is done
    16

    View Slide

  17. Discussion Guidelines
    ● Be polite and friendly
    ● Be willing to move
    ● Seek for reasons for acceptance
    ● Update reviews where necessary
    ● Focus on the key decision factors
    ● Let final meta-review reflect key factors resulting in the decision.
    Be responsive
    Stick to the deadlines
    Answer as early as
    possible
    17

    View Slide

  18. Conditional accepts
    ● Conditional accept will be possible
    ● Requires a clear and easily checkable explicit acceptance condition
    ○ Easily checkable = even the chair(s) can check it!
    ● One PC member should be willing to check the condition.
    ● Conditions most likely about presentation
    18

    View Slide

  19. Open Science and Artefacts
    ● ICSE follows (new!) SIGSOFT Open Science policies
    ● Authors can upload supplementary material
    ● Reviewers can look at material, but don’t have to
    ● Once accepted, authors can submit material to the artifact evaluation track
    ○ For reusable, available, replicated or reproduced artifacts
    19

    View Slide

  20. Double Blind
    ● Authors should do best effort to blind
    ● Reviewers should do best effort not to search / discover
    ● Clear violations will lead to (desk) reject
    ○ Checked in first week by chairs
    ○ If later discovered by reviewers instead, later desk reject still possible
    ● Reviewers can continue reviewing if they discover identity by accident
    ● Authors can publish preprint on e.g., Arxiv
    ● Further details on FAQ
    20

    View Slide

  21. ICSE 2021
    Reviewing Process
    Arie van Deursen, TU Delft, The Netherlands
    Tao Xie, Peking University, China
    21

    View Slide