$30 off During Our Annual Pro Sale. View Details »

Academic Peer-Review in Software Engineering Venues

Academic Peer-Review in Software Engineering Venues

This slide is presented at a PhD workshop at Monash University, Australia.

Patanamon (Pick) Thongtanunam

November 26, 2020
Tweet

Transcript

  1. Patanamon (Pick) Thongtanunam
    [email protected]
    @patanamon
    ARC DECRA 2021 Fellow
    Lecturer at School of Computing and
    Information Systems (CIS)

    http://patanamon.com

    View Slide

  2. Patanamon (Pick) Thongtanunam
    [email protected]
    @patanamon
    ARC DECRA 2021 Fellow
    Lecturer at School of Computing and
    Information Systems (CIS)

    http://patanamon.com
    Quality-Impacting Practices in Software
    Development and Management
    Code Review
    A bug is here…

    View Slide

  3. Create
    tasks
    Write code
    Build &
    test code
    Integrate
    Release/
    Deploy
    A General View of Continuous Integration
    Code Review
    A bug is here…
    Quality-Impacting Practices in Software
    Development and Management

    View Slide

  4. Create
    tasks
    Write code
    Build &
    test code
    Integrate
    Release/
    Deploy
    A General View of Continuous Integration
    Code Review
    Code Review
    A bug is here…
    Quality-Impacting Practices in Software
    Development and Management

    View Slide

  5. Create
    tasks
    Write code
    Build &
    test code
    Integrate
    Release/
    Deploy
    A General View of Continuous Integration
    Code Review
    Quality-Impacting Practices in Software
    Development and Management
    What are the best practices
    with a minimum effort?
    Do socio-technical factors
    influence the practices?
    Can historical data tell some
    mistakes in the past?

    View Slide

  6. Academic Peer-Review in Software Engineering Venues

    View Slide

  7. Disclaimer
    The slides are mainly based on the presenter's experience.

    The content does not represent the thoughts of reviewers in SE community.

    The content may be subjective.

    The content may not be generalized to all areas in SE.

    The slides are just finished 15 mins before the talk 

    Forgive me for some mistakes :)

    View Slide

  8. View Slide

  9. View Slide

  10. Why do I need to know or involve
    in academic peer reviews?

    View Slide

  11. Why do I need to know or involve
    in academic peer reviews?
    Know what are the common concerns of
    reviewers, which can be applied to your paper

    View Slide

  12. Why do I need to know or involve
    in academic peer reviews?
    Know what are the common concerns of
    reviewers, which can be applied to your paper
    Understand how pain reviewers are when
    reading a paper; so that you make a good
    paper that ease readers :)

    View Slide

  13. Why do I need to know or involve
    in academic peer reviews?
    Know what are the common concerns of
    reviewers, which can be applied to your paper
    Understand how pain reviewers are when
    reading a paper; so that you make a good
    paper that ease readers :)
    Keep up with the recent literature

    View Slide

  14. Why do I need to know or involve
    in academic peer reviews?
    Know what are the common concerns of
    reviewers, which can be applied to your paper
    Understand how pain reviewers are when
    reading a paper; so that you make a good
    paper that ease readers :)
    Keep up with the recent literature
    Have a chance to read top-notch papers

    View Slide

  15. I’m too young. I can’t be a good
    reviewer

    View Slide

  16. I’m too young. I can’t be a good
    reviewer

    View Slide

  17. I’m too young. I can’t be a good
    reviewer
    ECR (<5 years after graduation)

    Being a PC member or a
    reviewer since 2017

    11 full-length peer-review
    publications

    View Slide

  18. I’m too young. I can’t be a good
    reviewer
    ECR (<5 years after graduation)

    Being a PC member or a
    reviewer since 2017

    11 full-length peer-review
    publications
    Distinguished Reviewer Board of TOSEM
    Distinguished Reviewers/PC members of
    -ICSME 2017

    -ICSE 2020

    -ASE 2020

    -FSE 2020

    View Slide

  19. I’m too young. I can’t be a good
    reviewer
    ECR (<5 years after graduation)

    Being a PC member or a
    reviewer since 2017

    11 full-length peer-review
    publications
    Distinguished Reviewer Board of TOSEM
    Distinguished Reviewers/PC members of
    -ICSME 2017

    -ICSE 2020

    -ASE 2020

    -FSE 2020
    First time as a PC
    member

    View Slide

  20. I’m too young. I can’t be a good
    reviewer
    ECR (<5 years after graduation)

    Being a PC member or a
    reviewer since 2017

    11 full-length peer-review
    publications
    Distinguished Reviewer Board of TOSEM
    Distinguished Reviewers/PC members of
    -ICSME 2017

    -ICSE 2020

    -ASE 2020

    -FSE 2020
    First time as a PC
    member
    Young PC member 

    != Bad Reviewer
    But you should have some research
    experience :)

    View Slide

  21. Peer-Review Process in SE
    Author(s) submit
    manuscript

    View Slide

  22. Peer-Review Process in SE
    Author(s) submit
    manuscript
    PC Chairs pre-
    check manuscripts

    View Slide

  23. Peer-Review Process in SE
    Author(s) submit
    manuscript
    Desk Rejected
    PC Chairs pre-
    check manuscripts

    View Slide

  24. Peer-Review Process in SE
    Author(s) submit
    manuscript
    Desk Rejected
    PC Chairs pre-
    check manuscripts
    PC members bid
    (select) the papers
    they want to review

    View Slide

  25. Peer-Review Process in SE
    Author(s) submit
    manuscript
    Desk Rejected
    PC Chairs pre-
    check manuscripts
    PC members bid
    (select) the papers
    they want to review
    Papers assigned to
    PC members;
    review starts

    View Slide

  26. Peer-Review Process in SE
    Author(s) submit
    manuscript
    Desk Rejected
    PC Chairs pre-
    check manuscripts
    PC members bid
    (select) the papers
    they want to review
    Papers assigned to
    PC members;
    review starts
    Author(s) write a
    response to
    reviewers comments

    View Slide

  27. Peer-Review Process in SE
    Author(s) submit
    manuscript
    Desk Rejected
    PC Chairs pre-
    check manuscripts
    PC members bid
    (select) the papers
    they want to review
    Papers assigned to
    PC members;
    review starts
    PC members
    discuss
    Author(s) write a
    response to
    reviewers comments

    View Slide

  28. Peer-Review Process in SE
    Author(s) submit
    manuscript
    Desk Rejected
    Accepted
    PC Chairs pre-
    check manuscripts
    PC members bid
    (select) the papers
    they want to review
    Papers assigned to
    PC members;
    review starts
    PC members
    discuss
    Author(s) write a
    response to
    reviewers comments

    View Slide

  29. Peer-Review Process in SE
    Author(s) submit
    manuscript
    Desk Rejected
    Accepted Rejected
    PC Chairs pre-
    check manuscripts
    PC members bid
    (select) the papers
    they want to review
    Papers assigned to
    PC members;
    review starts
    PC members
    discuss
    Author(s) write a
    response to
    reviewers comments

    View Slide

  30. Review Criteria
    Soundness: The extent to which the paper’s contributions are supported by rigorous
    application of appropriate research methods

    Significance: The extent to which the paper’s contributions are important with respect to
    open software engineering challenges

    Novelty: The extent to which the contribution is sufficiently original and is clearly explained
    with respect to the state-of-the-art

    Verifiability: The extent to which the paper includes sufficient information to support
    independent verification or replication of the paper’s claimed contributions

    Presentation: The extent to which the paper’s quality of writing meets the high standards,
    including clear descriptions and explanations, absence of major ambiguity, etc

    View Slide

  31. Review Criteria: Soundness
    Soundness: The extent to which the paper’s contributions are supported by rigorous
    application of appropriate research methods

    View Slide

  32. Review Criteria: Soundness
    Soundness: The extent to which the paper’s contributions are supported by rigorous
    application of appropriate research methods
    Are the study design, methods, evaluation
    rigorous and sounded?

    View Slide

  33. Review Criteria: Soundness
    Soundness: The extent to which the paper’s contributions are supported by rigorous
    application of appropriate research methods
    Are the study design, methods, evaluation
    rigorous and sounded?
    Is there any thing that could potentially
    lead to faulty results/outcomes?
    Is there any thing that could potentially
    change the empirical findings?
    Is the choice of the design is clearly
    justified (and convincing)?

    View Slide

  34. Review Criteria: Soundness
    Soundness: The extent to which the paper’s contributions are supported by rigorous
    application of appropriate research methods
    Are the study design, methods, evaluation
    rigorous and sounded?
    Is there any thing that could potentially
    lead to faulty results/outcomes?
    Is there any thing that could potentially
    change the empirical findings?
    Is the choice of the design is clearly
    justified (and convincing)?
    Sections: Case Study Design,
    Experiment, Evaluation, Results

    View Slide

  35. Review Criteria: Significance
    Significance: The extent to which the paper’s contributions are important with respect to
    open software engineering challenges

    View Slide

  36. Review Criteria: Significance
    Significance: The extent to which the paper’s contributions are important with respect to
    open software engineering challenges
    How this work helps SE 

    (either SE practitioners or research community)?

    View Slide

  37. Review Criteria: Significance
    Significance: The extent to which the paper’s contributions are important with respect to
    open software engineering challenges
    How this work helps SE 

    (either SE practitioners or research community)?
    Does it (a tool or finding) intuitively help
    SE practitioners?
    Will it be useful for future research?
    Does it address an important
    problem in SE?

    View Slide

  38. Review Criteria: Significance
    Significance: The extent to which the paper’s contributions are important with respect to
    open software engineering challenges
    How this work helps SE 

    (either SE practitioners or research community)?
    Does it (a tool or finding) intuitively help
    SE practitioners?
    Will it be useful for future research?
    Does it address an important
    problem in SE?
    Sections: Introduction, Background,
    Discussion, Conclusion

    View Slide

  39. Review Criteria: Novelty
    Novelty: The extent to which the contribution is sufficiently original and is clearly explained
    with respect to the state-of-the-art

    View Slide

  40. Review Criteria: Novelty
    Novelty: The extent to which the contribution is sufficiently original and is clearly explained
    with respect to the state-of-the-art
    How this work is different from the state-of-the-art?

    View Slide

  41. Review Criteria: Novelty
    Novelty: The extent to which the contribution is sufficiently original and is clearly explained
    with respect to the state-of-the-art
    How this work is different from the state-of-the-art?
    Do the work address a significant
    limitation of prior work?
    What is the gap between this work
    and prior work?

    View Slide

  42. Review Criteria: Novelty
    Novelty: The extent to which the contribution is sufficiently original and is clearly explained
    with respect to the state-of-the-art
    How this work is different from the state-of-the-art?
    Do the work address a significant
    limitation of prior work?
    What is the gap between this work
    and prior work?
    Sections: Introduction, Related
    Work, Discussion

    View Slide

  43. Review Criteria: Verifiability
    Verifiability: The extent to which the paper includes sufficient information to support
    independent verification or replication of the paper’s claimed contributions

    View Slide

  44. Review Criteria: Verifiability
    Verifiability: The extent to which the paper includes sufficient information to support
    independent verification or replication of the paper’s claimed contributions
    Can I replicate this work (if I want to)?

    View Slide

  45. Review Criteria: Verifiability
    Verifiability: The extent to which the paper includes sufficient information to support
    independent verification or replication of the paper’s claimed contributions
    Can I replicate this work (if I want to)?
    Is the detail of method/design sufficient
    and clear?
    Is there any key step that lacks the
    details?

    View Slide

  46. Review Criteria: Verifiability
    Verifiability: The extent to which the paper includes sufficient information to support
    independent verification or replication of the paper’s claimed contributions
    Can I replicate this work (if I want to)?
    Is the detail of method/design sufficient
    and clear?
    Is there any key step that lacks the
    details?
    Sections: Case Study Design,
    Experiment, Evaluation, Results

    View Slide

  47. Review Criteria: Presentation
    Presentation: The extent to which the paper’s quality of writing meets the high standards,
    including clear descriptions and explanations, absence of major ambiguity, etc

    View Slide

  48. Review Criteria: Presentation
    Presentation: The extent to which the paper’s quality of writing meets the high standards,
    including clear descriptions and explanations, absence of major ambiguity, etc
    Is the paper well-written, well-structured

    and easy to follow?

    View Slide

  49. Review Criteria: Presentation
    Presentation: The extent to which the paper’s quality of writing meets the high standards,
    including clear descriptions and explanations, absence of major ambiguity, etc
    Is the paper well-written, well-structured

    and easy to follow?
    Can I grasp the key message in each
    paragraph/sections?
    Is the content is organised in a
    logical order?
    Is the complex part accompanied with a
    visual representation?

    View Slide

  50. Review Criteria: Presentation
    Presentation: The extent to which the paper’s quality of writing meets the high standards,
    including clear descriptions and explanations, absence of major ambiguity, etc
    Is the paper well-written, well-structured

    and easy to follow?
    Can I grasp the key message in each
    paragraph/sections?
    Is the content is organised in a
    logical order?
    Is the complex part accompanied with a
    visual representation?
    Sections: Whole paper :)

    View Slide

  51. One Mistake Can Send You Home
    Your
    manuscript
    You
    Reviewers

    View Slide

  52. One Mistake Can Send You Home
    **This is just one possible consequence that I observed

    View Slide

  53. Presentation
    One Mistake Can Send You Home
    **This is just one possible consequence that I observed

    View Slide

  54. Presentation
    Verifiability
    Soundness
    One Mistake Can Send You Home
    **This is just one possible consequence that I observed

    View Slide

  55. Presentation
    Verifiability
    Soundness
    Novelty
    Significance
    One Mistake Can Send You Home
    **This is just one possible consequence that I observed

    View Slide

  56. How can I write a good
    paper?

    View Slide

  57. How can I write a good
    paper?
    Writing a (good) review can help
    you write a good paper

    View Slide

  58. How can I write a good
    paper?
    Writing a (good) review can help
    you write a good paper
    How can I write a good
    review?

    View Slide

  59. How can I write a good
    paper?
    Writing a (good) review can help
    you write a good paper
    How can I write a good
    review?
    Learn from the reviewers of your
    papers

    View Slide

  60. How can I write a good
    paper?
    Writing a (good) review can help
    you write a good paper
    How can I write a good
    review?
    Learn from the reviewers of your
    papers
    If you like the reviews, just follow that style of
    reviews

    View Slide

  61. How can I write a good
    paper?
    Writing a (good) review can help
    you write a good paper
    How can I write a good
    review?
    Learn from the reviewers of your
    papers
    If you like the reviews, just follow that style of
    reviews
    If you don’t like reviews, just do better
    than that (you know what you want)

    View Slide

  62. How can I write a good
    paper?
    Writing a (good) review can help
    you write a good paper
    How can I write a good
    review?
    Learn from the reviewers of your
    papers
    If you like the reviews, just follow that style of
    reviews
    If you don’t like reviews, just do better
    than that (you know what you want)
    In my Reviews, I’m trying to
    Clearly explain “why” it is a concern
    Suggest how should the authors do
    Detangle between presentation and other
    aspects (is it poor because of writing or
    something else?)

    View Slide

  63. Some Anti-Patterns of Reviews
    Andreas Zeller’s April Fool’s Day Project
    Shallow Reviews: Reviews take only a very shallow,
    "syntactical" look into a paper without considering its potential
    and deeper implications.
    Cheap shots: A very generic critique to a paper in a manner
    that is usually not well-matched to the specific context
    Premkumar Devanbu’s slides

    View Slide

  64. Where can I learn more
    about writing a good review?

    View Slide

  65. Where can I learn more
    about writing a good review?
    Be a sub-reviewer of your
    supervisor

    View Slide

  66. Where can I learn more
    about writing a good review?
    Be a sub-reviewer of your
    supervisor
    Check “Birds of a feather: Reviewing SE research
    papers” at ICSE2020

    View Slide

  67. Where can I learn more
    about writing a good review?
    Be a sub-reviewer of your
    supervisor
    Check “Birds of a feather: Reviewing SE research
    papers” at ICSE2020
    https://2021.msrconf.org/track/msr-2021-shadow-pc
    Gain first-hand experience as Shadow PC at MSR2021!

    View Slide

  68. [email protected]
    @patanamon
    http://patanamon.com
    Thank You & Happy Reviewing!

    View Slide