Academic Peer-Review in Software Engineering Venues

This slide is presented at a PhD workshop at Monash University, Australia.

Transcript

1. Patanamon (Pick) Thongtanunam patanamon.t@unimelb.edu.au @patanamon ARC DECRA 2021 Fellow Lecturer at School of Computing and Information Systems (CIS) http://patanamon.com
2. Patanamon (Pick) Thongtanunam patanamon.t@unimelb.edu.au @patanamon ARC DECRA 2021 Fellow Lecturer at School of Computing and Information Systems (CIS) http://patanamon.com Quality-Impacting Practices in Software Development and Management Code Review A bug is here…
3. Create tasks Write code Build & test code Integrate Release/ Deploy A General View of Continuous Integration Code Review A bug is here… Quality-Impacting Practices in Software Development and Management
4. Create tasks Write code Build & test code Integrate Release/ Deploy A General View of Continuous Integration Code Review Code Review A bug is here… Quality-Impacting Practices in Software Development and Management
5. Create tasks Write code Build & test code Integrate Release/ Deploy A General View of Continuous Integration Code Review Quality-Impacting Practices in Software Development and Management What are the best practices with a minimum effort? Do socio-technical factors influence the practices? Can historical data tell some mistakes in the past?
6. Academic Peer-Review in Software Engineering Venues
7. Disclaimer The slides are mainly based on the presenter's experience. The content does not represent the thoughts of reviewers in SE community. The content may be subjective. The content may not be generalized to all areas in SE. The slides are just finished 15 mins before the talk 
 Forgive me for some mistakes :)
10. Why do I need to know or involve in academic peer reviews?
11. Why do I need to know or involve in academic peer reviews? Know what are the common concerns of reviewers, which can be applied to your paper
12. Why do I need to know or involve in academic peer reviews? Know what are the common concerns of reviewers, which can be applied to your paper Understand how pain reviewers are when reading a paper; so that you make a good paper that ease readers :)
13. Why do I need to know or involve in academic peer reviews? Know what are the common concerns of reviewers, which can be applied to your paper Understand how pain reviewers are when reading a paper; so that you make a good paper that ease readers :) Keep up with the recent literature
14. Why do I need to know or involve in academic peer reviews? Know what are the common concerns of reviewers, which can be applied to your paper Understand how pain reviewers are when reading a paper; so that you make a good paper that ease readers :) Keep up with the recent literature Have a chance to read top-notch papers
15. I’m too young. I can’t be a good reviewer
16. I’m too young. I can’t be a good reviewer
17. I’m too young. I can’t be a good reviewer ECR (<5 years after graduation) Being a PC member or a reviewer since 2017 11 full-length peer-review publications
18. I’m too young. I can’t be a good reviewer ECR (<5 years after graduation) Being a PC member or a reviewer since 2017 11 full-length peer-review publications Distinguished Reviewer Board of TOSEM Distinguished Reviewers/PC members of -ICSME 2017 -ICSE 2020 -ASE 2020 -FSE 2020
19. I’m too young. I can’t be a good reviewer ECR (<5 years after graduation) Being a PC member or a reviewer since 2017 11 full-length peer-review publications Distinguished Reviewer Board of TOSEM Distinguished Reviewers/PC members of -ICSME 2017 -ICSE 2020 -ASE 2020 -FSE 2020 First time as a PC member
20. I’m too young. I can’t be a good reviewer ECR (<5 years after graduation) Being a PC member or a reviewer since 2017 11 full-length peer-review publications Distinguished Reviewer Board of TOSEM Distinguished Reviewers/PC members of -ICSME 2017 -ICSE 2020 -ASE 2020 -FSE 2020 First time as a PC member Young PC member 
 != Bad Reviewer But you should have some research experience :)
21. Peer-Review Process in SE Author(s) submit manuscript
22. Peer-Review Process in SE Author(s) submit manuscript PC Chairs pre- check manuscripts
23. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC Chairs pre- check manuscripts
24. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC Chairs pre- check manuscripts PC members bid (select) the papers they want to review
25. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts
26. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts Author(s) write a response to reviewers comments
27. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts PC members discuss Author(s) write a response to reviewers comments
28. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected Accepted PC Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts PC members discuss Author(s) write a response to reviewers comments
29. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected Accepted Rejected PC Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts PC members discuss Author(s) write a response to reviewers comments
30. Review Criteria Soundness: The extent to which the paper’s contributions are supported by rigorous application of appropriate research methods Significance: The extent to which the paper’s contributions are important with respect to open software engineering challenges Novelty: The extent to which the contribution is sufficiently original and is clearly explained with respect to the state-of-the-art Verifiability: The extent to which the paper includes sufficient information to support independent verification or replication of the paper’s claimed contributions Presentation: The extent to which the paper’s quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc
31. Review Criteria: Soundness Soundness: The extent to which the paper’s contributions are supported by rigorous application of appropriate research methods
32. Review Criteria: Soundness Soundness: The extent to which the paper’s contributions are supported by rigorous application of appropriate research methods Are the study design, methods, evaluation rigorous and sounded?
33. Review Criteria: Soundness Soundness: The extent to which the paper’s contributions are supported by rigorous application of appropriate research methods Are the study design, methods, evaluation rigorous and sounded? Is there any thing that could potentially lead to faulty results/outcomes? Is there any thing that could potentially change the empirical findings? Is the choice of the design is clearly justified (and convincing)?
34. Review Criteria: Soundness Soundness: The extent to which the paper’s contributions are supported by rigorous application of appropriate research methods Are the study design, methods, evaluation rigorous and sounded? Is there any thing that could potentially lead to faulty results/outcomes? Is there any thing that could potentially change the empirical findings? Is the choice of the design is clearly justified (and convincing)? Sections: Case Study Design, Experiment, Evaluation, Results
35. Review Criteria: Significance Significance: The extent to which the paper’s contributions are important with respect to open software engineering challenges
36. Review Criteria: Significance Significance: The extent to which the paper’s contributions are important with respect to open software engineering challenges How this work helps SE 
 (either SE practitioners or research community)?
37. Review Criteria: Significance Significance: The extent to which the paper’s contributions are important with respect to open software engineering challenges How this work helps SE 
 (either SE practitioners or research community)? Does it (a tool or finding) intuitively help SE practitioners? Will it be useful for future research? Does it address an important problem in SE?
38. Review Criteria: Significance Significance: The extent to which the paper’s contributions are important with respect to open software engineering challenges How this work helps SE 
 (either SE practitioners or research community)? Does it (a tool or finding) intuitively help SE practitioners? Will it be useful for future research? Does it address an important problem in SE? Sections: Introduction, Background, Discussion, Conclusion
39. Review Criteria: Novelty Novelty: The extent to which the contribution is sufficiently original and is clearly explained with respect to the state-of-the-art
40. Review Criteria: Novelty Novelty: The extent to which the contribution is sufficiently original and is clearly explained with respect to the state-of-the-art How this work is different from the state-of-the-art?
41. Review Criteria: Novelty Novelty: The extent to which the contribution is sufficiently original and is clearly explained with respect to the state-of-the-art How this work is different from the state-of-the-art? Do the work address a significant limitation of prior work? What is the gap between this work and prior work?
42. Review Criteria: Novelty Novelty: The extent to which the contribution is sufficiently original and is clearly explained with respect to the state-of-the-art How this work is different from the state-of-the-art? Do the work address a significant limitation of prior work? What is the gap between this work and prior work? Sections: Introduction, Related Work, Discussion
43. Review Criteria: Verifiability Verifiability: The extent to which the paper includes sufficient information to support independent verification or replication of the paper’s claimed contributions
44. Review Criteria: Verifiability Verifiability: The extent to which the paper includes sufficient information to support independent verification or replication of the paper’s claimed contributions Can I replicate this work (if I want to)?
45. Review Criteria: Verifiability Verifiability: The extent to which the paper includes sufficient information to support independent verification or replication of the paper’s claimed contributions Can I replicate this work (if I want to)? Is the detail of method/design sufficient and clear? Is there any key step that lacks the details?
46. Review Criteria: Verifiability Verifiability: The extent to which the paper includes sufficient information to support independent verification or replication of the paper’s claimed contributions Can I replicate this work (if I want to)? Is the detail of method/design sufficient and clear? Is there any key step that lacks the details? Sections: Case Study Design, Experiment, Evaluation, Results
47. Review Criteria: Presentation Presentation: The extent to which the paper’s quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc
48. Review Criteria: Presentation Presentation: The extent to which the paper’s quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc Is the paper well-written, well-structured
 and easy to follow?
49. Review Criteria: Presentation Presentation: The extent to which the paper’s quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc Is the paper well-written, well-structured
 and easy to follow? Can I grasp the key message in each paragraph/sections? Is the content is organised in a logical order? Is the complex part accompanied with a visual representation?
50. Review Criteria: Presentation Presentation: The extent to which the paper’s quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc Is the paper well-written, well-structured
 and easy to follow? Can I grasp the key message in each paragraph/sections? Is the content is organised in a logical order? Is the complex part accompanied with a visual representation? Sections: Whole paper :)
51. One Mistake Can Send You Home Your manuscript You Reviewers
52. One Mistake Can Send You Home **This is just one possible consequence that I observed
53. Presentation One Mistake Can Send You Home **This is just one possible consequence that I observed
54. Presentation Verifiability Soundness One Mistake Can Send You Home **This is just one possible consequence that I observed
55. Presentation Verifiability Soundness Novelty Significance One Mistake Can Send You Home **This is just one possible consequence that I observed
56. How can I write a good paper?
57. How can I write a good paper? Writing a (good) review can help you write a good paper
58. How can I write a good paper? Writing a (good) review can help you write a good paper How can I write a good review?
59. How can I write a good paper? Writing a (good) review can help you write a good paper How can I write a good review? Learn from the reviewers of your papers
60. How can I write a good paper? Writing a (good) review can help you write a good paper How can I write a good review? Learn from the reviewers of your papers If you like the reviews, just follow that style of reviews
61. How can I write a good paper? Writing a (good) review can help you write a good paper How can I write a good review? Learn from the reviewers of your papers If you like the reviews, just follow that style of reviews If you don’t like reviews, just do better than that (you know what you want)
62. How can I write a good paper? Writing a (good) review can help you write a good paper How can I write a good review? Learn from the reviewers of your papers If you like the reviews, just follow that style of reviews If you don’t like reviews, just do better than that (you know what you want) In my Reviews, I’m trying to Clearly explain “why” it is a concern Suggest how should the authors do Detangle between presentation and other aspects (is it poor because of writing or something else?)
63. Some Anti-Patterns of Reviews Andreas Zeller’s April Fool’s Day Project Shallow Reviews: Reviews take only a very shallow, "syntactical" look into a paper without considering its potential and deeper implications. Cheap shots: A very generic critique to a paper in a manner that is usually not well-matched to the specific context Premkumar Devanbu’s slides
64. Where can I learn more about writing a good review?
65. Where can I learn more about writing a good review? Be a sub-reviewer of your supervisor
66. Where can I learn more about writing a good review? Be a sub-reviewer of your supervisor Check “Birds of a feather: Reviewing SE research papers” at ICSE2020
67. Where can I learn more about writing a good review? Be a sub-reviewer of your supervisor Check “Birds of a feather: Reviewing SE research papers” at ICSE2020 https://2021.msrconf.org/track/msr-2021-shadow-pc Gain first-hand experience as Shadow PC at MSR2021!
68. patanamon.t@unimelb.edu.au @patanamon http://patanamon.com Thank You & Happy Reviewing!