Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Academic Peer-Review in Software Engineering Venues

Academic Peer-Review in Software Engineering Venues

This slide is presented at a PhD workshop at Monash University, Australia.

Patanamon (Pick) Thongtanunam

November 26, 2020
Tweet

Transcript

  1. Patanamon (Pick) Thongtanunam [email protected] @patanamon ARC DECRA 2021 Fellow Lecturer

    at School of Computing and Information Systems (CIS) http://patanamon.com
  2. Patanamon (Pick) Thongtanunam [email protected] @patanamon ARC DECRA 2021 Fellow Lecturer

    at School of Computing and Information Systems (CIS) http://patanamon.com Quality-Impacting Practices in Software Development and Management Code Review A bug is here…
  3. Create tasks Write code Build & test code Integrate Release/

    Deploy A General View of Continuous Integration Code Review A bug is here… Quality-Impacting Practices in Software Development and Management
  4. Create tasks Write code Build & test code Integrate Release/

    Deploy A General View of Continuous Integration Code Review Code Review A bug is here… Quality-Impacting Practices in Software Development and Management
  5. Create tasks Write code Build & test code Integrate Release/

    Deploy A General View of Continuous Integration Code Review Quality-Impacting Practices in Software Development and Management What are the best practices with a minimum effort? Do socio-technical factors influence the practices? Can historical data tell some mistakes in the past?
  6. Disclaimer The slides are mainly based on the presenter's experience.

    The content does not represent the thoughts of reviewers in SE community. The content may be subjective. The content may not be generalized to all areas in SE. The slides are just finished 15 mins before the talk 
 Forgive me for some mistakes :)
  7. Why do I need to know or involve in academic

    peer reviews? Know what are the common concerns of reviewers, which can be applied to your paper
  8. Why do I need to know or involve in academic

    peer reviews? Know what are the common concerns of reviewers, which can be applied to your paper Understand how pain reviewers are when reading a paper; so that you make a good paper that ease readers :)
  9. Why do I need to know or involve in academic

    peer reviews? Know what are the common concerns of reviewers, which can be applied to your paper Understand how pain reviewers are when reading a paper; so that you make a good paper that ease readers :) Keep up with the recent literature
  10. Why do I need to know or involve in academic

    peer reviews? Know what are the common concerns of reviewers, which can be applied to your paper Understand how pain reviewers are when reading a paper; so that you make a good paper that ease readers :) Keep up with the recent literature Have a chance to read top-notch papers
  11. I’m too young. I can’t be a good reviewer ECR

    (<5 years after graduation) Being a PC member or a reviewer since 2017 11 full-length peer-review publications
  12. I’m too young. I can’t be a good reviewer ECR

    (<5 years after graduation) Being a PC member or a reviewer since 2017 11 full-length peer-review publications Distinguished Reviewer Board of TOSEM Distinguished Reviewers/PC members of -ICSME 2017 -ICSE 2020 -ASE 2020 -FSE 2020
  13. I’m too young. I can’t be a good reviewer ECR

    (<5 years after graduation) Being a PC member or a reviewer since 2017 11 full-length peer-review publications Distinguished Reviewer Board of TOSEM Distinguished Reviewers/PC members of -ICSME 2017 -ICSE 2020 -ASE 2020 -FSE 2020 First time as a PC member
  14. I’m too young. I can’t be a good reviewer ECR

    (<5 years after graduation) Being a PC member or a reviewer since 2017 11 full-length peer-review publications Distinguished Reviewer Board of TOSEM Distinguished Reviewers/PC members of -ICSME 2017 -ICSE 2020 -ASE 2020 -FSE 2020 First time as a PC member Young PC member 
 != Bad Reviewer But you should have some research experience :)
  15. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC

    Chairs pre- check manuscripts PC members bid (select) the papers they want to review
  16. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC

    Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts
  17. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC

    Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts Author(s) write a response to reviewers comments
  18. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC

    Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts PC members discuss Author(s) write a response to reviewers comments
  19. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected Accepted

    PC Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts PC members discuss Author(s) write a response to reviewers comments
  20. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected Accepted

    Rejected PC Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts PC members discuss Author(s) write a response to reviewers comments
  21. Review Criteria Soundness: The extent to which the paper’s contributions

    are supported by rigorous application of appropriate research methods Significance: The extent to which the paper’s contributions are important with respect to open software engineering challenges Novelty: The extent to which the contribution is sufficiently original and is clearly explained with respect to the state-of-the-art Verifiability: The extent to which the paper includes sufficient information to support independent verification or replication of the paper’s claimed contributions Presentation: The extent to which the paper’s quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc
  22. Review Criteria: Soundness Soundness: The extent to which the paper’s

    contributions are supported by rigorous application of appropriate research methods
  23. Review Criteria: Soundness Soundness: The extent to which the paper’s

    contributions are supported by rigorous application of appropriate research methods Are the study design, methods, evaluation rigorous and sounded?
  24. Review Criteria: Soundness Soundness: The extent to which the paper’s

    contributions are supported by rigorous application of appropriate research methods Are the study design, methods, evaluation rigorous and sounded? Is there any thing that could potentially lead to faulty results/outcomes? Is there any thing that could potentially change the empirical findings? Is the choice of the design is clearly justified (and convincing)?
  25. Review Criteria: Soundness Soundness: The extent to which the paper’s

    contributions are supported by rigorous application of appropriate research methods Are the study design, methods, evaluation rigorous and sounded? Is there any thing that could potentially lead to faulty results/outcomes? Is there any thing that could potentially change the empirical findings? Is the choice of the design is clearly justified (and convincing)? Sections: Case Study Design, Experiment, Evaluation, Results
  26. Review Criteria: Significance Significance: The extent to which the paper’s

    contributions are important with respect to open software engineering challenges
  27. Review Criteria: Significance Significance: The extent to which the paper’s

    contributions are important with respect to open software engineering challenges How this work helps SE 
 (either SE practitioners or research community)?
  28. Review Criteria: Significance Significance: The extent to which the paper’s

    contributions are important with respect to open software engineering challenges How this work helps SE 
 (either SE practitioners or research community)? Does it (a tool or finding) intuitively help SE practitioners? Will it be useful for future research? Does it address an important problem in SE?
  29. Review Criteria: Significance Significance: The extent to which the paper’s

    contributions are important with respect to open software engineering challenges How this work helps SE 
 (either SE practitioners or research community)? Does it (a tool or finding) intuitively help SE practitioners? Will it be useful for future research? Does it address an important problem in SE? Sections: Introduction, Background, Discussion, Conclusion
  30. Review Criteria: Novelty Novelty: The extent to which the contribution

    is sufficiently original and is clearly explained with respect to the state-of-the-art
  31. Review Criteria: Novelty Novelty: The extent to which the contribution

    is sufficiently original and is clearly explained with respect to the state-of-the-art How this work is different from the state-of-the-art?
  32. Review Criteria: Novelty Novelty: The extent to which the contribution

    is sufficiently original and is clearly explained with respect to the state-of-the-art How this work is different from the state-of-the-art? Do the work address a significant limitation of prior work? What is the gap between this work and prior work?
  33. Review Criteria: Novelty Novelty: The extent to which the contribution

    is sufficiently original and is clearly explained with respect to the state-of-the-art How this work is different from the state-of-the-art? Do the work address a significant limitation of prior work? What is the gap between this work and prior work? Sections: Introduction, Related Work, Discussion
  34. Review Criteria: Verifiability Verifiability: The extent to which the paper

    includes sufficient information to support independent verification or replication of the paper’s claimed contributions
  35. Review Criteria: Verifiability Verifiability: The extent to which the paper

    includes sufficient information to support independent verification or replication of the paper’s claimed contributions Can I replicate this work (if I want to)?
  36. Review Criteria: Verifiability Verifiability: The extent to which the paper

    includes sufficient information to support independent verification or replication of the paper’s claimed contributions Can I replicate this work (if I want to)? Is the detail of method/design sufficient and clear? Is there any key step that lacks the details?
  37. Review Criteria: Verifiability Verifiability: The extent to which the paper

    includes sufficient information to support independent verification or replication of the paper’s claimed contributions Can I replicate this work (if I want to)? Is the detail of method/design sufficient and clear? Is there any key step that lacks the details? Sections: Case Study Design, Experiment, Evaluation, Results
  38. Review Criteria: Presentation Presentation: The extent to which the paper’s

    quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc
  39. Review Criteria: Presentation Presentation: The extent to which the paper’s

    quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc Is the paper well-written, well-structured
 and easy to follow?
  40. Review Criteria: Presentation Presentation: The extent to which the paper’s

    quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc Is the paper well-written, well-structured
 and easy to follow? Can I grasp the key message in each paragraph/sections? Is the content is organised in a logical order? Is the complex part accompanied with a visual representation?
  41. Review Criteria: Presentation Presentation: The extent to which the paper’s

    quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc Is the paper well-written, well-structured
 and easy to follow? Can I grasp the key message in each paragraph/sections? Is the content is organised in a logical order? Is the complex part accompanied with a visual representation? Sections: Whole paper :)
  42. One Mistake Can Send You Home **This is just one

    possible consequence that I observed
  43. Presentation One Mistake Can Send You Home **This is just

    one possible consequence that I observed
  44. Presentation Verifiability Soundness One Mistake Can Send You Home **This

    is just one possible consequence that I observed
  45. Presentation Verifiability Soundness Novelty Significance One Mistake Can Send You

    Home **This is just one possible consequence that I observed
  46. How can I write a good paper? Writing a (good)

    review can help you write a good paper
  47. How can I write a good paper? Writing a (good)

    review can help you write a good paper How can I write a good review?
  48. How can I write a good paper? Writing a (good)

    review can help you write a good paper How can I write a good review? Learn from the reviewers of your papers
  49. How can I write a good paper? Writing a (good)

    review can help you write a good paper How can I write a good review? Learn from the reviewers of your papers If you like the reviews, just follow that style of reviews
  50. How can I write a good paper? Writing a (good)

    review can help you write a good paper How can I write a good review? Learn from the reviewers of your papers If you like the reviews, just follow that style of reviews If you don’t like reviews, just do better than that (you know what you want)
  51. How can I write a good paper? Writing a (good)

    review can help you write a good paper How can I write a good review? Learn from the reviewers of your papers If you like the reviews, just follow that style of reviews If you don’t like reviews, just do better than that (you know what you want) In my Reviews, I’m trying to Clearly explain “why” it is a concern Suggest how should the authors do Detangle between presentation and other aspects (is it poor because of writing or something else?)
  52. Some Anti-Patterns of Reviews Andreas Zeller’s April Fool’s Day Project

    Shallow Reviews: Reviews take only a very shallow, "syntactical" look into a paper without considering its potential and deeper implications. Cheap shots: A very generic critique to a paper in a manner that is usually not well-matched to the specific context Premkumar Devanbu’s slides
  53. Where can I learn more about writing a good review?

    Be a sub-reviewer of your supervisor
  54. Where can I learn more about writing a good review?

    Be a sub-reviewer of your supervisor Check “Birds of a feather: Reviewing SE research papers” at ICSE2020
  55. Where can I learn more about writing a good review?

    Be a sub-reviewer of your supervisor Check “Birds of a feather: Reviewing SE research papers” at ICSE2020 https://2021.msrconf.org/track/msr-2021-shadow-pc Gain first-hand experience as Shadow PC at MSR2021!