Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Academic Peer-Review in Software Engineering Venues

Academic Peer-Review in Software Engineering Venues

This slide is presented at a PhD workshop at Monash University, Australia.

Af063a6baa6f4b5543c76c2fad554a06?s=128

Patanamon (Pick) Thongtanunam

November 26, 2020
Tweet

Transcript

  1. Patanamon (Pick) Thongtanunam patanamon.t@unimelb.edu.au @patanamon ARC DECRA 2021 Fellow Lecturer

    at School of Computing and Information Systems (CIS) http://patanamon.com
  2. Patanamon (Pick) Thongtanunam patanamon.t@unimelb.edu.au @patanamon ARC DECRA 2021 Fellow Lecturer

    at School of Computing and Information Systems (CIS) http://patanamon.com Quality-Impacting Practices in Software Development and Management Code Review A bug is here…
  3. Create tasks Write code Build & test code Integrate Release/

    Deploy A General View of Continuous Integration Code Review A bug is here… Quality-Impacting Practices in Software Development and Management
  4. Create tasks Write code Build & test code Integrate Release/

    Deploy A General View of Continuous Integration Code Review Code Review A bug is here… Quality-Impacting Practices in Software Development and Management
  5. Create tasks Write code Build & test code Integrate Release/

    Deploy A General View of Continuous Integration Code Review Quality-Impacting Practices in Software Development and Management What are the best practices with a minimum effort? Do socio-technical factors influence the practices? Can historical data tell some mistakes in the past?
  6. Academic Peer-Review in Software Engineering Venues

  7. Disclaimer The slides are mainly based on the presenter's experience.

    The content does not represent the thoughts of reviewers in SE community. The content may be subjective. The content may not be generalized to all areas in SE. The slides are just finished 15 mins before the talk 
 Forgive me for some mistakes :)
  8. None
  9. None
  10. Why do I need to know or involve in academic

    peer reviews?
  11. Why do I need to know or involve in academic

    peer reviews? Know what are the common concerns of reviewers, which can be applied to your paper
  12. Why do I need to know or involve in academic

    peer reviews? Know what are the common concerns of reviewers, which can be applied to your paper Understand how pain reviewers are when reading a paper; so that you make a good paper that ease readers :)
  13. Why do I need to know or involve in academic

    peer reviews? Know what are the common concerns of reviewers, which can be applied to your paper Understand how pain reviewers are when reading a paper; so that you make a good paper that ease readers :) Keep up with the recent literature
  14. Why do I need to know or involve in academic

    peer reviews? Know what are the common concerns of reviewers, which can be applied to your paper Understand how pain reviewers are when reading a paper; so that you make a good paper that ease readers :) Keep up with the recent literature Have a chance to read top-notch papers
  15. I’m too young. I can’t be a good reviewer

  16. I’m too young. I can’t be a good reviewer

  17. I’m too young. I can’t be a good reviewer ECR

    (<5 years after graduation) Being a PC member or a reviewer since 2017 11 full-length peer-review publications
  18. I’m too young. I can’t be a good reviewer ECR

    (<5 years after graduation) Being a PC member or a reviewer since 2017 11 full-length peer-review publications Distinguished Reviewer Board of TOSEM Distinguished Reviewers/PC members of -ICSME 2017 -ICSE 2020 -ASE 2020 -FSE 2020
  19. I’m too young. I can’t be a good reviewer ECR

    (<5 years after graduation) Being a PC member or a reviewer since 2017 11 full-length peer-review publications Distinguished Reviewer Board of TOSEM Distinguished Reviewers/PC members of -ICSME 2017 -ICSE 2020 -ASE 2020 -FSE 2020 First time as a PC member
  20. I’m too young. I can’t be a good reviewer ECR

    (<5 years after graduation) Being a PC member or a reviewer since 2017 11 full-length peer-review publications Distinguished Reviewer Board of TOSEM Distinguished Reviewers/PC members of -ICSME 2017 -ICSE 2020 -ASE 2020 -FSE 2020 First time as a PC member Young PC member 
 != Bad Reviewer But you should have some research experience :)
  21. Peer-Review Process in SE Author(s) submit manuscript

  22. Peer-Review Process in SE Author(s) submit manuscript PC Chairs pre-

    check manuscripts
  23. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC

    Chairs pre- check manuscripts
  24. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC

    Chairs pre- check manuscripts PC members bid (select) the papers they want to review
  25. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC

    Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts
  26. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC

    Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts Author(s) write a response to reviewers comments
  27. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC

    Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts PC members discuss Author(s) write a response to reviewers comments
  28. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected Accepted

    PC Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts PC members discuss Author(s) write a response to reviewers comments
  29. Peer-Review Process in SE Author(s) submit manuscript Desk Rejected Accepted

    Rejected PC Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts PC members discuss Author(s) write a response to reviewers comments
  30. Review Criteria Soundness: The extent to which the paper’s contributions

    are supported by rigorous application of appropriate research methods Significance: The extent to which the paper’s contributions are important with respect to open software engineering challenges Novelty: The extent to which the contribution is sufficiently original and is clearly explained with respect to the state-of-the-art Verifiability: The extent to which the paper includes sufficient information to support independent verification or replication of the paper’s claimed contributions Presentation: The extent to which the paper’s quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc
  31. Review Criteria: Soundness Soundness: The extent to which the paper’s

    contributions are supported by rigorous application of appropriate research methods
  32. Review Criteria: Soundness Soundness: The extent to which the paper’s

    contributions are supported by rigorous application of appropriate research methods Are the study design, methods, evaluation rigorous and sounded?
  33. Review Criteria: Soundness Soundness: The extent to which the paper’s

    contributions are supported by rigorous application of appropriate research methods Are the study design, methods, evaluation rigorous and sounded? Is there any thing that could potentially lead to faulty results/outcomes? Is there any thing that could potentially change the empirical findings? Is the choice of the design is clearly justified (and convincing)?
  34. Review Criteria: Soundness Soundness: The extent to which the paper’s

    contributions are supported by rigorous application of appropriate research methods Are the study design, methods, evaluation rigorous and sounded? Is there any thing that could potentially lead to faulty results/outcomes? Is there any thing that could potentially change the empirical findings? Is the choice of the design is clearly justified (and convincing)? Sections: Case Study Design, Experiment, Evaluation, Results
  35. Review Criteria: Significance Significance: The extent to which the paper’s

    contributions are important with respect to open software engineering challenges
  36. Review Criteria: Significance Significance: The extent to which the paper’s

    contributions are important with respect to open software engineering challenges How this work helps SE 
 (either SE practitioners or research community)?
  37. Review Criteria: Significance Significance: The extent to which the paper’s

    contributions are important with respect to open software engineering challenges How this work helps SE 
 (either SE practitioners or research community)? Does it (a tool or finding) intuitively help SE practitioners? Will it be useful for future research? Does it address an important problem in SE?
  38. Review Criteria: Significance Significance: The extent to which the paper’s

    contributions are important with respect to open software engineering challenges How this work helps SE 
 (either SE practitioners or research community)? Does it (a tool or finding) intuitively help SE practitioners? Will it be useful for future research? Does it address an important problem in SE? Sections: Introduction, Background, Discussion, Conclusion
  39. Review Criteria: Novelty Novelty: The extent to which the contribution

    is sufficiently original and is clearly explained with respect to the state-of-the-art
  40. Review Criteria: Novelty Novelty: The extent to which the contribution

    is sufficiently original and is clearly explained with respect to the state-of-the-art How this work is different from the state-of-the-art?
  41. Review Criteria: Novelty Novelty: The extent to which the contribution

    is sufficiently original and is clearly explained with respect to the state-of-the-art How this work is different from the state-of-the-art? Do the work address a significant limitation of prior work? What is the gap between this work and prior work?
  42. Review Criteria: Novelty Novelty: The extent to which the contribution

    is sufficiently original and is clearly explained with respect to the state-of-the-art How this work is different from the state-of-the-art? Do the work address a significant limitation of prior work? What is the gap between this work and prior work? Sections: Introduction, Related Work, Discussion
  43. Review Criteria: Verifiability Verifiability: The extent to which the paper

    includes sufficient information to support independent verification or replication of the paper’s claimed contributions
  44. Review Criteria: Verifiability Verifiability: The extent to which the paper

    includes sufficient information to support independent verification or replication of the paper’s claimed contributions Can I replicate this work (if I want to)?
  45. Review Criteria: Verifiability Verifiability: The extent to which the paper

    includes sufficient information to support independent verification or replication of the paper’s claimed contributions Can I replicate this work (if I want to)? Is the detail of method/design sufficient and clear? Is there any key step that lacks the details?
  46. Review Criteria: Verifiability Verifiability: The extent to which the paper

    includes sufficient information to support independent verification or replication of the paper’s claimed contributions Can I replicate this work (if I want to)? Is the detail of method/design sufficient and clear? Is there any key step that lacks the details? Sections: Case Study Design, Experiment, Evaluation, Results
  47. Review Criteria: Presentation Presentation: The extent to which the paper’s

    quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc
  48. Review Criteria: Presentation Presentation: The extent to which the paper’s

    quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc Is the paper well-written, well-structured
 and easy to follow?
  49. Review Criteria: Presentation Presentation: The extent to which the paper’s

    quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc Is the paper well-written, well-structured
 and easy to follow? Can I grasp the key message in each paragraph/sections? Is the content is organised in a logical order? Is the complex part accompanied with a visual representation?
  50. Review Criteria: Presentation Presentation: The extent to which the paper’s

    quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc Is the paper well-written, well-structured
 and easy to follow? Can I grasp the key message in each paragraph/sections? Is the content is organised in a logical order? Is the complex part accompanied with a visual representation? Sections: Whole paper :)
  51. One Mistake Can Send You Home Your manuscript You Reviewers

  52. One Mistake Can Send You Home **This is just one

    possible consequence that I observed
  53. Presentation One Mistake Can Send You Home **This is just

    one possible consequence that I observed
  54. Presentation Verifiability Soundness One Mistake Can Send You Home **This

    is just one possible consequence that I observed
  55. Presentation Verifiability Soundness Novelty Significance One Mistake Can Send You

    Home **This is just one possible consequence that I observed
  56. How can I write a good paper?

  57. How can I write a good paper? Writing a (good)

    review can help you write a good paper
  58. How can I write a good paper? Writing a (good)

    review can help you write a good paper How can I write a good review?
  59. How can I write a good paper? Writing a (good)

    review can help you write a good paper How can I write a good review? Learn from the reviewers of your papers
  60. How can I write a good paper? Writing a (good)

    review can help you write a good paper How can I write a good review? Learn from the reviewers of your papers If you like the reviews, just follow that style of reviews
  61. How can I write a good paper? Writing a (good)

    review can help you write a good paper How can I write a good review? Learn from the reviewers of your papers If you like the reviews, just follow that style of reviews If you don’t like reviews, just do better than that (you know what you want)
  62. How can I write a good paper? Writing a (good)

    review can help you write a good paper How can I write a good review? Learn from the reviewers of your papers If you like the reviews, just follow that style of reviews If you don’t like reviews, just do better than that (you know what you want) In my Reviews, I’m trying to Clearly explain “why” it is a concern Suggest how should the authors do Detangle between presentation and other aspects (is it poor because of writing or something else?)
  63. Some Anti-Patterns of Reviews Andreas Zeller’s April Fool’s Day Project

    Shallow Reviews: Reviews take only a very shallow, "syntactical" look into a paper without considering its potential and deeper implications. Cheap shots: A very generic critique to a paper in a manner that is usually not well-matched to the specific context Premkumar Devanbu’s slides
  64. Where can I learn more about writing a good review?

  65. Where can I learn more about writing a good review?

    Be a sub-reviewer of your supervisor
  66. Where can I learn more about writing a good review?

    Be a sub-reviewer of your supervisor Check “Birds of a feather: Reviewing SE research papers” at ICSE2020
  67. Where can I learn more about writing a good review?

    Be a sub-reviewer of your supervisor Check “Birds of a feather: Reviewing SE research papers” at ICSE2020 https://2021.msrconf.org/track/msr-2021-shadow-pc Gain first-hand experience as Shadow PC at MSR2021!
  68. patanamon.t@unimelb.edu.au @patanamon http://patanamon.com Thank You & Happy Reviewing!