Create tasks Write code Build & test code Integrate Release/ Deploy A General View of Continuous Integration Code Review A bug is here… Quality-Impacting Practices in Software Development and Management
Create tasks Write code Build & test code Integrate Release/ Deploy A General View of Continuous Integration Code Review Code Review A bug is here… Quality-Impacting Practices in Software Development and Management
Create tasks Write code Build & test code Integrate Release/ Deploy A General View of Continuous Integration Code Review Quality-Impacting Practices in Software Development and Management What are the best practices with a minimum effort? Do socio-technical factors influence the practices? Can historical data tell some mistakes in the past?
Why do I need to know or involve in academic peer reviews? Know what are the common concerns of reviewers, which can be applied to your paper Understand how pain reviewers are when reading a paper; so that you make a good paper that ease readers :)
Why do I need to know or involve in academic peer reviews? Know what are the common concerns of reviewers, which can be applied to your paper Understand how pain reviewers are when reading a paper; so that you make a good paper that ease readers :) Keep up with the recent literature
Why do I need to know or involve in academic peer reviews? Know what are the common concerns of reviewers, which can be applied to your paper Understand how pain reviewers are when reading a paper; so that you make a good paper that ease readers :) Keep up with the recent literature Have a chance to read top-notch papers
Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC Chairs pre- check manuscripts PC members bid (select) the papers they want to review
Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts
Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts Author(s) write a response to reviewers comments
Peer-Review Process in SE Author(s) submit manuscript Desk Rejected PC Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts PC members discuss Author(s) write a response to reviewers comments
Peer-Review Process in SE Author(s) submit manuscript Desk Rejected Accepted PC Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts PC members discuss Author(s) write a response to reviewers comments
Peer-Review Process in SE Author(s) submit manuscript Desk Rejected Accepted Rejected PC Chairs pre- check manuscripts PC members bid (select) the papers they want to review Papers assigned to PC members; review starts PC members discuss Author(s) write a response to reviewers comments
Review Criteria Soundness: The extent to which the paper’s contributions are supported by rigorous application of appropriate research methods
Significance: The extent to which the paper’s contributions are important with respect to open software engineering challenges
Novelty: The extent to which the contribution is sufficiently original and is clearly explained with respect to the state-of-the-art
Verifiability: The extent to which the paper includes sufficient information to support independent verification or replication of the paper’s claimed contributions
Presentation: The extent to which the paper’s quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc
Review Criteria: Soundness Soundness: The extent to which the paper’s contributions are supported by rigorous application of appropriate research methods
Review Criteria: Soundness Soundness: The extent to which the paper’s contributions are supported by rigorous application of appropriate research methods Are the study design, methods, evaluation rigorous and sounded?
Review Criteria: Soundness Soundness: The extent to which the paper’s contributions are supported by rigorous application of appropriate research methods Are the study design, methods, evaluation rigorous and sounded? Is there any thing that could potentially lead to faulty results/outcomes? Is there any thing that could potentially change the empirical findings? Is the choice of the design is clearly justified (and convincing)?
Review Criteria: Soundness Soundness: The extent to which the paper’s contributions are supported by rigorous application of appropriate research methods Are the study design, methods, evaluation rigorous and sounded? Is there any thing that could potentially lead to faulty results/outcomes? Is there any thing that could potentially change the empirical findings? Is the choice of the design is clearly justified (and convincing)? Sections: Case Study Design, Experiment, Evaluation, Results
Review Criteria: Significance Significance: The extent to which the paper’s contributions are important with respect to open software engineering challenges
Review Criteria: Significance Significance: The extent to which the paper’s contributions are important with respect to open software engineering challenges How this work helps SE (either SE practitioners or research community)?
Review Criteria: Significance Significance: The extent to which the paper’s contributions are important with respect to open software engineering challenges How this work helps SE (either SE practitioners or research community)? Does it (a tool or finding) intuitively help SE practitioners? Will it be useful for future research? Does it address an important problem in SE?
Review Criteria: Significance Significance: The extent to which the paper’s contributions are important with respect to open software engineering challenges How this work helps SE (either SE practitioners or research community)? Does it (a tool or finding) intuitively help SE practitioners? Will it be useful for future research? Does it address an important problem in SE? Sections: Introduction, Background, Discussion, Conclusion
Review Criteria: Novelty Novelty: The extent to which the contribution is sufficiently original and is clearly explained with respect to the state-of-the-art
Review Criteria: Novelty Novelty: The extent to which the contribution is sufficiently original and is clearly explained with respect to the state-of-the-art How this work is different from the state-of-the-art?
Review Criteria: Novelty Novelty: The extent to which the contribution is sufficiently original and is clearly explained with respect to the state-of-the-art How this work is different from the state-of-the-art? Do the work address a significant limitation of prior work? What is the gap between this work and prior work?
Review Criteria: Novelty Novelty: The extent to which the contribution is sufficiently original and is clearly explained with respect to the state-of-the-art How this work is different from the state-of-the-art? Do the work address a significant limitation of prior work? What is the gap between this work and prior work? Sections: Introduction, Related Work, Discussion
Review Criteria: Verifiability Verifiability: The extent to which the paper includes sufficient information to support independent verification or replication of the paper’s claimed contributions
Review Criteria: Verifiability Verifiability: The extent to which the paper includes sufficient information to support independent verification or replication of the paper’s claimed contributions Can I replicate this work (if I want to)?
Review Criteria: Verifiability Verifiability: The extent to which the paper includes sufficient information to support independent verification or replication of the paper’s claimed contributions Can I replicate this work (if I want to)? Is the detail of method/design sufficient and clear? Is there any key step that lacks the details?
Review Criteria: Verifiability Verifiability: The extent to which the paper includes sufficient information to support independent verification or replication of the paper’s claimed contributions Can I replicate this work (if I want to)? Is the detail of method/design sufficient and clear? Is there any key step that lacks the details? Sections: Case Study Design, Experiment, Evaluation, Results
Review Criteria: Presentation Presentation: The extent to which the paper’s quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc
Review Criteria: Presentation Presentation: The extent to which the paper’s quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc Is the paper well-written, well-structured and easy to follow?
Review Criteria: Presentation Presentation: The extent to which the paper’s quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc Is the paper well-written, well-structured and easy to follow? Can I grasp the key message in each paragraph/sections? Is the content is organised in a logical order? Is the complex part accompanied with a visual representation?
Review Criteria: Presentation Presentation: The extent to which the paper’s quality of writing meets the high standards, including clear descriptions and explanations, absence of major ambiguity, etc Is the paper well-written, well-structured and easy to follow? Can I grasp the key message in each paragraph/sections? Is the content is organised in a logical order? Is the complex part accompanied with a visual representation? Sections: Whole paper :)
How can I write a good paper? Writing a (good) review can help you write a good paper How can I write a good review? Learn from the reviewers of your papers
How can I write a good paper? Writing a (good) review can help you write a good paper How can I write a good review? Learn from the reviewers of your papers If you like the reviews, just follow that style of reviews
How can I write a good paper? Writing a (good) review can help you write a good paper How can I write a good review? Learn from the reviewers of your papers If you like the reviews, just follow that style of reviews If you don’t like reviews, just do better than that (you know what you want)
How can I write a good paper? Writing a (good) review can help you write a good paper How can I write a good review? Learn from the reviewers of your papers If you like the reviews, just follow that style of reviews If you don’t like reviews, just do better than that (you know what you want) In my Reviews, I’m trying to Clearly explain “why” it is a concern Suggest how should the authors do Detangle between presentation and other aspects (is it poor because of writing or something else?)
Some Anti-Patterns of Reviews Andreas Zeller’s April Fool’s Day Project Shallow Reviews: Reviews take only a very shallow, "syntactical" look into a paper without considering its potential and deeper implications. Cheap shots: A very generic critique to a paper in a manner that is usually not well-matched to the specific context Premkumar Devanbu’s slides
Where can I learn more about writing a good review? Be a sub-reviewer of your supervisor Check “Birds of a feather: Reviewing SE research papers” at ICSE2020
Where can I learn more about writing a good review? Be a sub-reviewer of your supervisor Check “Birds of a feather: Reviewing SE research papers” at ICSE2020 https://2021.msrconf.org/track/msr-2021-shadow-pc Gain first-hand experience as Shadow PC at MSR2021!
patanamon
brianwarren
destraynor
lauravandoore
jasonvnalue
mthomps
zakiwarfel
jacobian
shpigford
danielanewman
robhawkes
zenorocha
phodgson