As a researcher, either you are pursuing a PhD. or an experienced researcher, we need to the research community time to time. For example, your supervisor might assign you as a sub-reviewer to review paper(s) of your interests/expertise, or you might get invited more and more as you grow as an independent researcher in your field after your PhD.
Therefore, as a person in the research community, and since the review pushes the research community moving forward, it is our responsibility to provide quality, constructive and not offensive reviews. Despite the importance of providing high quality reviews, unfortunately, we normally are not trained on this task during the PhD or afterwards.
Recently, professor Noah Smith from University of Washington gave a podcast which provides some fantastic advice on how to review a conference paper for the research community. Below, I summarize the main take-home messages from this podcast, which might be helpful to reseachers who would like to write a high quality reviews.
Brief Summary of the Paper
The first section provides a brief summary of the paper for others, e.g., an area chair who is extremely busy. This sectionn should:
- say as it is, (what's new and contributions claimed by authors etc.)
- that is, not judgement for this section
- make it easy for others to get the idea of the paper quickly
Major Pros & Cons
Then we can move on to the pros & cons. Remember to give pros! of the paper. Even you do not like the paper, we should always analyze pros of a paper just as for finding cons of it.
This section can provide some other minor issues, such as grammer errors. Please note that these minor things are not enough to reject a paper though.- particularly, we need to encourage (early) researchers keep going
- there should be pros of a paper, e.g., the question itself is good and challenging
Even when we write cons of a paper, we still need to be constructive and positive.
- always give actionable details
- e.g., instead of the method is confusing, talk about what equations or details are confusing
This section is main part of the review and needs to take into account of many key questions, e.g., Professor Emery Berger (Computer Science at the University of Massachusetts Amherst) listed some of those key questions as follows:
- Is the paper well-motivated? What problem does it address, and is it an important problem?
- Does the paper significantly advance the state of the art or break new ground?
- What are the paper’s key insights?
- What are the paper’s key scientific and technical contributions?
- Does the paper credibly support its claimed contributions?
- What did you learn from the paper?
- Is the paper sufficiently clear that most venue attendees will be able to read and understand it?
- Does the paper clearly establish its context with respect to prior work? Does it discuss prior work accurately and completely? Are comparisons with previous work clear and explicit?
- Does the paper describe something that has actually been implemented? If so, has it been evaluated properly? Is it publicly available so that these results can be verified?
- What impact is this paper likely to have (on theory & practice)? Is the work of broad appeal and interest to the research community?
Minor Things/Comments/Corrections
About Reviewing Process
Prof. Noah Smith also discussed about his personal process of reviewing a paper, which is useful to me to adopt for future reviews.
- quick scan (introduction, figures, conclusions, reference list)
- where it is positioned in the literature?
- what kind of paper it is? (position paper? theoretical one? system one?)
- top-down reading with red pen
- come back later with my review notes
- any unclear things can be asked as a reviewer
- e.g., the explaination/notation/prove is confusing or did the preprocessing also applied to baselines etc.
- note that it's not criticizing, it is just want to encourage the paper make clear on the next version
To sum up, we should keep in mind that the reviews that we write should
- help the authors to improve the paper
- be what we expect to see in the next version of the paper
Did you have experience or any thought or advice which might be helpful for writing a high quality review? Leave a comment to share it with others:)
No comments:
Post a Comment