How to write a high quality review?


As a researcher, either you are pursuing a PhD. or an experienced researcher, we need to review conference or journal papers for the research community time to time. For example, your supervisor might assign you as a sub-reviewer to review paper(s) of your interests/expertise, or you might get invited more and more as you grow as an independent researcher in your field after your PhD.

Therefore, as a researcher in your research community, and since the review process pushes the research community moving forward, it is our responsibility to provide quality, constructive and not offensive reviews. Despite the importance of providing high quality reviews, unfortunately, we normally are not trained on this task during the PhD or afterwards.

Recently, professor Noah Smith from University of Washington gave a podcast which provides some fantastic advice on how to review a conference paper for the research community. Below, I summarize the main take-home messages from this podcast, as well as advice from other resources which might be helpful to reseachers who would like to write a high quality reviews.

Content

A review has three important parts:
  1. A brief summary of the paper
  2. Major pros and cons
  3. Minor things/comments/corrections
Let's discuss each part in the following in detail.

Brief Summary of the Paper

The first section provides a brief summary of the paper for others, e.g., an area chair who is extremely busy. This sectionn should:
  • say as it is, (what's new and contributions claimed by authors etc.)
  • that is, not judgement for this section
  • make it easy for others to get the idea of the paper quickly

Major Pros & Cons

Then we can move on to the pros & cons. Remember to give pros! of the paper. Even you do not like the paper, we should always analyze pros of a paper just as for finding cons of it.
  • particularly, we need to encourage (early) researchers keep going
  • there should be pros of a paper, e.g., the question itself is good and challenging

Even when we write cons of a paper, we still need to be constructive and positive. 
  • always give actionable details
  • e.g., instead of the method is confusing, talk about what equations or details are confusing

This section is main part of the review and needs to take into account of many key questions. 

For example, we can consider some of the questions below which include the ones from Professor Emery Berger (Computer Science at the University of Massachusetts Amherst):

Problem
  • Is the paper well-motivated? 
  • What problem does it address, and is it an important problem?
Quality
  • Is the submission technically sound? 
  • Are claims well supported by theoretical analysis or experimental results? Does the paper credibly support its claimed contributions? 
  • Is this a complete piece of work or work in progress? 
  • Does the paper describe something that has actually been implemented? If so, has it been evaluated properly? Is it publicly available so that these results can be verified? Are the results in the paper able to be reproduced easily?
  • Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?
Clarity
  • Is the submission clearly written? Is the paper sufficiently clear that most venue attendees will be able to read and understand it? 
  • Is it well organized? (If not, please make constructive suggestions for improving its clarity.) 
  • Does it adequately inform the reader? (Note: a superbly written paper provides enough information for an expert reader to reproduce its results.)
  • What is the intuition behind certain choices?
Originality
  • What are the paper’s key insights? 
  • What are the paper’s key scientific and technical contributions? 
  • What did you learn from the paper? 
  • Does the paper significantly advance the state of the art or break new ground? 
  • Are the tasks or methods new? 
  • Is the work a novel combination of well-known techniques? 
  • Is it clear how this work differs from previous contributions? 
  • Is related work adequately cited? Does the paper clearly establish its context with respect to prior work? Does it discuss prior work accurately and completely? Are comparisons with previous work clear and explicit? 
  • Incremental approach could be challenged to accept, check what contribution is left after removing others.
Significance
  • Are the results important? 
  • Are others (researchers or practitioners) likely to use the ideas or build on them? 
  • Does the submission address a difficult task in a better way than previous work? 
  • Does it advance the state of the art in a demonstrable way? 
  • Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?
  • What impact is this paper likely to have (on theory & practice)? Is the work of broad appeal and interest to the research community?




Minor Things/Comments/Corrections

This section can provide some other minor issues, such as grammer errors. Please note that these minor things are not enough to reject a paper though.




About Reviewing Process

Prof. Noah Smith also discussed about his personal process of reviewing a paper, which is useful to me to adopt for future reviews.
  • quick scan (introduction, figures, conclusions, reference list)
    • where it is positioned in the literature?
    • what kind of paper it is? (position paper? theoretical one? system one?)
  • top-down reading with red pen
  • come back later with my review notes
  • any unclear things can be asked as a reviewer
    • e.g., the explaination/notation/prove is confusing or did the preprocessing also applied to baselines etc.
    • note that it's not criticizing, it is just want to encourage the paper make clear on the next version


Summary

To sum up, in this post we discussed the structure of review, and the review process and a couple of things can be considered or checked while reviewing a paper. 

Finally, we should keep in mind that the reviews that we write should
  • help the authors to improve the paper 
  • be what we expect to see in the next version of the paper

Did you have experience or any thought or advice which might be helpful for writing a high quality review? Leave a comment to share it with others:)


Related resources

Labels: , , , , , ,