Technical Reviewer Instructions

A technical review is more or less a list of technical errors in the paper. There is no commentary on novelty, significance, or impact. This is not an oversight on our part; it is very intentional. For the wider context of these instructions, see the RLJ Reviewers Guide.

Recommended Process

1. Read the cover page, identifying the claimed contributions.
2. Read the paper, with a focus on:
   • Identifying the support for each claimed contribution, and evaluating whether this support is sufficient to establish the claim.
   • Identifying additional claims made by the paper that are not listed on the cover page, and evaluating whether this support is sufficient to establish the claim. Major contributions should not be claimed in the main text if they do not appear on the cover page.
   • Identifying other technical errors or points of imprecision.
3. Draft your response to the questions below, and enter them in OpenReview.

Questions You Will Be Asked

Below are the questions you will be asked in OpenReview—there will be a textbox for each sub-bullet here.

1. For each claimed contribution:
   • State the claim in your own words.
   • State what evidence in the paper was provided (or intended) as support for the claim.
   • State whether the evidence was sufficient to establish the claim. This should be a short statement of yes / no / partially. The next question asks you to explain and support these brief statements.
   • Explain why the provided evidence was or was not sufficient to establish the claim. See Supporting Claims for a discussion of what does and does not constitute sufficient support for claims.
2. Comment on the polish and presentation of the paper. See Polished and Well Written for a discussion of what constitutes a polished and well-written submission.
3. Miscellaneous - list any other errors you found or comments you have.

The assessment also needs to be written so that it can be easily understood by three audiences: 1) the authors, 2) other reviewers who have read the paper, and 3) people up the review chain who have likely not read the paper. The key to establishing clear communication for all three audiences is to be specific (referencing parts of the paper) and to provide the context necessary so everyone will understand the claim, the assessment, and the reasons for it being sufficient or not.

What to Discuss in the Free-Form Textbox

• If the paper has any theoretical results, list the theorems and proofs you checked for correctness.
• List the main result(s) that back up each one of the main contributions on the cover page (a figure, a theorem, etc.).
• Are the assumptions in each proof clearly reflected in the theorem statement?
• Did the authors discuss whether the assumptions that proofs rely on are reasonable?

What Not to Do

• Do not comment on the significance of the chosen claims.
• Do not suggest or imply that authors should have made different claims. If the provided claim is not supported by results in the paper, it is okay to state what claim is actually supported as a way to provide feedback on the claimed contributions.
• Do not suggest that the authors should have compared to other work published within the last six months.
• Do not comment on whether you think RLJ is the appropriate venue for the paper, nor on whether you think there is an audience for the work at RLJ/RLC. This is the Editor's responsibility.