Senior Reviewer Instructions

Senior reviewers are responsible for writing the type of review authors traditionally expect when submitting a manuscript. They should comment on how the work is situated in the literature, its relevance, and the validity of its claims (that is, its correctness). Note that one of the goals of RLJ's review process is to avoid tastemaking and to discourage reviews that focus on the paper the reviewer wishes had been submitted rather than the paper that was actually submitted. The instructions below are intended to clarify these expectations and to promote reviews that focus on the manuscript as submitted. For the wider context, see the RLJ Reviewers Guide.

The overall advice to senior reviewers is to focus on answering three questions:

1. Does the work provide evidence supporting the main contributions proposed by the authors—are claims matched by evidence? This includes technical correctness.
2. Is there an audience for the work?
3. What knowledge gap did the paper address?

The cover page in every submission should be seen almost as a rubric for such an evaluation: Is every claimed contribution properly contextualized, and does the work provide sufficient evidence to support each one of them?

Recommended Process

1. Read the cover page, identifying the claimed contributions and assessing whether they are accurately and appropriately situated within the literature.
2. Read the manuscript, with a focus on:
   • Identifying the evidence provided for each claimed contribution and evaluating whether it sufficiently supports the claim. This includes determining if results, whether theoretical or empirical, are correct and scientifically rigorous.
   • Identifying additional claims made in the manuscript that are not listed on the cover page and evaluating whether they are adequately supported. Major contributions should not be claimed in the main text if they do not appear on the cover page.
   • Identifying technical errors, unsupported statements, or points of imprecision.
   • Assessing how the manuscript positions itself relative to prior work.
3. Consider whether:
   • The results are likely to be of interest to a subset of RLJ readers (or RLC attendees) beyond the authors themselves; that is, there should be at least five people (other than the authors) who read RLJ (or attend RLC) and who would be interested in reading the paper.
   • The presentation is professional, clear, well-organized, and written with a reinforcement learning audience in mind.
4. Reflect on your overall assessment of the work and how you would answer the review questions below.
5. Draft your responses to the questions below and enter them in OpenReview.

Additional Considerations

• If the paper contains both theoretical and empirical contributions, assess the relationship between them. Does the theory provide insight into, justify, or analyze the proposed algorithm? Does it apply to the algorithm as implemented and evaluated experimentally? If there is a disconnect, is it clearly acknowledged and explained?
• Evaluate claims in proportion to the evidence provided. Broad claims, such as state-of-the-art performance across a class of environments, require correspondingly broad and convincing empirical support.
• Pay close attention to the references. The reference list is part of the manuscript and should be reviewed accordingly. Verify that all cited papers exist, that the listed authors are correct, and that publication venues are accurately reported. This is particularly important when generative AI tools are used, which can introduce citation errors and fabricated references.

Organize Your Review as Follows

Unlike technical reviewers, senior reviewers are not asked to complete a series of structured questions. Instead, they are provided with two text boxes: one to explicitly discuss the paper's claimed contributions, and another to provide a traditional review.

Claimed contributions box

Reviewers are expected to discuss where the evidence to support each claimed contribution is in the paper, and whether the provided evidence is sufficient to match the claims. The technical reviewer instructions explicitly require them to answer the questions below. We encourage senior reviewers to address the same points for each contribution. Editors are expected to check every review for this.

• State the claimed contribution in your own words.
• State what evidence in the paper was provided (or intended) as support for the claim.
• State whether the evidence was sufficient to establish the claim. This should be a short statement of yes / no / partially.
• Explain why the provided evidence was or was not sufficient to establish the claim. See Supporting Claims for a discussion of what does and does not constitute sufficient support for claims.

General review

Although the claimed contributions should be used as a rubric that will often decide on the acceptance or rejection of the paper, a comprehensive review will also talk about the manuscript's presentation, scholarship, and more, including a statement on whether the manuscript would be of interest to five or more readers of RLJ. Comment on the polish and presentation of the paper—see Polished and Well Written—or list any other errors you found or comments you have.

Inspired by Sutton's guide for writing good reviews, we recommend that reviewers structure their reviews as follows:

• Summarize what the paper claims to do/contribute and what knowledge gap it addresses. Be positive and generous (assuming the paper achieves what it sets out to do).
• Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.
• Provide supporting arguments for the reasons for the decision.
• Provide additional feedback to improve the paper. Make it clear that these points are here to help, not necessarily part of your decision-making.

What Not to Do

• Do not suggest that a different set of claims would have been more interesting, impactful, or worthy of publication. If you believe a claimed contribution is not supported by the evidence provided, it is appropriate to explain what claim you believe the evidence does support. However, this should be done to clarify the scope of the conclusions that can be drawn from the paper, not to redefine the paper around a contribution you would have preferred to see.
• Do not suggest that the authors should have compared to other work published within the last six months.
• Do not request additional results unless those are required to support the contributions already claimed.