Home Page

Papers

Submissions

Authors Guide

Reviewers Guide

Technical Reviewers

Senior Reviewers

Supporting Claims

Polished & Well Written

Editors Guide

Editorial Board

Search

Contact

Publication Agreement

RLJ Reviewers Guide

The RLJ peer review process prioritizes rigorous methodology over perceived importance, aiming to foster scholarly discussions on both well-established and emerging topics in RL.

RLJ uses a reviewing process with editors and two types of reviewers: senior reviewers (SRs) and technical reviewers (TRs). The purpose of this guide is to make each of these roles clear and to describe how to write each type of review. It is worth stressing that this is notably different from the typical NeurIPS/ICLR/ICML review process.

This guide is public. It covers the criteria a paper is judged against and how each kind of review is written; the mechanics of the process—timeline, decisions, resubmission—live on Author Guide so that they are stated in one place. Authors are strongly encouraged to read both before submitting.

At a high level, the two types of reviews differ as follows. A technical review is more or less a list of technical errors in the paper. There is no commentary on novelty, significance, or impact. This is not an oversight on our part; it is very intentional. A senior review is a typical review, but with a focus on claims matching evidence while still including comments on technical correctness.

Editors correspond to the previous Senior Area Chair (SAC) and Area Chair (AC) roles, and senior reviewers correspond to the previous Holistic Reviewers (HR). Editors will find their own per-stage duties and deadlines in the Editors Guide.

How Papers are Evaluated

Acceptance is based only on the submission, not on the potential of the paper with further edits. A paper that requires even minor changes may be desk rejected or returned for revisions.

The criteria for acceptance of a paper into RLJ are:

  1. The paper clearly describes its claimed contributions.
  2. The contributions are of interest to the RL community.
  3. The paper provides sufficient support for its claimed contributions.
  4. The paper does not contain other errors.
  5. The paper is polished and well written.

Below we describe each of these criteria in more detail.

Criterion 1: The paper clearly describes its claimed contributions

All major contributions should be listed on the cover page. Additional claims are unavoidable, but all major contributions should be listed on the cover page. A claim not appearing on the cover page still needs to be sufficiently supported. The claims themselves should be precise (not ambiguous or unclear).

Criterion 2: The contributions are of interest to the RL community

As a general rule, there should be at least five people (other than the authors) who read RLJ (or attend RLC) and who would be interested in reading the paper. This allows for papers to be published in RLJ containing research that one may not view as RL research, if the research is of interest to the RL community. However, papers must be written with the RL community as the intended audience.

Criterion 3: The paper provides sufficient support for its claimed contributions

See Supporting Claims for a description of what does and does not constitute "sufficient support".

Criterion 4: The paper does not contain other errors

Other errors that do not impact whether or not the paper provides sufficient support for its claims can preclude acceptance, but typically only motivate the need for revisions, not rejection.

Criterion 5: The paper is polished and well written

A submission that is not polished and well written can result in rejection or revisions, depending on the scale of the issues. See Polished and Well Written for a description of what does and does not constitute a polished and well-written paper.

Instructions for Each Role

Editors have their own Editors Guide.

Reference Material

Both reviewers and editors assess submissions against these two pages:

Additional Material

We encourage reviewers to read the following resources: