RLJ Editors Guide
Editors are responsible for: (1) deciding whether to recommend desk-rejection of a submission, (2) assigning (or recruiting) reviewers, (3) skimming the paper themselves, (4) writing a meta-review, (5) recommending acceptance, revisions, or rejection, and (6) in the case of revisions, handling the resubmission if it arrives within three months. For the wider context, see the RLJ Reviewers Guide.
While the decision should be heavily informed by the reviews, note that Editors are expected to skim submitted papers to inform their decisions. The depth of this skim (or read) may scale based on the reviews. For example, if the reviews overwhelmingly recommend rejection, the Editor may skim to confirm these reasons for rejection. Thorough reads are primarily necessary for borderline cases.
Sequence of Steps
When assigned a paper, editors should perform the following sequence of steps.
1. Desk rejection assessment
Perform a quick read of the paper to assess whether it should be desk-rejected. Criteria for desk rejection (and a description of the expected desk rejection text) can be found in Desk Rejection Criteria and Process.
2. Assign and recruit reviewers
Assign a Senior Reviewer and a Technical Reviewer in OpenReview. If you know of particularly suitable reviewers for the paper who are not already in the system, you may request that they be added to the reviewer pool. You may also ask them to review just this one paper, without joining the RLJ reviewer pool. You should complete reviewer assignment within 2 weeks of being assigned the paper. This step is critical - identifying or recruiting reliable reviewers whom you trust will make your subsequent job much easier.
3. Obtain reviews
Wait for the reviewers to complete their reviews, within 3 weeks. You are encouraged to send reminder emails (including outside OpenReview) as the deadline approaches to ensure reviews arrive on time. If reviewers are irresponsible, report them and replace them with new reviewers. These reports are critical to the functioning of the review process, and the Editors-in-Chief are committed to acting on your reports.
Ensure that you are familiar with the Senior Reviewer and Technical Reviewer instructions. If reviewers fail to follow the instructions, work with them to revise their reviews following the instructions.
4. Meta-review
Within 1 week of obtaining the reviews (or you may plan for this week to be 3 weeks after assigning reviewers, even if the reviews arrive far earlier), recommend a decision and complete the meta-review. This involves skimming or reading the paper yourself.
Updating Authors
If any stage of the process takes longer than the prescribed time (see the timeline), leave a comment for the authors in OpenReview to let them know about the delay and give them an updated timeline. Report the delay to the Editors-in-Chief.
Accept / Revisions / Reject Criteria
Review all of the other reference pages, particularly Supporting Claims, Desk Rejection Criteria and Process, and Polished and Well Written. These should be your primary resource for expectations regarding the acceptance criteria for RLJ.
Here we re-emphasize that RLJ aims to have the highest bar for scientific rigor. We fully support the rejection of papers that have errors in scientific methodology or reporting, and which therefore do not sufficiently support their claims. Examples include failure to report the number of trials or seeds in experiments, the omission of discussion of how hyperparameters were tuned, failure to report the meaning of error bar widths, and failing to state implicit assumptions in theorems.
Note that this applies even if these issues could be fixed with minor edits. That is, such errors are viewed as a significant issue that typically results in rejection, not revisions. Note that rejection comes with a 6-month wait to resubmit, while revision allows immediate resubmission. This deliberately strict policy is designed to encourage authors to carefully check the validity of their work, rather than relying on you and the review process to catch their mistakes.
We also re-emphasize that RLJ aims to avoid tastemaking. Acceptance should not be based on the perceived impact of the work, nor should it be based on whether one wishes the authors made different claims from the ones they chose (e.g., if the authors don't claim state-of-the-art performance, then a lack of state-of-the-art performance is not a reason for rejection). When incremental papers are rejected (by first discussing with an Editor-in-Chief for extreme cases), it should typically be because they over-claim in an attempt to appear more significant than they are. If authors are up-front and clear about the incremental nature of their work (and the limitations of the clearly claimed contributions), the work should still be accepted to RLJ, assuming there are no other issues.
Meta-Review Format
The meta-review need not be a full-length review (you can write one if you want). Note that your meta-review should primarily be phrased in terms of the claimed contributions on the cover page, whether their scope is appropriate, and whether they are properly contextualized. This is something more senior researchers (like you!) are much better positioned to do.
Revision Process
If the paper receives "Revisions", the authors will be encouraged to resubmit within 3 months. In such cases, ensure that the meta-review clearly states what issues the authors need to address for their work to be accepted.
If the authors do resubmit, their resubmission should come with a letter to you, the Editor, that discusses the primary concerns that you and the reviewers had with the initial submission, and how any changes to their submission address them. Note that authors may disagree with concerns that were raised, and this should be discussed in the letter even if the authors choose not to revise the manuscript.
You may then decide whether to accept or reject the paper immediately, whether to consult the previous reviewers, or whether to initiate an entirely new review of the paper. Barring extenuating circumstances, resubmitted papers should either be accepted or rejected—they should not be returned for further revisions.