An Unreasonably Simple Approach to Safe RL

Geraud Nangue Tasse, Mark Nemecek, Tamlin Love, Steven James, Benjamin Rosman; 7, 2026.

Presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.

Abstract

An important problem in reinforcement learning is designing agents that learn to solve tasks safely in an environment. A common solution is to define either a penalty in the reward function or a cost to be minimised when reaching unsafe states. However, designing reward or cost functions to incentivise safe behaviours with maximal probability is non-trivial and can increase with the complexity of the problem. To address this, we propose a simple algorithm for agents to automatically learn the penalty for unsafe transitions while learning to solve the given task. Precisely, the agent keeps track of its lowest and highest estimated state-values, then simply uses their difference as the penalty for unsafe transitions. Importantly, our proposed algorithm can be used with any base reinforcement learning algorithm since it only modifies the rewards perceived by an agent, and introduces no new hyperparameters that would require tuning. Our experiments demonstrate the effectiveness of this approach in enabling agents to learn safe policies in high-dimensional continuous control environments.

[abs][pdf]

BibTeX

@article{tasse2026an,
    title={An Unreasonably Simple Approach to Safe RL},
    author={Geraud Nangue Tasse and Mark Nemecek and Tamlin Love and Steven James and Benjamin Rosman},
    journal={Reinforcement Learning Journal},
    volume={7},
    pages={},
    year={2026}
}