Representation Regularization in Distributional Reinforcement Learning

André Inge, Jonas Nordqvist, Björn Lindenberg, Karl-Olof Lindahl; 7, 2026.

Presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.

Abstract

This paper proposes a regularization method for \emph{categorical distributional reinforcement learning} by generalizing the standard projection step to redistribute probability mass across multiple neighboring support locations using higher-order cardinal B-splines. The resulting operator induces a parameterized regularization of the categorical return representation. We establish convergence guarantees for the corresponding projected distributional Bellman operator in the policy evaluation setting. The proposed approach yields a more dispersed representation of the return distribution. This can be interpreted as an uncertainty-aware modeling choice or as an explicit regularization mechanism that may improve training performance. We demonstrate these effects empirically in both evaluation and control settings

[abs][pdf]

BibTeX

@article{inge2026representation,
    title={Representation Regularization in Distributional Reinforcement Learning},
    author={André Inge and Jonas Nordqvist and Björn Lindenberg and Karl-Olof Lindahl},
    journal={Reinforcement Learning Journal},
    volume={7},
    pages={},
    year={2026}
}