Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecification

Haoyang Hong, Zichen Wang, Quanquan Gu, Huazheng Wang; 7, 2026.

Presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.

Abstract

We study KL-regularized contextual bandits and episodic reinforcement learning (RL) under general function approximation with model misspecification. Existing guarantees rely on realizability and therefore do not extend to misspecified models, where classical regret bounds may fail. This work introduces KL misspecification formulations for contextual bandits and episodic RL and analyzes regression-based algorithms that via Gibbs policies updates. High-probability KL-regret guarantees with explicit misspecification terms are established, recovering the standard realizable KL-regularized setting as a special case.

[abs][pdf]

BibTeX

@article{hong2026online,
    title={Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecification},
    author={Haoyang Hong and Zichen Wang and Quanquan Gu and Huazheng Wang},
    journal={Reinforcement Learning Journal},
    volume={7},
    pages={},
    year={2026}
}