Generalization in Monitored Markov Decision Processes (Mon-MDPs)

Montaser Mohammedalamen, Michael Bowling; 7, 2026.

Presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.

Abstract

Reinforcement learning (RL) typically models the interaction between the agent and environment as a Markov decision process (MDP), assuming fully observable rewards. In many real-world settings, this assumption fails, motivating the monitored Markov decision process (Mon-MDP), where rewards may be unobserved. Prior work on Mon-MDPs has been limited to simple, tabular cases, restricting their applicability to real-world problems. This work explores Mon-MDPs using function approximation and investigates the challenges. We show that combining function approximation with a learned reward model enables agents to generalize from monitored states with observable rewards to unmonitored states with unobservable rewards. Therefore, we demonstrate that such generalization with a reward model achieves near-optimal policies in environments formally defined as unsolvable. However, we also uncover a critical limitation: agents may incorrectly extrapolate rewards due to overgeneralization, which can lead to undesirable behaviors. To mitigate overgeneralization, we propose a cautious policy optimization method leveraging reward uncertainty. This work serves as a step towards bridging the gap between Mon-MDP theory and real-world applications.

[abs][pdf]

BibTeX

@article{mohammedalamen2026generalization,
    title={Generalization in Monitored Markov Decision Processes (Mon-MDPs)},
    author={Montaser Mohammedalamen and Michael Bowling},
    journal={Reinforcement Learning Journal},
    volume={7},
    pages={},
    year={2026}
}