Rethinking the Suitability of RL Algorithms Under Practical Transfer Constraints

Hany Hamed, Abhishek Naik, Colin Bellinger, A. Rupam Mahmood; 7, 2026.

Presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.

Abstract

Transfer-oriented reinforcement learning requires evaluating algorithms along dimensions that go beyond standard sample efficiency. We focus on two dimensions: \emph{practical efficiency}, which asks whether conclusions about algorithm suitability change under wall-clock rather than interaction-based budgets, and \emph{robustness under dynamics mismatch}, which asks how different learning paradigms respond to variability in the training distribution induced by domain randomization. We provide two insights to reinforcement-learning practitioners. First, comparing the sample efficiency of different algorithms is often an insufficient criteria in transfer-oriented settings. The wall-clock time required to train a decent policy is an important consideration for practitioners, and we find that the sample-inefficient PPO algorithm can result in a performant policy faster than the relatively more sample efficient algorithms like SAC and TD-MPC2—validating the common knowledge about massively parallel training paradigms. Second, domain randomization can help different kinds of algorithms learn robust policies. In particular, despite PPO, SAC, and TD-MPC2 representing different RL paradigms—on-policy, off-policy, and model-based learning and planning—we found that domain randomization affects all three algorithms in a similar way. To the best of our knowledge, this is the first controlled comparison of the effect of domain-randomization coverage on PPO, SAC, and TD-MPC2 under the same transfer protocol. Taken together, these two insights highlight the importance of evaluating RL algorithms not only by sample efficiency, but also by practical considerations such as training time and the algorithms’ ability to produce usable policies.

[abs][pdf]

BibTeX

@article{hamed2026rethinking,
    title={Rethinking the Suitability of RL Algorithms Under Practical Transfer Constraints},
    author={Hany Hamed and Abhishek Naik and Colin Bellinger and A. Rupam Mahmood},
    journal={Reinforcement Learning Journal},
    volume={7},
    pages={},
    year={2026}
}