DART: Dual Adaptive Residual Tracking for Low-Bias Advantage Estimation and Credit Assignment in AI Agents
; 7, 2026.
Presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Abstract
Proximal Policy Optimization (PPO) is the primary workhorse for complex continuous control and Reinforcement Learning from Human Feedback (RLHF). However, the reliability of Generalized Advantage Estimation (GAE) has recently been called into question: critic-free methods like RLOO and GRPO have raised the question of whether value-induced bias is doing more harm than the variance reduction it provides. We show this is the wrong dichotomy. The bottleneck is the \textit{dissemination} bias of GAE, where value approximation error propagates into advantage estimates and misleads the policy gradient, a bias that can be corrected without discarding the critic. We propose \textbf{\textit{Dual Adaptive Residual Tracking (DART)}}, a versatile two-critic scheme that decouples the bias--variance trade-off. DART pairs a stable base critic learning from low-variance bootstrapped targets with a residual critic that tracks a high-fidelity Monte Carlo correction via a frozen-target mechanism. Theoretically, under linear function approximation assumptions, we prove that DART achieves a strictly smaller steady-state mean-squared error (MSE) upper bound than single-critic methods in high-variance regimes. Empirically, across all \texttt{dm\_control} continuous control environments, DART matches or outperforms PPO in aggregate, with clear advantages in sparse and long-horizon tasks. We further provide initial evidence of DART's effectiveness in LLM fine-tuning.
[abs][pdf]
BibTeX
@article{mohammadzadeh2026dart,
title={DART: Dual Adaptive Residual Tracking for Low-Bias Advantage Estimation and Credit Assignment in AI Agents},
author={Shahrad Mohammadzadeh and Amir-massoud Farahmand and Reihaneh Rabbany and Doina Precup},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}