Decentralized Asymmetric DQN: Decentralization without Factorization in Multi-Agent Reinforcement Learning

Rupali Bhati, Anurag Kadkol, Andrea Baisero, Christopher Amato; 7, 2026.

Presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.

Abstract

Centralized training with decentralized execution (CTDE) is a dominant paradigm for cooperative multi-agent reinforcement learning. Most successful value-based CTDE methods rely on value factorization, decomposing a joint value function into per-agent utilities under structural constraints such as additivity or monotonicity. While effective, these utilities are architectural components of a mixing network and lack a principled interpretation as standalone value functions. Moreover, existing approaches provide limited insight into the convergence properties of decentralized learning dynamics. We propose Decentralized Asymmetric Deep Q-Networks (Dec-ADQN), which decouples decentralization from structural value factorization. During training, a centralized, stateful action-value function is learned and used as a regression target for independently trained per-agent Q-functions that are executed in a decentralized manner. We show that these per-agent Q-functions approximate the conditional expectation of the centralized evaluator given each agent’s local information. Under exact optimization in a finite Dec-POMDP, we prove that Dec-ADQN implements a form of decentralized asymmetric policy iteration and converges to person-by-person optimal policies. Empirically, Dec-ADQN performs well across both independently decomposable tasks and tasks requiring tight coordination in the OvercookedV2 and SMAX benchmarks. These results suggest that projection-based decentralized learning offers an interpretable and theoretically grounded alternative to structural value factorization in cooperative MARL.

[abs][pdf]

BibTeX

@article{bhati2026decentralized,
    title={Decentralized Asymmetric DQN: Decentralization without Factorization in Multi-Agent Reinforcement Learning},
    author={Rupali Bhati and Anurag Kadkol and Andrea Baisero and Christopher Amato},
    journal={Reinforcement Learning Journal},
    volume={7},
    pages={},
    year={2026}
}