CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning
; 7, 2026.
Presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.
Abstract
Offline multi-agent reinforcement learning (MARL) enables policy learning from fixed datasets, but is prone to $\textbf{coordination failure}$: agents trained on static, off-policy data converge to suboptimal joint behaviours because they cannot co-adapt as their policies change. We introduce $\textbf{CODA}$ ($\textbf{C}$oordination via $\textbf{O}$n-Policy $\textbf{D}$iffusion for Multi-$\textbf{A}$gent Reinforcement Learning}), a diffusion-based multi-agent trajectory generator for data augmentation that samples conditioned on the current joint policy, producing synthetic experience which better reflects the evolving behaviours of the agents, thereby providing a mechanism for co-adaptation. We find that previous diffusion-based augmentation approaches are insufficient for fostering multi-agent coordination because they produce static augmented datasets that do not evolve as the current joint policy changes during training; CODA tackles this by more closely simulating on-policy learning and is a meaningful step toward coordinated behaviours in the offline setting. CODA is algorithm-agnostic and can be layered onto both model-free and model-based offline reinforcement learning pipelines as an augmentation module. Empirically, CODA resolves canonical coordination pathologies in continuous polynomial games.
[abs][pdf]
BibTeX
@article{hedman2026coda,
title={CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning},
author={Marcel Hedman and Kale-ab Tessera and Juan Claude Formanek and Anya Sims and Riccardo Zamboni and Trevor McInroe and John Torr and Elliot Fosong},
journal={Reinforcement Learning Journal},
volume={7},
pages={},
year={2026}
}