ICPL: Few-shot In-context Preference Learning via LLMs

Chao Yu, Qixin Tan, Hong Lu, Jiaxuan Gao, Xinting Yang, Yu Wang, Yi Wu, Eugene Vinitsky; 7, 2026.

Presented at the Reinforcement Learning Conference (RLC), Montréal, Quebec, Canada, August 15–17, 2026.

Abstract

Preference-based reinforcement learning is an effective way to handle tasks where rewards are hard to specify but can be exceedingly inefficient as preference learning is often tabula rasa. To address this challenge, we propose In-Context Preference Learning (ICPL), which uses in-context learning capabilities of LLMs to reduce human query inefficiency. ICPL uses the task description and basic environment code to create sets of reward functions which are iteratively refined by placing human feedback over videos of the resultant policies into the context of an LLM and then requesting better rewards. We first demonstrate ICPL’s effectiveness through a synthetic preference study, providing quantitative evidence that it significantly outperforms baseline preference-based methods with much higher performance and orders of magnitude greater efficiency. We observe that these improvements are not solely coming from LLM grounding in the task but also the quality of the rewards improvement via preference feedback. Additionally, we perform a series of real human preference-learning trials and observe that ICPL extends beyond synthetic settings and can work effectively with humans-in-the-loop.

[abs][pdf]

BibTeX

@article{yu2026icpl,
    title={ICPL: Few-shot In-context Preference Learning via LLMs},
    author={Chao Yu and Qixin Tan and Hong Lu and Jiaxuan Gao and Xinting Yang and Yu Wang and Yi Wu and Eugene Vinitsky},
    journal={Reinforcement Learning Journal},
    volume={7},
    pages={},
    year={2026}
}