强化学习 颠覆级 有讲解视频
发表时间
2026-07-21
arXiv
2607.16204

收录解读

强化学习需要多样化训练环境,但手工环境难以适应模型性能提升,稀疏奖励导致模式崩溃。世界模型可模拟环境状态,但自回归模型存在从左到右偏差,无法基于全局锚点(如工具模式、先前轮次、预期结果)进行条件生成。

本文形式化文本世界建模为可控转移动力学问题,分解为初始状态、任务上下文、工具模式、领域规则和引导指令,并提出掩码扩散语言模型(MDLM)通过双向锚点感知去噪实现更好的连贯性和基础性。同时构建了包含239,403条轨迹的数据集,覆盖9个开源环境和12个前沿模型家族。

实验表明,MDLM在连贯性、基础性和轨迹多样性上优于参数规模大4倍的自回归模型,推理延迟相当。在3个分布外环境(ScienceWorld、ALFWorld、AppWorld)上,使用3种1.2B-7B智能体骨干进行零样本迁移,实现了高达47%的绝对性能提升,无需环境特定微调。还包括对抗场景下的行为分析和人工评估。

该工作开源了数据集和训练框架,为可控文本世界模型提供了新范式。局限性在于方法原创性未达范式级,且验证局限于文本环境,未来可扩展到多模态或物理世界。

原始摘要

Recent growth in reinforcement learning (RL) has surfaced a need for diverse, specialized training environments. Hand-curated environments with fixed task and reward difficulties become ineffective signals as model performance improves, and sparse rewards over long horizons induce mode collapse on specific workflows or tool structures. World models that simulate environment states have matched pure rollout performance, making them promising for scaling diversity on-demand. However, autoregressive (AR) world models suffer from a left-to-right bias preventing conditioning on globally interdependent state anchors such as tool schemas, prior turns, and expected outcomes. We (i) formalize text-based world modeling as a steerable transition-dynamics problem decomposed into initial state, task context, tool schemas, domain rules, and steering directives, and (ii) curate 239,403 grounded state-action trajectories spanning nine open-source environments and twelve frontier model families. We compare AR LMs and masked diffusion language models (MDLMs), showing MDLMs, via bidirectional anchor-aware denoising, achieve better coherence, groundedness, and empirically validated rollout diversity than LLMs over 4x their parameter size, at comparable inference latency. We introduce a plug-and-play GRPO training framework with deterministic state checks, and perform zero-shot transfer ablations on three OOD environments (ScienceWorld, ALFWorld, AppWorld) across three 1.2B-7B agent backbones (LFM2.5, Qwen3, Mistral), achieving up to 47% absolute gains over baselines without environment-specific fine-tuning. We further conduct behavioral analysis of failure modes under adversarial scenarios and human evaluation on realism, outcome correctness, and training utility. We open-source our work to encourage research in this direction.

解读视频

相关论文

链接