收录解读
当前推理数据依赖人工标注或从强LLM蒸馏,忽略了专家系统(如游戏引擎、经典规划器、定理证明器)产生的动作中蕴含的知识,但这些专家系统不输出链式思维(CoT)。
提出LeAct框架,将CoT视为潜在变量,通过优化学生模型恢复动作的概率来从专家动作中学习推理链,从而蒸馏专家知识到可泛化的学生模型中。
在多个不完全信息游戏(从小规模枚举到大规模Flop Hold'em)和模拟机器人基准上验证,LeAct达到小规模求解器下限,大规模远超基线(5倍接近求解器,对战中+60 mbb/g),在机器人任务上唯一优于直接模仿。
该工作开创了利用专家系统作为推理教师的新范式,具有重要价值,但泛化范围目前限于游戏和机器人领域,跨更多推理任务(如数学、代码)的验证尚未展开。
原始摘要
Modern reasoning models depend on reasoning data, today sourced from human annotations or distilled from stronger LLMs. However, a rich and largely untapped source of supervision lies in expert systems (e.g., game engines, classical planners, theorem provers), which routinely produce near-optimal actions across diverse domains. But these experts are silent: they commit to an action without writing down the chain of thought (CoT) behind it. Recovering that CoT as natural-language reasoning would distill expert knowledge into a student that generalizes beyond the demonstrated actions. We treat it as a latent variable and study how to recover it from the action alone. Our approach, LeAct (Learning to reason from Actions), optimizes this latent variable: the student samples candidate CoTs for each expert action, and we retain those that measurably improve its own probability of recovering the action. Across imperfect-information games at multiple scales and a simulated robotics benchmark, LeAct reaches the solver's numerical floor on small enumerable games. At larger scale, it is $5\times$ closer to the solver than the strongest expert-iteration baseline. At Flop Hold'em ($\sim 10^9$ infosets), LeAct wins head-to-head by $+60$ mbb/g, and on the robotics probe it is the only training recipe that improves on direct imitation. We present a principled framework and the result: expert systems become a categorically new source of reasoning teachers for foundation models.