收录解读
该论文针对大语言模型后训练中推理能力难以检查、验证和迁移的问题,提出将推理过程显式化为由确定性和神经原语组成的程序。
核心方法PLVR通过符号反向传播(基于类型推理的链式法则)实现逐步骤的信用分配,每个程序层携带类型本体,损失通过类型签名反向传播。
在LiveCodeBench v6和Tau2Bench上,30B模型使用PLVR在相同预算下平均比RLVR高27.8分,并超过大一个数量级的顶尖模型13.6分;消融实验证明反向传播而非类型系统是优势来源。
该方法开源了符号反向传播库和一致性检查器,单个原语库可服务多个基准,新任务仅需100个示例进行程序搜索,但通用性限于具有可验证中间步骤的任务。
原始摘要
Post training a language model to reason means updating its weights. Supervised finetuning and reinforcement learning both place the acquired capability inside the model where it cannot be inspected cannot be checked step by step and cannot be moved to another model. We argue that for tasks whose intermediate steps admit verification, reasoning is better placed outside the base models weights as an explicit program composed from deterministic and neural primitives. We introduce PLVR (Program Learning with Verifiable Rewards): a post training method that learns such programs directly from input-output examples. Its mechanism is symbolic backpropagation: each program layer carries a typed ontology a loss is computed at the output against ground truth and required input ontologies are propagated backward by type inference over primitive signatures: an analogue of the chain rule in which credit assignment is a derivation rather than an estimate. Where RLVR verifies a terminal outcome, PLVRs reward is a per step contract verdict dense over program structure. On LiveCodeBench v6 and Tau2Bench, 30B base models with PLVR outperform RL at matched budget by 27.8 points on average and frontier models an order of magnitude larger by 13.6 points. A single primitive library serves two benchmarks, so the marginal cost of a new task is 100 examples of program search and no new finetuning data. Replacing the loss guided search with uniform sampling over the same type admissible space at equal budget collapses the median program from 65.6 to 17.5, identifying the backward pass rather than the type system as the source of the advantage. We release the symbolic backpropagation library and a conformance checker so the method can be applied to primitive libraries other than our own.