收录解读
LLM-as-judge是评估开放文本和指导后训练的关键技术,但传统方法依赖昂贵的人工标注、奖励模型或强教师蒸馏。本文针对这一问题,提出无需外部监督的递归自我评估方法。
核心方法RecurSE包含两个阶段:Pass 1中可训练的判断器按规则评估候选回复;Pass 2中同步的策略副本检查器根据元规则审计判断器推理,提供过程奖励。接口解耦技术隔离检查器的标量分数与判断器的令牌向量,避免令牌复制捷径导致的自我奖励膨胀。同时,成对优势有效性(PAV)作为无偏验证监控器,联合跟踪判断器准确性和检查器保真度,确定最优早停窗口。
实验在Qwen3.5-9B、Gemma-4-E4B-it和Qwen3.6-27B三个模型上,在医学、成对比较、摘要和专业领域基准上展现一致的泛化提升。消融实验表明,同步判断器-检查器协同进化优于冻结检查器、外部元判断器、自一致性和缩放教师蒸馏。利用该方法生成的偏好对能有效增强下游策略对齐。
该工作证明了有界递归自我评估在LLM判分器中的可行性,但摘要中缺乏具体指标和效果量,验证强度不足。其方法具有原创性,但尚未达到需要正式收录的突破性标准,建议跟踪未来完整版本。
原始摘要
LLM-as-judge is essential for evaluating open-ended text and steering post-training, yet improving the judge itself typically relies on expensive annotations, reward models, or distillation from stronger teachers. In this work, we eliminate external gold supervision from the RL training reward: the model's own evaluative capability generates learning signals for its optimization -- a closed-loop setting of bounded recursive self-improvement (RSI) termed Recursive Self-Evaluation (RecurSE). We study two central questions: when can self-improvement occur, and when must it stop? First, RecurSE pairs a trainable judge evaluating candidate responses under per-rule rubrics (Pass 1) with a synchronized policy-copy checker that audits the judge's reasoning against meta-rubrics to supply a scalar process reward (Pass 2). To enable learning, interface decoupling structurally isolates the checker's scalar score from the judge's verdict tokens, eliminating a degenerative token-copying shortcut that inflates self-assigned rewards. Second, because unanchored recursive learning is inherently bounded, Pairwise Advantage Validity (PAV) serves as an unbiased validation monitor that jointly tracks judge accuracy and checker fidelity to reliably identify the optimal early-stopping window. Across Qwen3.5-9B, Gemma-4-E4B-it, and Qwen3.6-27B, RecurSE achieves consistent generalization gains across held-out medical, pairwise, summarization, and professional benchmarks. Ablations demonstrate that synchronized judge-checker co-evolution outperforms frozen checkers, external meta-judges, self-consistency, and scaled teacher distillation. Furthermore, preference pairs curated by our judge effectively enhance downstream policy alignment. Bounded RSI for LLM-as-judge is thus viable when self-produced reward validity is explicitly decoupled and monitored.