推理、记忆与推理时控制 颠覆级 有讲解视频
发表时间
2026-08-28
arXiv
2608.26623

收录解读

LLM裁判在评估智能体工具调用系统时被广泛使用,但其在结构化、依赖驱动的工作流上的可靠性尚未被系统研究。现有评估多聚焦于开放式文本或偏好判断,忽略了智能体工具调用中DAG工作流的独特挑战。

本文提出AgentJudgeBench,首个系统研究LLM裁判在智能体工具调用中可靠性的基准。基准包含3808个实例,覆盖六种DAG拓扑和三个难度层级,使用五个生成器和六个裁判,在有/无真实标签条件下进行评估。

实验发现裁判对齐度随任务难度单调下降,无真实标签时下降速度加快1.5倍;所有裁判在困难查询上收敛至77-82%的狭窄区间,表明存在结构性天花板。链式推理和温度调节效果微弱,结构化评估准则可提升对齐度6.5个百分点但泛化性不足。

该基准揭示了当前LLM裁判的根本局限性,为智能体系统的可靠评估提供了实用指南。基准和代码已公开,可被后续研究直接复用。主要限制在于仅覆盖工具调用场景,未涉及更广泛的智能体行为评估。

原始摘要

LLM judges are widely used to evaluate agentic tool-calling systems, yet their reliability on structured, dependency-driven workflows remains largely unexamined. We present AgentJudgeBench, the first benchmark to systematically study LLM-as-a-judge reliability for agentic tool-calling over workflow DAGs, as distinct from the broader LLM-as-a-judge task of open-ended text or preference evaluation. The benchmark comprises 3,808 instances spanning six DAG topologies and three difficulty tiers, evaluated with five generators (3B-70B open-weight models and GPT-5.4) and six judges (20B to frontier scale) under paired with- and without-ground-truth conditions. Judge alignment degrades monotonically with task difficulty, 1.5x faster without ground truth, and on hard queries without ground truth all six judges converge to a narrow 77-82% band regardless of scale, revealing a structural ceiling driven primarily by task difficulty, though its height is partly prompt-dependent for weaker generators, that model capacity alone cannot overcome. Ground-truth exposure is not uniformly beneficial: it reduces alignment for GPT-5.4 (1.5 pp) and Gemini-2.5-Pro (3.9 pp), consistent with over-anchoring. Among mitigation strategies, chain-of-thought reasoning and judge temperature both have negligible effect, while structured evaluation rubrics improve alignment by up to 6.5 pp but do not generalize uniformly across judge-generator pairs. With ground truth, QwQ-32B best matches the programmatic reference, while a human validation study identifies GPT-OSS-120B as the most human-aligned judge; without it, frontier judges lead only marginally within the shared ceiling. These results expose fundamental limitations of current LLM judges and yield practical guidelines for reliable evaluation in agentic systems.

解读视频

相关论文

链接