收录解读
LLM裁判在评估智能体工具调用系统时被广泛使用,但其在结构化、依赖驱动的工作流上的可靠性尚未被系统研究。现有评估多聚焦于开放式文本或偏好判断,忽略了智能体工具调用中DAG工作流的独特挑战。
本文提出AgentJudgeBench,首个系统研究LLM裁判在智能体工具调用中可靠性的基准。基准包含3808个实例,覆盖六种DAG拓扑和三个难度层级,使用五个生成器和六个裁判,在有/无真实标签条件下进行评估。
实验发现裁判对齐度随任务难度单调下降,无真实标签时下降速度加快1.5倍;所有裁判在困难查询上收敛至77-82%的狭窄区间,表明存在结构性天花板。链式推理和温度调节效果微弱,结构化评估准则可提升对齐度6.5个百分点但泛化性不足。
该基准揭示了当前LLM裁判的根本局限性,为智能体系统的可靠评估提供了实用指南。基准和代码已公开,可被后续研究直接复用。主要限制在于仅覆盖工具调用场景,未涉及更广泛的智能体行为评估。
原始摘要
LLM judges are widely used to evaluate agentic tool-calling systems, yet their reliability on structured, dependency-driven workflows remains largely unexamined. We present AgentJudgeBench, the first benchmark to systematically study LLM-as-a-judge reliability for agentic tool-calling over workflow DAGs, as distinct from the broader LLM-as-a-judge task of open-ended text or preference evaluation. The benchmark comprises 3,808 instances spanning six DAG topologies and three difficulty tiers, evaluated with five generators (3B-70B open-weight models and GPT-5.4) and six judges (20B to frontier scale) under paired with- and without-ground-truth conditions. Judge alignment degrades monotonically with task difficulty, 1.5x faster without ground truth, and on hard queries without ground truth all six judges converge to a narrow 77-82% band regardless of scale, revealing a structural ceiling driven primarily by task difficulty, though its height is partly prompt-dependent for weaker generators, that model capacity alone cannot overcome. Ground-truth exposure is not uniformly beneficial: it reduces alignment for GPT-5.4 (1.5 pp) and Gemini-2.5-Pro (3.9 pp), consistent with over-anchoring. Among mitigation strategies, chain-of-thought reasoning and judge temperature both have negligible effect, while structured evaluation rubrics improve alignment by up to 6.5 pp but do not generalize uniformly across judge-generator pairs. With ground truth, QwQ-32B best matches the programmatic reference, while a human validation study identifies GPT-OSS-120B as the most human-aligned judge; without it, frontier judges lead only marginally within the shared ceiling. These results expose fundamental limitations of current LLM judges and yield practical guidelines for reliable evaluation in agentic systems.