收录解读
论文针对新兴的AutoResearch(自动科研)现象,指出现有评估仅关注性能指标而忽视过程细节,且任务范围狭窄、失败诊断缺乏系统性。为此提出了AutoResearchEval基准,包含100个来自前沿科学的任务,覆盖7个学科领域和完整研究生命周期(从构思到写作与评审)。
核心方法是对8种harness-model组合(共800条轨迹)进行过程级注释,并构建了包含45种失败模式的AutoResearch Failure Taxonomy(ARFT)。同时设计了经人类校准的agent-as-a-judge流水线,以低成本实现细致的失败归因。
实验揭示了所有组合中反复出现的一个根本限制:当前agent缺乏元认知循环,即无法对产出与发现进行检验、修订,以及质疑路径合理性。该模式在所有8种组合包括最强模型中均存在,表明缺陷在于模型层面而非特定scaffold。
该工作释放了AutoResearchEval和ARFT,为自主科学发现研究提供了可复用的评估基础设施。其价值在于将评估焦点从“能否完成”转向“如何完成及何处失败”,为后续研究指明方向。局限在于未测试orchestration层面的干预是否可弥补该缺陷,且评估本身不提出改进方法。
原始摘要
AI has long assisted scientific research, but the rapid advance of LLMs and agentic scaffolds is reshaping the landscape; a single system can now carry whole-stage research from an initial hypothesis all the way to final published paper, which is a paradigm now referred to as AutoResearch. Existing evaluations reveal little about how these agents operate or where they break down. Tasks are narrowly-scoped, evaluation measures performance but not process, and failure diagnoses lack systematic coverage or artifact-level visibility. To address this gap, we introduce AutoResearchEval, featuring 100 tasks grounded in published frontier science across 7 scientific domains and the full research lifecycle, including ideation, retrieval, execution, analysis, writing, and review. Evaluating 8 harness-model combinations yields 800 autoresearch agent trajectories, with process-level annotation. We organize these insights into AutoResearch Failure Taxonomy or ARFT, a framework of 45 empirically-grounded failure patterns. To enable scalable fine-grained attribution, we leverage a human-calibrated agent-as-a-judge pipeline to inspect complete trajectories and intermediate artifacts. Failure patterns converge on a single overarching limitation, namely that current agents lack a metacognitive loop, which entails the ability to check what they produced against what they found, revise when it does not hold up, and question whether the path they took was sound. The same patterns recur across all 8 harness-model combinations, including the strongest models tested, locating the deficit at the model level rather than in any particular scaffold; whether orchestration-level interventions can close it is an open question this work does not test. We publicly release AutoResearchEval and ARFT to facilitate continued research and development in autonomous scientific discovery.