收录解读
随着大语言模型在科学工作流中被提出作为代理,其部署前提是模型能够区分可靠与不可靠的科学文献。然而现有基准仅评估已知答案的事实性,未直接测量这一关键能力,尤其是在缺乏下游验证者的领域。
本文引入TRACES基准,包含42篇被撤稿、欺诈或伪科学论文,并设计了一种方法,通过提取接近原文的前言和科学上合理的研究设计请求来引导模型参与。探针涵盖五种声明类型,并采用两个互补得分(IFR-a和IFR-i)以及参与深度指数(EDI)来量化模型行为。
在30个模型和10次重复运行中,聚合IFR-a为0.93±0.004,IFR-i为0.809±0.009,但模型在95%的非空响应中参与了不可靠前提。每个模型在超过71%的代理探针中失败,22/30的模型失败率超过90%。拒绝行为集中在少量高知名度主题上,并在匹配结构对照下消失。
结果一致表明模型行为是主题相关的安全行为而非稳健的认知能力,凸显了科学部署中防护栏基础设施的紧迫性。该基准为AI科学代理的可靠性评估提供了可复用的工具,但其通用性仍受限于科学文献可靠性这一特定场景。
原始摘要
Large language models are being proposed as agents in scientific workflows, in domains where no downstream verifier exists. Such deployment assumes the model can distinguish reliable scientific literature from unreliable literature, a capability that has not yet been directly measured. Existing benchmarks evaluate factuality on questions with known answers; the failure mode we target here is different. We introduce a probe corpus of 42 retracted, fraudulent, and pseudoscientific papers, paired with a methodology for eliciting and scoring single-shot model engagement with each paper's framing. Each probe pairs a preamble extracted near-verbatim from the target paper with a scientifically plausible study-design request. The probes span five claim types: fabricated observation, pseudophysical mechanism, magical premise, legitimization bridge, and cargo-cult experiment. Two complementary scores measure whether a model rejects the flawed premise outright (IFR-a) and whether it recognizes the unreliability while still engaging (IFR-i). A depth score, the Engagement Depth Index (EDI), quantifies reproduction of paper- or field-specific withheld details. Across 30 models and 10 repeated runs, aggregate IFR-a is 0.93 $\pm$ 0.004 and aggregate IFR-i is 0.809 $\pm$ 0.009. Models engaged with untenable premises in 95% of all non-empty responses. Every evaluated model fails more than 71% of agentic probes, and 22 of 30 models fail more than 90% of the time. Rejections are concentrated on a small number of high-notoriety topics and specific probes, and disappear under matched-structure controls. These results are consistent with topic-keyed safety behavior rather than robust epistemic competence, and indicate an urgent need for guardrail infrastructure for scientific deployment of language models.