安全、治理与可靠性 颠覆级 有讲解视频
发表时间
2026-09-14
arXiv
2609.11952

收录解读

化学与材料智能体将文献检索、候选生成、性质预测和方案规划整合为连续发现流程,其安全风险从回答危险问题转向通过工具中介流程释放危险协议。现有安全评估多聚焦单轮问答,缺乏对长时程、多步骤工具交互中攻击与防御的系统评测。

本文提出ChemMat-AgentSafetyBench,包含432个固定危险案例规范,覆盖8类危害、3种场景壳、4种工具与记忆环境,并设计5种在线长时程攻击(意图劫持、工具链、目标漂移、任务注入、记忆投毒)。攻击语言由运行时轨迹动态生成,避免静态基准的局限性。

全文完成1,728次主实验、攻击入口分解、防御配对和真实智能体接口转移,证明最终回答过滤无法覆盖工具状态与持久记忆风险。

该基准实质改变科学智能体安全的评估对象与实验协议,达到颠覆性;尚非真实实验室验证,也未形成通用防御范式。

原始摘要

Chemistry and materials agents integrate literature retrieval, candidate generation, property prediction, and protocol planning into continuous discovery workflows. Consequently, the relevant safety question is shifting from whether a model answers a hazardous question to whether an agent releases a hazardous protocol through a tool-mediated workflow. We introduce \bench, a benchmark that evaluates whether chemistry and materials agents can be steered toward hazardous endpoints through user input, tool observations, or persistent memory. The benchmark contains 432 fixed harmful case specifications spanning eight hazard classes, three scenario shells, four tool-and-memory environments, a single-turn direct-attack baseline, and five online long-horizon attacks: intent hijacking, tool chaining, objective drifting, task injection, and memory poisoning. The concrete language of each online attack is generated from the evolving trajectory at runtime and is therefore not counted in the static benchmark size. In the four-model main experiment with a fixed attacker, agents release complete hazardous synthesis or preparation procedures in 25.6\% of runs. Replacing the attacker model yields mean success rates from 18.4\% to 26.5\%, indicating that the risk is not an artifact of a single attacker. Input- and state-level defenses adapted from general-purpose agent safety, as well as candidate checks designed for chemistry and materials, reduce some failures but still leave complete-path release rates between 9.2\% and 22.5\%. Existing defenses therefore do not simultaneously cover multi-entry contamination, tool state, and the final artifact boundary. These results highlight a widening gap between the rapid development of scientific agents and the safety evaluation and defenses available to the chemistry and materials community.

解读视频

相关论文

链接