收录解读
AI代理可能因模型未对齐、提示注入或操作员滥用而泄露模型权重、投毒训练数据或削弱发布门控。现有基准未测试防御者能否在有限审核预算下从日常工作中检测此类活动。
MOLE 将智能体内部威胁检测从单条恶意输入扩展到持续组织活动日志,并按审查预算评价监控器。
环境含 150 个账户、九类有状态服务和 30 天活动,构造 12 类威胁及八套语料;评测 39 个攻击模型和 40 个监控器,并设置时间及账户保留划分。
读过 PDF 环境、划分与预算受限监控实验;组织级连续日志和审查成本共同评价构成显著可复用的方法推进,但将原 27 分收紧为 24 分。 仍是一家合成组织中的指定威胁及离线检测;未证明真实组织、自发失范和自适应攻防的普遍有效性,因此不到 paradigm。
原始摘要
Model misalignment, prompt injection, or operator misuse could lead AI agents operating frontier-lab accounts to exfiltrate model weights, poison training data, or weaken release gates. Existing benchmarks do not test whether defenders can detect this activity among routine work under a limited review budget. We introduce MOLE, an open benchmark of 150 AI-operated accounts sharing 9 stateful services over 30 workdays, with 12 threats and 8 corpora from four models totaling roughly 20 billion tokens. Of 39 agent models, 72% complete most assigned harmful objectives and agent refusal does not predict completion. MOLE enables comparison of 40 monitors across corpus generators, observability levels, and threats; even the best evaluated monitor in our single-day audit-event comparison misses nearly half of completed harm. MOLE also enables monitor development: benchmark-guided search improves a mid-tier monitor by 49-64%, while selective use of a stronger monitor improves budget-AUC by 10% over applying it to every account-day at comparable modeled cost.