推理、记忆与推理时控制 颠覆级 暂无讲解视频
发表时间
2026-10-26
arXiv
2610.32305

收录解读

智能体能正确复述诊断结果,并不代表会在前提失败时停止后续无效分析。

DMAIC-Bench以15个工业案例的97项任务构造成对继续/停止分支,独立评分执行痕迹,并用专家审计最终结论。

13模型平均GO路由正确率92%、NO-GO为19%;专家审计估计47%的已到达NO-GO决策在最终报告中使用无效证据,区间31%–61%,二者不是同一指标。

相对 paradigm,基准仍依赖预设统计前提、有限案例和专家标签,未提供通用纠正机制,执行与语义失败率也明显不同。

原始摘要

Multi-stage analytical tasks require LLM agents to use intermediate evidence to decide whether to continue, redirect or stop. Self-reflection, in which an agent critiques its output, is widely studied, yet a critique does not show whether evidence changed the next action. We call that capability self-restraint. Existing agent benchmarks largely measure task completion or output quality and leave it untested. DMAIC-Bench evaluates it with 97 seeded tasks across 15 industrial cases in four sectors, built around Lean Six Sigma's explicit statistical prerequisites. Paired counterfactual variants require opposite actions at the same workflow position, distinguishing warranted restraint from indiscriminate stopping. We assess execution traces with a private scorer and final claims through independent expert review. Across 13 models from six providers, execution-level routing accuracy averages 92% on matched GO decisions and 19% on NO-GO decisions, counting any forbidden downstream computation as a route error. Execution traces show agents reporting correct diagnostics before performing analyses that those results disallow. Separately, blinded expert review of the 11 audited agents estimates that reports act on invalid evidence at 47% of reached NO-GO decisions, with an interval from 31% to 61%. Explicit stopping permission helps some models without consistently restoring correct decisions at both GO and NO-GO forks. Sixteen life-sciences tasks in bulk ribonucleic acid sequencing (RNA-seq) and case-control genome-wide association studies (GWAS) show that the same framework can evaluate evidence-dependent decisions beyond industrial workflows. Project code and data are available at: https://github.com/psptpx/DMAIC-Bench-public.

相关论文

链接