收录解读
前沿大语言模型在嵌套条件规则评估中表现出一种称为“异常链崩溃”的失败模式,即模型在复杂规则链下持续出错,且该错误模式随模型更新而不稳定,严重影响合规应用。
本文提出Aethis Eligibility Module,一种神经符号架构:LLM从权威来源编写规则,由SMT层确定性执行,从而将不确定性从推理边界转移到规范边界。
在跨四个监管领域的225个场景基准测试、20个对抗场景以及9个LegalBench任务(949个案例)上,该引擎显著优于GPT-5.4等前沿模型,最高准确率提升41个点,统计检验p≤0.003。
该工作为高风险规则评估提供了可审计、可复现的解决方案,但神经符号方法的通用性和跨领域迁移能力有待进一步验证。
原始摘要
We document a failure class in frontier large language models -- exception chain collapse -- observed in eligibility evaluation under nested conditional rules of the form "A is required UNLESS B applies, UNLESS C overrides B". The failure reproduces at first observation, but its empirical surface is unstable: between March and April 2026 several failure cells closed silently under the same model alias, with no version bump (GPT-5.4 on construction insurance moved from 96.6% to 100%, same prompt and harness). For regulated workflows, frontier-model accuracy is a moving compliance boundary that shifts without notice. We present the Aethis Eligibility Module, a neuro-symbolic architecture in which LLMs author rules from authoritative sources and an SMT-based layer executes them deterministically, consistent with the authored specification regardless of model drift, reasoning-effort defaults, or prompt format. Three evidence bases: (i) a controlled benchmark of 225 scenarios across four regulatory domains documents the pattern and, in replication, the drift that partially closed it; (ii) a 20-scenario adversarial extension on construction insurance, where the engine scores 20/20, as does one of four frontier configurations (GPT-5.4 at low reasoning effort), while the other three, including Anthropic's strongest model at evaluation time, fail the same coverage-gap edge case; (iii) external validation on nine peer-reviewed LegalBench tasks, 949 held-out cases, where the engine is significantly more accurate than all three frontier models (combined McNemar's p <= 0.003), with margins up to +41 points on the curated multi-prong tasks against the Anthropic models. The contribution is to relocate uncertainty from the inference boundary, where it is silent, to the specification boundary, where it is deliberate and audited. All scenarios, rule encodings, and results are public and reproducible.