推理、记忆与推理时控制 颠覆级 暂无讲解视频
发表时间
2026-09-14
arXiv
2609.13141

收录解读

该论文针对预训练Transformer的注意力稀疏化问题,指出现有可训练方法通常使用轻量选择器对上下文单元打分,然后通过硬Top-K选择阻断梯度,导致选择器只能通过蒸馏稠密注意力分布来学习排序,无法直接优化在固定注意力预算下的预测影响。这种错位可能浪费有限的注意力预算在不太有用的单元上。

作者提出简单注意力稀疏化(SAS),一种门控稀疏注意力机制,通过将选择器的连续分数注入注意力logits,使语言建模损失能够通过标准反向传播端到端更新选择器。关键设计包括将门置于softmax内部的对数形式、使用归一化softmax门校准历史上下文与始终保留的当前块,以及保留连续选择器分数以学习相对优先级。

全文用同结构选择器的受控对照证明语言建模损失优于注意力蒸馏,并在推理、长上下文、agentic任务和真实解码内核上复现。

SAS显著改变后训练稀疏注意力的监督方式并同时改善精度与系统速度,达到颠覆性;尚未解决稠密prefill与超长针式检索。

原始摘要

Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context units (tokens or blocks) for each query. Existing trainable methods usually use a lightweight selector to score context units, followed by hard Top-K selection that blocks gradients from the language modeling loss. Consequently, these methods commonly distill layer-wise dense attention distributions. Although this encourages the selector to rank context units by dense attention weights in the original model, the ranking is not directly aligned with their impact on predictions under a fixed attention budget (i.e., the number of attended context units per query), potentially wasting the limited budget on less useful units. To address this misalignment, we propose Simple Attention Sparsification (SAS), a gated sparse attention mechanism that optimizes context ranking end-to-end with the language modeling loss. The key idea is to inject the selector's continuous scores into attention logits during training, allowing the loss to update the selector through standard backpropagation. We identify several choices crucial for this simple design to work well in practice: placing the gate inside the attention softmax in log form, using normalized softmax gates to calibrate historical context against the always-retained current block, and preserving continuous selector scores so the model learns relative priorities rather than only hard selections. To support long-sequence training, we implement a memory-efficient Triton kernel that integrates SAS into FlashAttention-style computation. Across reasoning, long-context understanding, and agentic tasks, SAS consistently outperforms trainable sparse attention baselines across attention budgets, with especially large gains under tight budgets, demonstrating more effective context ranking for downstream tasks.

相关论文

链接