收录解读
线性注意力模型用固定大小的循环状态替代增长的KV缓存,但在并发服务中这些状态成为内存瓶颈。直接量化循环状态会导致精度严重下降,因为量化误差会通过状态更新传播。本文发现误差影响取决于两个维度:时间上,长寿命记忆中的误差会跨解码步骤持续;空间上,不同键行对输出的影响不同,且状态幅度在行列间变化很大。
基于这些观察,本文提出STEPQuant,一种针对Delta-rule循环状态的空间-时间后训练量化框架。STEPQuant根据误差幅度和记忆寿命分配精度,并联合拟合键行和值列的缩放因子,基于状态分布和键行对输出误差的影响。
已读 PDF 表 1–4、紧凑存储核算与吞吐附录:STEPQuant 按误差持久性和读出影响分配精度并作双轴拟合,让 GDN/KDA 两种 Delta-rule 状态在有效约 6.3bit 达近 FP32 准确率,改变低比特部署可行性。
距离下一档 paradigm,仍局限 Delta-rule 两模型,未建立全架构新范式;并发 DAMP 为不同设置的相对准确率比较,压缩比含元数据但排除 allocator/TP padding,不等于总显存缩小五倍。
原始摘要
Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through successive state updates. We discover that the impact of these errors depends on two complementary dimensions: temporally, errors in long-lived memory can persist across many decoding steps; spatially, errors in different key rows affect model outputs differently, while state magnitudes vary substantially along both rows and columns. Motivated by these observations, we propose STEPQuant, a spatial-temporal post-training quantization framework for Delta-rule recurrent states. STEPQuant allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales based on state distributions and key-row impact on output error. Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across both long- and short-generation benchmarks show that STEPQuant closely matches FP32-state accuracy under a nominal 6-bit budget and outperforms uniform INT8 in its 4-bit configuration. Integrated into SGLang with optimized GPU kernels, 6-bit STEPQuant achieves over 5x recurrent-state compression and reduces total serving memory by up to 68.7%. Our code is available at https://github.com/Dreamer-Toby/STEPQuant.