Systems And Hardware 颠覆级 暂无讲解视频
发表时间
2026-08-07
arXiv
2608.07009

收录解读

长上下文大模型解码中,稀疏注意力虽减少了计算量,但KV缓存仍占用大量显存,导致内存成为瓶颈。现有服务系统为保持所有KV可选择,通常将所有缓存放入GPU HBM,当上下文超过显存时无法服务。

HiSparse提出一种精确、与索引器无关的分层KV缓存管理方法,将完整KV历史保存在主机内存中,GPU上仅保留固定大小的缓存。通过融合CUDA内核在解码图中处理选择命中、LRU替换和主机到设备抓取,并利用跨层选择共享性进行预取,掩盖一半缺失开销。

实验在H200、B200、GH200平台上对DSA、NSA、Quest三种稀疏注意力家族进行评测,峰值生成吞吐提升最多4.7倍,同时保持每token延迟相当,并减少高负载下的首token时间。无IO对照实验表明解析机制本身无额外开销,主要开销仅为主机-设备IO。

HiSparse已合并进SGLang,具备实际可复用性。其分层缓存策略对稀疏注意力服务系统具有广泛影响,但本质是系统层面的优化,未改变稀疏注意力的研究范式,仍属工程创新而非理论突破。

原始摘要

Top-k sparse attention makes long-context LLM decoding cheap to compute: each step reads only a few thousand selected KV entries rather than the full context. Serving systems, however, typically keep the entire KV cache in GPU HBM so that every position stays selectable, so a request's memory bill still grows with its full context length--decoding hits a capacity wall long before it runs out of compute, and a context whose KV cache exceeds HBM cannot be served at all. We present HiSparse, an exact, indexer-agnostic hierarchical KV cache for sparse-attention serving. HiSparse keeps each request's full KV history in host memory and bounds its decode footprint with a small, fixed-size GPU cache; a fused CUDA kernel resolves each layer's selections--hit detection, LRU replacement, and host-to-device fetches--inside the decode CUDA graph; and, for models that share selections across layers, exact layer-wise prefetching hides roughly half of the remaining miss overhead. Because only KV placement changes, model outputs are unchanged. HiSparse is merged into upstream SGLang and evaluated across three sparse-attention families (DSA, NSA, and Quest) on H200, B200, and GH200 platforms: it improves peak generation throughput by up to 4.7x on long-context workloads while preserving comparable per-token latency and reducing time-to-first-token at high load--and a no-IO oracle shows the resolution mechanism itself adds no measurable per-token cost, leaving host-device IO as the only price of bounded residency.

相关论文

链接