收录解读
大语言模型推理逐渐受限于内存而非计算,长上下文和长思维链任务使KV缓存占据主导内存开销,HBM容量成为限制批处理大小和系统吞吐的关键瓶颈。
OasisKV提出一种面向内存的推理系统设计,将完整KV缓存从HBM解耦,利用解码时注意力稀疏性仅保留最相关token的KV项,并通过投机解码生成的lookahead token准确预测未来重要块,从高容量内存层预取到HBM供后续解码使用。
基于vLLM的实现显示,在2048 token预算下精度损失不超过0.7点,推理负载相对密集vLLM获得1.69x吞吐提升,多GPU长上下文服务最高2.1x,在prefill-decode分离场景下约2x吞吐,同时显著降低KV传输与内存占用。
该工作有效缓解了KV缓存对HBM的依赖,为长上下文和推理强度高的场景提供了可落地的系统优化方案,但作为现有框架内的优化,尚未根本改变推理架构的设计范式。
原始摘要
Large language model (LLM) inference serving is increasingly constrained by memory rather than compute. As long-context and long-form reasoning workloads become more prevalent, the key-value (KV) cache dominates both memory footprint and memory traffic during LLM token generation, i.e., decode. In particular, HBM capacity has become a scarce and costly resource that heavily limits inference batch size and system throughput. This paper presents OasisKV, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding. Because decode-time attention is naturally sparse, OasisKV keeps only the KV entries of the most relevant tokens in HBMs for attention computation. We observe that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD). OasisKV employs an efficient attention background pipeline to identify important KV blocks. They are then prefetched from higher-capacity memory tiers (e.g., host or remote memory) and staged in HBMs before being used in the next decode step. We implement OasisKV based on vLLM. The lookahead prediction is accurate enough to keep accuracy within 0.7 points of full attention under a 2,048-token KV budget. This lets OasisKV turn sparsity into throughput gain: $1.69\times$ over dense vLLM on the reasoning workload at 0.1 points of accuracy loss, and up to $2.1\times$ on multi-GPU long-context serving. Under prefill--decode disaggregation, OasisKV reaches about $2\times$ dense throughput while admitting each request with $6.5$--$9.7\times$ less KV and holding $2.2$-$2.6$ less decode-node host memory than full KV transfer.