收录解读
投机解码通过轻量级草稿模型加速大语言模型推理,但现有系统未能充分发挥并行草稿的潜力,面临单请求接受率低、批处理验证开销大、部署场景依赖性强三大挑战。
本文提出SpecPilot,一个自主智能体循环系统,通过成本模型定位瓶颈、并行智能体探索优化方向、令牌强制检测奖励黑客、结构化记忆跨轮次迁移有效机制,自动发现算法与系统优化方案。
代价上限引导的代理搜索结合逐token校验,四目标模型/三drafter/两GPU/五基准验证;单请求相对AR加速4.46到10.87倍、批量1.94到3.93倍,并产出可复用W12精确权重格式。
greedy解码容许浮点近并列差异,非所有采样分布的逐字相同保证;13轮需164小时/530GPU小时,部署瓶颈高度相关,尚无足以称paradigm的广域范式替代证据。
原始摘要
Speculative decoding accelerates memory-bound LLM decoding with a lightweight drafter, and parallel drafters extend this speedup by drafting a whole block in one forward pass. However, it remains unclear whether their released systems (stock) attain the full speedup that parallel drafting permits. We identify three challenges. First, with one request, each step accepts few tokens but reads every target weight. Second, with batching, per-request verification work roughly halves the speedup over autoregressive (AR) decoding. Third, speculative decoding is deployment-specific, and no universal optimization fits every serving regime, drafter, target model and GPU. To address these challenges, we present SpecPilot, an autonomous agentic loop that discovers algorithms and systems optimizations for each deployment. In each round, a cost model locates the largest gaps between the current system and its ceiling. Parallel agents explore directions and integrate the successful ones. Token forcing checks every output token against stock to detect reward hacking. A structured memory carries validated mechanisms to new rounds and deployments. SpecPilot discovered a general lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification and three batching mechanisms. We evaluate it across four target models, three drafters, two GPU platforms and five benchmarks. Without training or quality loss, it raises the speedup over AR from stock's 4.46$\times$ to 10.87$\times$ with one request, and from 1.94$\times$ to 3.93$\times$ with batching.