收录解读
逐token生成使语言模型推理受到串行依赖限制。PLM通过变分解码将并行预测作为训练目标的一部分,探索可学习的并行生成机制。
模型用潜变量辅助解码器组织多个token的联合预测,同时保留自回归能力;论文区分精确和放宽接受的推理策略,以调节速度与生成质量。
正文覆盖80M至3B模型,1.3B匹配自回归训练基线,并评测数学、代码与指令任务;报告2.3–5.7倍加速,但不同设置存在质量折中,3B比较对象主要是自身自回归模式。
联合训练目标和跨规模验证使其达到disruptive 24分。最大规模、硬件验证与放宽接受的质量影响仍限制结论,尚非普适的自回归替代范式。
原始摘要
Autoregressive language models emit one token per forward pass. This causes high inference latency, in particular as models are scaled up for quality. In this paper, we introduce Parallel Language Models (PLMs), which generate mutually consistent tokens in one forward pass by routing dependencies through latent auxiliary variables. PLMs are trained using a variational bound that reduces to an autoregressive loss plus a parallel decoding loss. Empirically, PLMs are on a faster and better Pareto frontier than autoregressive and discrete diffusion baselines. Pretrained PLMs achieve near-autoregressive pretraining loss on model scales of 80M to 3B parameters, and speedup follows a power law in model size. When post-trained, our 1.3B model matches or exceeds an identically trained autoregressive model on math, coding, and instruction-following benchmarks, speeding up inference $2.3\text{-}5.7\times$. Together, we show that parallel generation can be trained from a single training objective at autoregressive quality. We will make our code available at https://github.com/mandt-lab/plm.