核心要点
- 问题/背景
- 这篇 NeurIPS 2025 Oral / Best Paper Runner-up 论文试图解释神经 scaling law 的来源：LLM 需要在有限隐藏维度中表示远多于维度数的 token、概念和特征，因此不可避免处在 representation superposition 状态。
- 方法/机制
- 作者基于 Anthropic Toy Models of Superposition 引入可调 superposition 强度的实验旋钮，用 weight decay / growth 控制表示向量的叠加程度，从而把数据频率分布和表示几何干扰对 loss scaling 的贡献拆开。
- 结果/证据
- 核心结果是：弱叠加时，loss 幂律依赖特征频率本身是否是幂律；强叠加时，loss 在广泛频率分布下都近似按模型维度的倒数下降，原因来自高维球面中大量表示向量的平方重叠随维度缩小。
- 收录价值
- 它值得收录，因为它把 scaling law、mechanistic interpretability 中的 superposition、LLM hidden-space geometry 连接成一个可检验框架，并给出实际开源 LLM 处于强叠加区间的证据，对模型宽度、表示容量和何时 scaling 会失效的理论判断有长期参考价值。
原始摘要与中文对照
中文对照翻译
今天大型语言模型（LLMs）的成功取决于一个观察结果：模型越大，性能越好。然而，这种神经缩放定律（即损失随模型规模呈幂律下降）的起源仍不清楚。我们提出，表征叠加（即LLMs表示的特征多于其维度）可能是损失的关键贡献者并导致神经缩放。基于Anthropic的玩具模型，我们使用权重衰减来控制叠加程度，从而系统地研究损失如何随模型规模缩放。当叠加较弱时，损失仅在数据特征频率呈幂律分布时才遵循幂律。相比之下，在强叠加下，由于表征向量之间的几何重叠，损失通常会随模型维度呈反比缩放，适用于广泛的频率分布类别。我们证实了开源LLMs在强叠加机制下运行，并且损失随模型维度呈反比缩放，Chinchilla缩放定律也与此行为一致。我们的结果将表征叠加确定为神经缩放定律的核心驱动因素，为诸如神经缩放定律何时可以改进以及何时会失效等问题提供了见解。
原始摘要
The success of today’s large language models (LLMs) depends on the observation that larger models perform better. However, the origin of this neural scaling law, that loss decreases as a power law with model size, remains unclear. We propose that representation superposition, meaning that LLMs represent more features than they have dimensions, can be a key contributor to loss and cause neural scaling. Based on Anthropic’s toy model, we use weight decay to control the degree of superposition, allowing us to systematically study how loss scales with model size. When superposition is weak, the loss follows a power law only if data feature frequencies are power-law distributed. In contrast, under strong superposition, the loss generically scales inversely with model dimension across a broad class of frequency distributions, due to geometric overlaps between representation vectors. We confirmed that open-sourced LLMs operate in the strong superposition regime and have loss scaling inversely with model dimension, and that the Chinchilla scaling laws are also consistent with this behavior. Our results identify representation superposition as a central driver of neural scaling laws, providing insights into questions like when neural scaling laws can be improved and when they will break down.1