收录解读
多模态表示对齐是人工智能的核心问题,柏拉图表示假说(PRH)认为不同模态的模型在规模增大时会收敛到相同表示,但缺乏严格证明。本文将该假说转化为定理,并提供了实证验证。
本文证明了联合嵌入预测架构(JEPA)的柏拉图表示定理:当正样本对为同一场景的“好视角”且错位有界时,不同模态独立训练的JEPA能学到仅差一个旋转的相同表示。同时指出,下一个token预测本质上是使用“坏视角”的JEPA,解释了为何语言模型需要大规模才能与视觉对齐。
实验基于145k字幕和940M网络图像文本,发现好视角只需42倍少于下一个token预测的样本即可达到相同对齐度。109M参数的文本JEPA与DINOv2和CLIP图像编码器的对齐度超过所有评估的语言模型(最高72B参数)以及CLIP文本编码器。此外,一次旋转即可将独立训练的编码器转化为图像描述模型(COCO CIDEr 0.838),并在无配对基准上以64% top-5准确率超越8B参数语言模型(最高56%)。
该工作为多模态表示对齐提供了理论基础和实用指南,但理论假设(好视角、有界错位)可能不覆盖所有场景,实验仅涉及文本和图像模态。尽管如此,其结论对理解表示学习和指导未来多模态模型设计具有重要意义。
原始摘要
Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a \emph{Platonic Representation Theorem}: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene--\emph{good views}--and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with \emph{bad views}, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64\% against at most 56\% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.