收录解读
第1段:AI与认知科学共同追求理解人类智能,但随着模型能力增强,现有方法难以在同等实验条件下大规模、规范化地比较人类与模型行为;任务多样性、实验范式不一致以及可复现性不足成为主要障碍。
第2段:CogGym提出一个可扩展的统一框架,利用半自动、人在回路的流水线,将不同认知实验范式标准化为任务无关的Experiment Markup Language (EML),实现对人类与模型行为的大规模、忠实比较;初始版本覆盖来自100篇论文的258项认知实验,并评估了50个大语言模型。
第3段:结果显示,更大、更新的模型整体上更接近人类判断,形成清晰scaling趋势;但在人类常识推理任务上的提升速度明显慢于数学和编程等正式推理基准。最佳模型与人类的拟合度R²分别为文本0.59、图像0.58、视频0.43,显著低于人类自身split-half信度(文本0.93、图像0.95、视频0.92),说明人类与模型行为仍存在显著差距。
第4段:该工作提供了大规模、可复用的认知评测基准与标准化工作流,有望成为持续演化的评估框架,对理解和衡量模型类人推理、安全对齐等有重要意义;但其初始版本集中于常识推理,且数据/代码开放程度、标准化过程偏差和长尾任务覆盖仍需进一步验证与完善。
原始摘要
Understanding and modeling human intelligence are parallel goals shared by artificial intelligence (AI) and cognitive science. As AI systems grow increasingly capable, in what ways do model responses resemble human responses, and where do they systematically diverge? The sheer breadth and diversity of the tasks humans can perform and think about pose a challenge for scalable and rigorous comparison between humans and models. We introduce CogGym, a scalable, unified framework grounded in cognitive science for systematically comparing model and human behavior on matched experimental trials. CogGym uses a semi-automated, human-in-the-loop pipeline to standardize diverse experimental paradigms into a task-agnostic Experiment Markup Language (EML), enabling reproducible and faithful comparison at scale. For initial release, we curate and standardize 258 cognitive experiments from 100 papers that focuses on human commonsense reasoning, and evaluate 50 large language models against human responses. We find a clear scaling trend where larger and more recent AI models better reproduce human judgments. Yet AI models' improvement on such common reasoning tasks is considerably slower than the gains observed on formal-reasoning benchmarks like math and coding, and model--human fit remains well below human splithalf reliability ($R^2 = 0.93$ on text, $0.95$ on image, and $0.92$ on video) with the best models achieving $R^2 = 0.59$ on text, $0.58$ on image, and $0.43$ on video experiments. We intend for CogGym to provide a living evaluation framework that continually incorporates new cognitive science experiments to characterize where model behavior resembles human behavior, where it systematically diverges, and how those patterns change as models and experiments evolve.