核心要点
- 问题/背景
- 这篇论文处理的是 genomic foundation models 进入快速扩张后出现的评测失真问题：不同模型使用碎片化任务、不同 protocol 和任务特定报告方式，导致所谓更强模型往往不可直接比较。
- 方法/机制
- 作者提出 GENEB，用统一 probing protocol 评估 40 个 genomic foundation models 在 100 个任务、13 个功能类别上的 frozen representations，并纳入 few-shot 条件。结果显示 aggregate leaderboard 很不稳定，模型排名随任务类别剧烈变化，规模收益有限且不一致，架构、tokenization 和预训练数据匹配经常比参数量更重要。
- 结果/证据
- 正式收录价值在于它不是又一个 genomics 模型，而是为生命科学 foundation model 建立诊断式评测框架。对 AI x biology 主线，它能帮助区分真正可迁移的基因组表示和只在局部任务上好看的模型。
- 收录价值
- 它不是更高一级，因为 GENEB 主要解决评测与模型选择问题，不直接提出新的生物发现工作流或模型训练范式。但作为 genomic model evaluation 的基准框架，它具有明确的长期参考价值。
原始摘要与中文对照
中文对照翻译
GENEB：为什么基因组模型难以比较。由于基准测试碎片化、评估协议不兼容以及任务特定报告，基因组基础模型的进展难以评估。因此，关于模型优越性或通用性的主张通常无法直接比较。我们引入了GENEB，这是一个大规模诊断基准，它在一个统一的探针式协议下（包括少样本机制），评估了来自40个基因组基础模型在涵盖13个功能类别的100项任务上的冻结表示。GENEB实现了在模型规模、架构、分词和预训练数据方面的受控比较，同时明确揭示了任务层面的权衡。我们的分析表明，综合排行榜不稳定：模型排名在不同任务类别之间差异显著，规模带来的增益微小且不一致，并且架构和预训练的对齐程度常常超过参数量。这些结果突出了当前评估实践的局限性，并将GENEB定位为基因组机器学习中进行原则性比较和类别感知模型选择的参考框架。图1. 基因组基础模型的碎片化比较格局。每个节点代表一个已发表的模型；有向边表示在相应论文中明确用作基线或比较器的模型。稀疏、不连通的图反映了基因组机器学习中缺乏统一的跨模型评估。这种碎片化使得即使是基本问题也难以回答。目前，对DNA-GPT (Zhang et al., 2023)、G ENOME O CEAN (Zhou et al., 2025) 和 E VO (Nguyen et al., 2024) 等广泛讨论的模型进行原则性比较是不可行的：每个模型都在不同的任务集、预处理流程和评估协议上进行评估。同一个模型有时在一个语境中被描述为重大突破，而在另一个语境中则表现不佳，这并非反映了矛盾的证据，而是缺乏一个共同的评估框架。
原始摘要
Progress in genomic foundation models is difficult to assess due to fragmented benchmarks, incompatible evaluation protocols, and task-specific reporting. As a result, claims of superiority or generality across models are often not directly comparable. We introduce GENEB, a large-scale diagnostic benchmark that evaluates frozen representations from 40 genomic foundation models across 100 tasks spanning 13 functional categories under a unified probing-based protocol, including few-shot regimes. GENEB enables controlled comparison across model scale, architecture, tokenization, and pretraining data while explicitly exposing task-level trade-offs. Our analysis shows that aggregate leaderboards are unstable: model rankings vary sharply across task categories, scale provides only modest and inconsistent gains, and architectural and pretraining alignment frequently outweigh parameter count. These results highlight limitations of current evaluation practices and position GENEB as a reference framework for principled comparison and category-aware model selection in genomic machine learning. Figure 1. Fragmented comparison landscape of genomic foundation models. Each node represents a published model; directed edges denote models explicitly used as baselines or comparators in the corresponding paper. The sparse, disconnected graph reflects the absence of unified cross-model evaluation in genomic machine learning. This fragmentation makes even basic questions difficult to answer. Principled comparison between widely discussed models such as DNA-GPT (Zhang et al., 2023), G ENOME O CEAN (Zhou et al., 2025), and E VO (Nguyen et al., 2024) is currently not feasible: each is evaluated on different task sets, preprocessing pipelines, and evaluation protocols. The same model is sometimes characterised as a major breakthrough in one context and as underperforming in another, reflecting not contradictory evidence but the absence of a common evaluation framework.