收录解读
子词分词隐藏了字符级信息,纯字节模型又往往在性能与效率上落后。论文研究能否复用已有语言模型而不是从头训练字节模型。
byteification通过两阶段训练将子词模型改造为字节输入/输出模型,重用源模型内部能力并设计高效推理路径;作者验证Olmo、Qwen3和Llama三类模型及转换预算。
两阶段byteification以49.1B额外token改造Olmo/Qwen3/Llama三模型家族,低于通常预训练预算1%;表1平均能力接近子词源模型(Bwen69.8对70.3),字符推理明显提升,提供可复用字节级改造与推理架构。
未达到paradigm:未充分验证前沿大规模与广泛多语言场景,代码pass@1/数学仍有退化,不能视作通用分词替代。
原始摘要
Abstract Recent advances in artificial intelligence (AI) have largely been driven by large language models, deep neural networks that operate over discrete units called tokens. To represent text, most large language models use words or word fragments as the tokens, known as subword tokenization 1 . Subword tokenization obscures fine-grained information, which is problematic, especially for scientific data—such as computer code or biological sequences—where meaning depends on the individual characters or bytes 2 . Models that instead operate directly on the byte encoding of text avoid these limitations, but until now they have lagged behind subword-based models in performance. Here we introduce a general method for creating byte-level large language models through byteification that approach the capabilities of subword-based systems. We use a two-stage conversion procedure to retrofit existing subword-based models into byte-level models with minimal extra training. The resulting models outperform earlier byte-level approaches and excel on character-level reasoning tasks, achieving practical inference speeds by efficiently processing byte-level information and adaptability by reusing the existing ecosystem around the source large language model. Our results remove a long-standing performance barrier to end-to-end byte-level language modelling, demonstrating that models operating on raw text encodings can scale competitively while offering advantages in domains requiring fine-grained textual understanding.