Abstract
Tokenization -- the process of decomposing a given text into a sequence of subwords called tokens -- is one of the key components in the development of language models. Particularly, auto-regressive language models generate texts token by token, i.e., by predicting the next-token distribution given the previous ones, and thus tokenization directly affects their efficiency in text generation. Since each language model has their own vocabulary as a set of possible tokens, they struggle to cooperate with each other at the level of next-token distributions such as model ensemble. In this paper, we establish a theoretical framework of lossless vocabulary reduction, which efficiently converts a given auto-regressive language model into the one with an arbitrarily small vocabulary without any loss in accuracy. As an application, we demonstrate that language models with different tokenization can cooperate with each other efficiently through their maximal common vocabulary.
Abstract (translated)
令牌化——即将给定的文本分解为一系列称为标记的子词的过程——是语言模型开发的关键组成部分。特别是，自回归语言模型通过逐个标记生成文本，即根据之前的标记预测下一个标记的概率分布，因此，令牌化直接影响它们在文本生成中的效率。由于每个语言模型都有其独特的词汇集作为可能的标记集合，它们难以在其概率分布层面（如模型集成）上相互合作。本文中，我们建立了一个无损词汇缩减的理论框架，该框架可以高效地将给定自回归语言模型转换为具有任意小词汇量的形式，而不会牺牲准确性。作为一种应用，我们展示了不同令牌化方法的语言模型可以通过它们的最大公共词汇库有效地进行协作。 这段翻译解释了令牌化的概念和其在语言模型中的重要性，并概述了一种新的理论框架，该框架能够将自回归语言模型转换为具有更小词汇量的形式而不损失精度。此外，还描述了一个具体应用案例，即展示不同令牌化方法的语言模型如何通过共享的最大公共词汇进行协作。
URL
https://arxiv.org/abs/2510.08102
