标题： 无损词汇缩减用于自回归语言模型 显示英文标题

标题： Lossless Vocabulary Reduction for Auto-Regressive Language Models

摘要： 分词——将给定文本分解为称为标记的子词序列的过程——是语言模型开发中的关键组件之一。 特别是，自回归语言模型逐个标记生成文本，即通过预测给定前面标记的下一个标记分布来生成，因此分词直接影响它们在文本生成中的效率。 由于每个语言模型都有自己的词汇表作为可能标记的集合，它们在下一个标记分布层面（如模型集成）难以相互协作。 在本文中，我们建立了一个无损词汇缩减的理论框架，该框架可以高效地将给定的自回归语言模型转换为具有任意小词汇表的模型，而不会损失任何准确性。 作为应用，我们证明了具有不同分词方式的语言模型可以通过它们的最大公共词汇表高效地相互协作。 显示英文摘要

摘要： Tokenization -- the process of decomposing a given text into a sequence of subwords called tokens -- is one of the key components in the development of language models. Particularly, auto-regressive language models generate texts token by token, i.e., by predicting the next-token distribution given the previous ones, and thus tokenization directly affects their efficiency in text generation. Since each language model has their own vocabulary as a set of possible tokens, they struggle to cooperate with each other at the level of next-token distributions such as model ensemble. In this paper, we establish a theoretical framework of lossless vocabulary reduction, which efficiently converts a given auto-regressive language model into the one with an arbitrarily small vocabulary without any loss in accuracy. As an application, we demonstrate that language models with different tokenization can cooperate with each other efficiently through their maximal common vocabulary.