标题： 预训练模型中令牌合并的影响 显示英文标题

标题： On the Effect of Token Merging on Pre-trained Models for Code

摘要： 分词是代码语言模型的基本组成部分。 它涉及将输入分解为单元，这些单元随后传递给语言模型堆栈，以学习在各种上下文中使用的高维表示，从分类到生成。 然而，这些分词器的输出通常比传统编译器和解释器中使用的更长。 这可能导致不希望的效果，例如增加计算开销。 在本工作中，我们研究了合并属于同一语义单元的子标记的隐藏表示的影响，例如形成单个标识符的子标记。 我们提出了两种策略：一种基于对表示进行平均，另一种则利用基于学习的方法。 这两种方法都可以无缝集成到现有的代码语言模型中。 我们使用六种代码语言模型进行了实验：CodeBERT、GraphCodeBERT、UniXCoder、CdoeT5、CodeT5+（220M）和CodeT5+（770M），在三个软件工程任务中进行：漏洞检测、代码分类和代码翻译。 结果表明，这些策略可以将浮点运算次数减少$1\%$到$19\%$。 关于下游性能，最显著的退化出现在漏洞检测任务中，与基线相比，F1分数下降了$1.82$个百分点。 相比之下，在代码翻译中，CodeBLEU提高了$2.47$个百分点。 这项工作有助于在多个维度上改进代码语言模型的广泛努力，包括计算效率和下游性能。 显示英文摘要

摘要： Tokenization is a fundamental component of language models for code. It involves breaking down the input into units that are later passed to the language model stack to learn high-dimensional representations used in various contexts, from classification to generation. However, the output of these tokenizers is often longer than that traditionally used in compilers and interpreters. This could result in undesirable effects, such as increased computational overhead. In this work, we investigate the effect of merging the hidden representations of subtokens that belong to the same semantic unit, such as subtokens that form a single identifier. We propose two strategies: one based on averaging the representations and another that leverages a learning-based approach. Both methods can be seamlessly integrated with existing language models for code. We conduct experiments using six language models for code: CodeBERT, GraphCodeBERT, UniXCoder, CdoeT5, CodeT5+ (220M), and CodeT5+ (770M), across three software engineering tasks: vulnerability detection, code classification, and code translation. Results show that these strategies can reduce the number of floating-point operations by $1\%$ to $19\%$. Regarding downstream performance, the most significant degradation was observed in the vulnerability detection task, where the F1 score decreased by $1.82$ points compared to the baseline. In contrast, for code translation, we observed an improvement of $2.47$ points in CodeBLEU. This work contributes to the broader effort of improving language models for code across multiple dimensions, including both computational efficiency and downstream performance.