标题： 语音到文本的前缀或交叉注意力？ 经验比较 显示英文标题

标题： Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison

摘要： 在大型语言模型（LLMs）在自然语言处理（NLP）任务中取得显著成功之后，人们越来越有兴趣将它们的能力扩展到语音——这种最常用的交流形式。将语音集成到LLMs中最普遍的方法是密集特征前置（DFP），它将投影后的语音表示前置到文本表示之前，使得可以使用语音编码器进行端到端训练。这引发了关于DFP是否需要复杂的语音编码器以及其性能与标准编码器-解码器（即交叉注意力）架构相比如何的问题。我们在各种配置下比较了DFP和交叉注意力，例如CTC压缩、序列级知识蒸馏，在单语、双语和多语模型上进行测试。为了进行受控的架构比较，我们从头开始训练所有模型，而不是使用大型预训练模型，并使用可比的数据和参数设置，在MuST-C v1.0和CoVoST2数据集上测试语音到文本识别（ASR）和翻译（ST）。尽管DFP被广泛采用，但我们的结果并未表明DFP在交叉注意力上有明显优势。 显示英文摘要

摘要： Following the remarkable success of Large Language Models (LLMs) in NLP tasks, there is increasing interest in extending their capabilities to speech -- the most common form of communication. The most widespread approach to integrating speech into LLMs is dense feature prepending (DFP), which prepends the projected speech representations to the textual representations, allowing end-to-end training with a speech encoder. This raises questions about the need for a sophisticated speech encoder for DFP and how its performance compares with a standard encoder-decoder (i.e., cross-attention) architecture. We compare DFP and cross-attention under a variety of configurations, such as CTC compression, sequence-level knowledge distillation, on monolingual, bilingual, and multilingual models. To perform a controlled architectural comparison, we train all models from scratch rather than using large pretrained models and use comparable data and parameter settings, testing speech-to-text recognition (ASR) and translation (ST) on MuST-C v1.0 and CoVoST2 datasets. Despite the wide adoption of DFP, our results do not indicate a clear advantage of DFP over cross-attention.