标题： 通过强化行为对齐增强语音大语言模型 显示英文标题

标题： Enhancing Speech Large Language Models through Reinforced Behavior Alignment

摘要： 最近大型语言模型（LLMs）的进展激发了研究者们将它们的语言能力扩展到文本以外的其他模态的兴趣，这导致了具有处理用户请求能力的语音基础LLMs（SpeechLMs）的出现，这些模型可以以语音或文本格式处理用户请求。然而，由于模态间的差异，这些SpeechLMs在遵循指令方面与文本基础的LLM相比仍存在显著性能差距，尤其是在面对用户语音的动态和多变性质时。为了解决这一挑战，本文介绍了一个称为强化行为对齐（RBA）的框架，旨在增强SpeechLMs的语言生成能力。RBA不依赖于从人类标注中进行监督微调，而是采用一种自我合成方法，通过一个强大的教师LLM生成大量高保真对齐数据。然后使用基于强化学习的方法将SpeechLMs的行为与教师的行为对齐。实验结果表明，这种方法有效提升了SpeechLMs的指令遵循能力，其表现优于传统的蒸馏基线。关键的是，我们证明RBA可以无缝扩展到包括语音问答和语音到文本翻译在内的任务，在仅使用自生成数据的情况下，在开放基准测试中取得了最先进的性能。 显示英文摘要

摘要： The recent advancements of Large Language Models (LLMs) have spurred considerable research interest in extending their linguistic capabilities beyond text to other modalities, which leads to emergence of speech-based LLMs (SpeechLMs) with capability of processing user request in either speech or textual formats. However, owing to inter-modal discrepancies, these SpeechLMs still exhibit a significant performance gap compared to their text-based LLM counterparts in instruction-following, particularly when confronted with the dynamic and variable nature of user speech. To address this challenge, this paper introduces a framework termed Reinforced Behavior Alignment (RBA), designed to bolster the language generation proficiency of SpeechLMs. Instead of relying on supervised fine-tuning from human annotations, RBA employs a self-synthesis methodology to generate extensive, high-fidelity alignment data by a powerful teacher LLM. Then SpeechLMs is aligned its behavior with that of a teacher using a reinforcement learning-based approach. Experimental results demonstrate that this method effectively enhances the instruction-following capabilities of SpeechLMs that outperform conventional distillation baselines. Crucially, we demonstrate that RBA can be seamlessly extended to tasks such including spoken question answering and speech-to-text translation, attaining state-of-the-art performance on open benchmarks with only self-generated data.