标题： 低秩适配器的语言建模多选学习 显示英文标题

标题： Multiple Choice Learning of Low Rank Adapters for Language Modeling

摘要： 我们提出LoRA-MCL，一种训练方案，该方案通过一种在推理时解码多样且合理的句子延续的方法，扩展了语言模型中的下一个标记预测。 传统的语言建模是一个本质上不适定的问题：给定一个上下文，可能有多个未来同样合理。 我们的方法利用多选学习（MCL）和胜者通吃（WTA）损失，通过低秩适应（LoRA）高效处理歧义。 我们提供了将多选学习应用于语言建模的理论解释，假设数据是从分布混合中生成的。 为了说明所提出的方法，我们使用从马尔可夫链混合中采样的数据。 然后，我们在现实世界的视觉和音频字幕任务上进行了大量实验，证明我们的方法在生成输出中实现了高多样性和相关性。 显示英文摘要

摘要： We propose LoRA-MCL, a training scheme that extends next-token prediction in language models with a method designed to decode diverse, plausible sentence continuations at inference time. Traditional language modeling is an intrinsically ill-posed problem: given a context, multiple futures may be equally plausible. Our approach leverages Multiple Choice Learning (MCL) and the Winner-Takes-All (WTA) loss to efficiently handle ambiguity through Low-Rank Adaptation (LoRA). We provide a theoretical interpretation of applying Multiple Choice Learning to Language Modeling, assuming the data is generated from a mixture of distributions. To illustrate the proposed approach, we use data sampled from mixtures of Markov chains. We then demonstrate with extensive experiments on real-world visual and audio captioning tasks that our method achieves high diversity and relevance in generated outputs.