标题： 3-in-1：用于高效微调、高效批处理和可组合性的二维旋转适应 显示英文标题

标题： 3-in-1: 2D Rotary Adaptation for Efficient Finetuning, Efficient Batching and Composability

摘要： 参数高效的微调（PEFT）方法能够有效地将大型语言模型（LLMs）适配到各种下游任务中，同时减少存储和GPU内存需求。尽管有这些优势，一些应用场景对PEFT提出了超越单纯参数效率的新挑战。一个显著的挑战是如何高效部署配备了多个任务或用户专用适配器的LLMs，特别是在同一批次内不同请求需要不同适配器的情况下。另一个挑战是LLMs的可解释性，这对于理解LLMs的工作原理至关重要。先前的研究引入了多种方法来应对不同的挑战。本文介绍了一种新颖的方法RoAd，它通过简单的二维旋转来适配LLMs，并解决了上述所有挑战：（1）RoAd具有极高的参数效率，在GLUE基准、八个常识推理任务以及四个算术推理任务上，使用$<0.1\%$个可训练参数实现了最优性能；（2）RoAd可以高效处理批量请求中需要不同适配器的情况，其开销相当于逐元素乘法而不是批量矩阵乘法；（3）RoAd通过集成到分布式互换干预框架内增强了LLMs的可解释性，并通过组成实验进行了验证。 显示英文摘要

摘要： Parameter-efficient finetuning (PEFT) methods effectively adapt large language models (LLMs) to diverse downstream tasks, reducing storage and GPU memory demands. Despite these advantages, several applications pose new challenges to PEFT beyond mere parameter efficiency. One notable challenge involves the efficient deployment of LLMs equipped with multiple task- or user-specific adapters, particularly when different adapters are needed for distinct requests within the same batch. Another challenge is the interpretability of LLMs, which is crucial for understanding how LLMs function. Previous studies introduced various approaches to address different challenges. In this paper, we introduce a novel method, RoAd, which employs a straightforward 2D rotation to adapt LLMs and addresses all the above challenges: (1) RoAd is remarkably parameter-efficient, delivering optimal performance on GLUE, eight commonsense reasoning tasks and four arithmetic reasoning tasks with $<0.1\%$ trainable parameters; (2) RoAd facilitates the efficient serving of requests requiring different adapters within a batch, with an overhead comparable to element-wise multiplication instead of batch matrix multiplication; (3) RoAd enhances LLM's interpretability through integration within a framework of distributed interchange intervention, demonstrated via composition experiments.