标题： 多任务学习中通过任务自适应低秩表示的知识高效迁移 显示英文标题

标题： Efficient Knowledge Transfer in Multi-Task Learning through Task-Adaptive Low-Rank Representation

摘要： 预训练语言模型（PLMs）展示了显著的智能，但在实际应用中面对训练期间未见过的新任务时表现不佳。为每个新任务单独训练模型通常是不切实际的。多任务学习（MTL）通过从源任务向目标任务转移共享知识来解决这一挑战。作为一种主要的参数高效微调方法，提示调优（PT）通过引入一个可适应的向量来增强MTL，该向量捕获任务特定的知识，作为原始提示的前缀以保留共享知识，同时保持PLM参数冻结。然而，由于表示能力有限，PT难以有效捕捉任务特定知识的异质性。为了解决这一挑战，我们提出了基于PT的任务自适应低秩表示（TA-LoRA），这是一种构建在PT上的MTL方法，采用低秩表示来建模任务异质性，并使用快速-慢速权重机制，其中慢速权重编码共享知识，而快速权重捕捉任务特定的细微差别，避免了由从头开始训练低秩表示引起的共享和任务特定知识混合。此外，引入了一个零初始化注意力机制，在预热时期内最小化不成熟的低秩组件对原始提示的干扰。在16个任务上的实验表明，TA-LoRA在全数据和少量样本设置下达到了最先进的性能，同时保持了优越的参数效率。 显示英文摘要

摘要： Pre-trained language models (PLMs) demonstrate remarkable intelligence but struggle with emerging tasks unseen during training in real-world applications. Training separate models for each new task is usually impractical. Multi-task learning (MTL) addresses this challenge by transferring shared knowledge from source tasks to target tasks. As an dominant parameter-efficient fine-tuning method, prompt tuning (PT) enhances MTL by introducing an adaptable vector that captures task-specific knowledge, which acts as a prefix to the original prompt that preserves shared knowledge, while keeping PLM parameters frozen. However, PT struggles to effectively capture the heterogeneity of task-specific knowledge due to its limited representational capacity. To address this challenge, we propose Task-Adaptive Low-Rank Representation (TA-LoRA), an MTL method built on PT, employing the low-rank representation to model task heterogeneity and a fast-slow weights mechanism where the slow weight encodes shared knowledge, while the fast weight captures task-specific nuances, avoiding the mixing of shared and task-specific knowledge, caused by training low-rank representations from scratch. Moreover, a zero-initialized attention mechanism is introduced to minimize the disruption of immature low-rank components on original prompts during warm-up epochs. Experiments on 16 tasks demonstrate that TA-LoRA achieves state-of-the-art performance in full-data and few-shot settings while maintaining superior parameter efficiency.