标题： 通过权重集成专家混合合并多任务模型 显示英文标题

标题： Merging Multi-Task Models via Weight-Ensembling Mixture of Experts

摘要： 将不同任务上训练的多种基于Transformer的任务专用模型合并为一个统一模型，可以同时执行所有任务。 以前的方法，如任务算术，已被证明是有效且可扩展的。 现有方法主要集中在原始模型参数空间中寻找静态最优解。 一个显著的挑战是减轻不同模型参数之间的干扰，这会显著降低性能。 在本文中，我们提出在扩大Transformer层的MLP为权重集成专家（MoE）模块的同时，合并大部分参数，该模块可以根据输入动态整合共享知识和任务特定知识，从而提供一种更灵活的解决方案，能够适应每个实例的具体需求。 我们的关键见解是，通过识别和分离共享知识和任务特定知识，然后动态整合它们，可以大大缓解参数干扰问题。 我们进行了传统的多任务模型合并实验，并评估了我们方法的泛化能力和鲁棒性。 结果证明了我们方法的有效性，并提供了对我们方法的全面理解。 代码可在 https://anonymous.4open.science/r/weight-ensembling_MoE-67C9/ 获取。 显示英文摘要

摘要： Merging various task-specific Transformer-based models trained on different tasks into a single unified model can execute all the tasks concurrently. Previous methods, exemplified by task arithmetic, have been proven to be both effective and scalable. Existing methods have primarily focused on seeking a static optimal solution within the original model parameter space. A notable challenge is mitigating the interference between parameters of different models, which can substantially deteriorate performance. In this paper, we propose to merge most of the parameters while upscaling the MLP of the Transformer layers to a weight-ensembling mixture of experts (MoE) module, which can dynamically integrate shared and task-specific knowledge based on the input, thereby providing a more flexible solution that can adapt to the specific needs of each instance. Our key insight is that by identifying and separating shared knowledge and task-specific knowledge, and then dynamically integrating them, we can mitigate the parameter interference problem to a great extent. We conduct the conventional multi-task model merging experiments and evaluate the generalization and robustness of our method. The results demonstrate the effectiveness of our method and provide a comprehensive understanding of our method. The code is available at https://anonymous.4open.science/r/weight-ensembling_MoE-67C9/