标题： 稀疏专家混合中的专家合并与纳什谈判 显示英文标题

标题： Expert Merging in Sparse Mixture of Experts with Nash Bargaining

摘要： 现有的稀疏专家混合（SMoE）专家融合策略通常依赖于输入相关或输入无关的专家参数平均，但往往缺乏一种有原则的加权机制。 在本工作中，我们通过博弈论的视角重新解释专家融合，揭示了专家之间的合作与竞争动态。 基于这一观点，我们引入了专家纳什融合（NAMEx），这是一种将纳什讨价还价纳入融合过程的新框架，使专家之间的协作更加平衡和高效。 此外，我们在NAMEx中引入了复杂的动量，以理论上保证收敛的方式加速专家传播。 在语言建模、文本分类、图像分类以及数据损坏下的零样本鲁棒性方面的广泛实验表明，NAMEx始终优于竞争方法，并能无缝集成到流行的MoE架构中。 最后，我们通过将其应用于大规模系统，包括Qwen1.5-MoE（14B）和DeepSeek-MoE（16B），展示了NAMEx的可扩展性，在零样本和微调设置中均证明了其有效性。 显示英文摘要

摘要： Existing expert merging strategies for Sparse Mixture of Experts (SMoE) typically rely on input-dependent or input-independent averaging of expert parameters, but often lack a principled weighting mechanism. In this work, we reinterpret expert merging through the lens of game theory, revealing cooperative and competitive dynamics among experts. Based on this perspective, we introduce Nash Merging of Experts (NAMEx), a novel framework that incorporates Nash Bargaining into the merging process, enabling more balanced and efficient collaboration among experts. Additionally, we incorporate complex momentum into NAMEx to accelerate expert propagation with theoretical guarantees for convergence. Extensive experiments across language modelling, text classification, image classification, and zero-shot robustness under data corruption show that NAMEx consistently outperforms competing methods while integrating seamlessly with popular MoE architectures. Finally, we demonstrate NAMEx's scalability by applying it to large-scale systems, including Qwen1.5-MoE (14B) and DeepSeek-MoE (16B), where it proves effective in both zero-shot and fine-tuning settings.