标题： 群体专家混合用于学习不变表示 显示英文标题

标题： Mixture of Group Experts for Learning Invariant Representations

摘要： 稀疏激活的专家混合（MoE）模型在保持每个标记的计算成本一致的同时，有效增加了参数数量。 然而，传统的MoE模型通常在专家之间存在有限的多样性和专业化，这限制了它们的性能和可扩展性，尤其是在专家数量增加时。 在本文中，我们提出了一种新的视角来看待传统的MoE模型，其基于由稀疏表示启发的top-$k$路由。 这使我们能够将稀疏表示中的已建立理论见解引入MoE模型。 在此基础上，我们提出了一种针对top-$k$路由输入的群体稀疏正则化方法，称为专家群体混合（MoGE）。 MoGE通过在路由输入上施加结构约束来间接对专家进行正则化，同时保留原始MoE架构。 此外，我们将路由输入组织成一个二维拓扑图，空间上将相邻元素分组。 这种结构使MoGE能够捕捉对微小变换不变的表示，从而显著增强专家的多样性和专业化。 在图像分类和语言建模任务的各种Transformer模型上的全面评估表明，MoGE在几乎没有额外内存和计算开销的情况下显著优于其MoE对应模型。 我们的方法提供了一种简单而有效的解决方案，以扩大专家数量并减少它们之间的冗余。 源代码包含在补充材料中，并将公开发布。 显示英文摘要

摘要： Sparsely activated Mixture-of-Experts (MoE) models effectively increase the number of parameters while maintaining consistent computational costs per token. However, vanilla MoE models often suffer from limited diversity and specialization among experts, constraining their performance and scalability, especially as the number of experts increases. In this paper, we present a novel perspective on vanilla MoE with top-$k$ routing inspired by sparse representation. This allows us to bridge established theoretical insights from sparse representation into MoE models. Building on this foundation, we propose a group sparse regularization approach for the input of top-$k$ routing, termed Mixture of Group Experts (MoGE). MoGE indirectly regularizes experts by imposing structural constraints on the routing inputs, while preserving the original MoE architecture. Furthermore, we organize the routing input into a 2D topographic map, spatially grouping neighboring elements. This structure enables MoGE to capture representations invariant to minor transformations, thereby significantly enhancing expert diversity and specialization. Comprehensive evaluations across various Transformer models for image classification and language modeling tasks demonstrate that MoGE substantially outperforms its MoE counterpart, with minimal additional memory and computation overhead. Our approach provides a simple yet effective solution to scale the number of experts and reduce redundancy among them. The source code is included in the supplementary material and will be publicly released.