标题： Mixpert：通过高效视觉专家混合缓解多模态学习冲突 显示英文标题

标题： Mixpert: Mitigating Multimodal Learning Conflicts with Efficient Mixture-of-Vision-Experts

摘要： 多模态大型语言模型（MLLMs）需要对复杂的图像信息进行细致的解读，通常通过视觉编码器来感知各种视觉场景。 然而，仅仅依赖单一的视觉编码器来处理多样化的任务领域既困难重重，又不可避免地导致冲突。 近期的研究通过直接集成多个领域特定的视觉编码器来增强数据感知，但这种结构增加了复杂性，并限制了联合优化的潜力。 本文提出Mixpert，这是一种高效的视觉专家混合架构，它继承了单一视觉编码器的联合学习优势，同时重新构建为一个多专家范式，以针对不同视觉任务进行特定微调。 此外，我们设计了一种动态路由机制，将输入图像分配给最合适的视觉专家。 Mixpert有效地缓解了单一视觉编码器在多任务学习中遇到的领域冲突，且额外计算成本极低，比多个编码器更为高效。 此外，Mixpert可以无缝集成到任何MLLM中，实验结果表明其在各种任务中都取得了显著的性能提升。 显示英文摘要

摘要： Multimodal large language models (MLLMs) require a nuanced interpretation of complex image information, typically leveraging a vision encoder to perceive various visual scenarios. However, relying solely on a single vision encoder to handle diverse task domains proves difficult and inevitably leads to conflicts. Recent work enhances data perception by directly integrating multiple domain-specific vision encoders, yet this structure adds complexity and limits the potential for joint optimization. In this paper, we introduce Mixpert, an efficient mixture-of-vision-experts architecture that inherits the joint learning advantages from a single vision encoder while being restructured into a multi-expert paradigm for task-specific fine-tuning across different visual tasks. Additionally, we design a dynamic routing mechanism that allocates input images to the most suitable visual expert. Mixpert effectively alleviates domain conflicts encountered by a single vision encoder in multi-task learning with minimal additional computational cost, making it more efficient than multiple encoders. Furthermore, Mixpert integrates seamlessly into any MLLM, with experimental results demonstrating substantial performance gains across various tasks.