标题： 基于指令的视觉投影器用于生成式视觉-语言模型的持续学习 显示英文标题

标题： Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Models

摘要： 持续学习使预训练的生成视觉-语言模型（VLMs）能够在不使用以前的再训练数据的情况下，将新任务的知识纳入其中。最近的方法更新了视觉投影器，以将视觉信息翻译为新任务，将预训练的视觉编码器与大型语言模型连接起来。然而，这样的调整可能导致模型更重视视觉输入而不是语言指令，特别是在学习具有重复类型文本指令的任务时。为了解决对语言指令的忽视，我们提出了一种新的框架，该框架基于语言模型的指令来实现视觉信息的翻译。我们引入了视觉投影器的混合，每个投影器根据给定的指令上下文作为专门的视觉到语言翻译专家，以适应新任务。为了避免在不相关的指令上下文中使用专家，我们提出了一个专家推荐策略，该策略重新使用之前学习过的类似任务的专家。此外，我们引入了专家剪枝，以减轻在先前任务中累积激活的专家使用带来的干扰。在多种视觉-语言任务上的广泛实验表明，我们的方法通过生成遵循指令的响应，优于现有的持续学习方法。 显示英文摘要

摘要： Continual learning enables pre-trained generative vision-language models (VLMs) to incorporate knowledge from new tasks without retraining data from previous ones. Recent methods update a visual projector to translate visual information for new tasks, connecting pre-trained vision encoders with large language models. However, such adjustments may cause the models to prioritize visual inputs over language instructions, particularly learning tasks with repetitive types of textual instructions. To address the neglect of language instructions, we propose a novel framework that grounds the translation of visual information on instructions for language models. We introduce a mixture of visual projectors, each serving as a specialized visual-to-language translation expert based on the given instruction context to adapt to new tasks. To avoid using experts for irrelevant instruction contexts, we propose an expert recommendation strategy that reuses experts for tasks similar to those previously learned. Additionally, we introduce expert pruning to alleviate interference from the use of experts that cumulatively activated in previous tasks. Extensive experiments on diverse vision-language tasks demonstrate that our method outperforms existing continual learning approaches by generating instruction-following responses.