标题： 语言解锁的ViT（LUViT）：利用大语言模型增强自监督视觉变换器 显示英文标题

标题： Language-Unlocked ViT (LUViT): Empowering Self-Supervised Vision Transformers with LLMs

摘要： 将大型语言模型（LLMs）模块与视觉变换器（ViTs）相结合，在利用LLMs丰富的语义知识和推理能力的基础上，为仅视觉任务带来了巨大的前景。 然而，一个基本的挑战在于LLMs以文本为中心的预训练和ViTs以视觉为中心的训练之间的固有模态不匹配。 直接融合往往无法充分发挥LLMs的潜力，并且在微调过程中表现出不稳定性。 因此，LLM模块保持冻结状态，而仅学习视觉组件。 为了解决这些挑战，我们引入了语言解锁视觉变换器（LUViT），这是一种通过协同预训练策略来弥合这种模态不匹配的新方法。 LUViT通过（1）使用掩码自动编码（MAE）对ViT进行预训练以获得更丰富的视觉表示，以及（2）同时使用MAE目标在LLM模块内训练低秩适应（LoRA）层，来共同适应ViT主干和LLM融合模块。 这种联合优化引导ViT生成与LLM对齐的特征，并使LLM能够有效解释视觉信息。 我们通过大量实验证明，LUViT在各种下游视觉任务中显著提升了性能，展示了更有效和高效的途径来利用LLM知识进行视觉理解。 显示英文摘要

摘要： The integration of Large Language Model (LLMs) blocks with Vision Transformers (ViTs) holds immense promise for vision-only tasks by leveraging the rich semantic knowledge and reasoning capabilities of LLMs. However, a fundamental challenge lies in the inherent modality mismatch between text-centric pretraining of LLMs and vision-centric training of ViTs. Direct fusion often fails to fully exploit the LLM's potential and suffers from unstable finetuning. As a result, LLM blocks are kept frozen while only the vision components are learned. As a remedy to these challenges, we introduce Language-Unlocked Vision Transformers (LUViT), a novel approach that bridges this modality mismatch through a synergistic pre-training strategy. LUViT co-adapts a ViT backbone and an LLM fusion block by (1) employing Masked Auto-Encoding (MAE) to pre-train the ViT for richer visual representations, and (2) concurrently training Low-Rank Adaptation (LoRA) layers within the LLM block using the MAE objective. This joint optimization guides the ViT to produce LLM-aligned features and the LLM to effectively interpret visual information. We demonstrate through extensive experiments that LUViT significantly improves performance on various downstream vision tasks, showcasing a more effective and efficient pathway to harness LLM knowledge for visual understanding.