标题： LLaVA-SP：通过视觉空间标记增强多模态大语言模型的视觉表示 显示英文标题

标题： LLaVA-SP: Enhancing Visual Representation with Visual Spatial Tokens for MLLMs

摘要： 多模态大语言模型（MLLMs）的架构通常将基于CLIP-ViT的视觉编码器与大语言模型连接起来。 虽然CLIP-ViT在捕捉全局图像特征方面表现良好，但它难以建模相邻块之间的局部关系，导致视觉表示较弱，进而影响MLLM的细节理解能力。 为了解决这个问题，我们提出了LLaVA-SP，它仅在原始视觉标记中添加六个空间视觉标记以增强视觉表示。 我们的方法具有三个关键优势：1）我们提出了一种新的投影器，使用卷积核从ViT块特征中推导出视觉空间标记，模拟两种视觉空间排序方法：“从中心区域到全局”和“从抽象到具体”。然后，应用交叉注意力机制融合细粒度的视觉信息，丰富整体视觉表示。2）我们提出了两种模型变体：LLaVA-SP-Cropping，通过逐步裁剪专注于细节特征，以及LLaVA-SP-Pooling，通过自适应池化捕捉全局语义，使模型能够处理多种视觉理解任务。3）大量实验表明，LLaVA-SP在LoRA微调后，在各种多模态基准测试中实现了显著的性能提升，在多个任务中超过了最先进的LLaVA-1.5模型，且推理延迟几乎相同。 代码和模型可在https://github.com/CnFaker/LLaVA-SP获取。 显示英文摘要

摘要： The architecture of multimodal large language models (MLLMs) commonly connects a vision encoder, often based on CLIP-ViT, to a large language model. While CLIP-ViT works well for capturing global image features, it struggles to model local relationships between adjacent patches, leading to weaker visual representation, which in turn affects the detailed understanding ability of MLLMs. To solve this, we propose LLaVA-SP, which only adds six spatial visual tokens to the original visual tokens to enhance the visual representation. Our approach offers three key advantages: 1) We propose a novel Projector, which uses convolutional kernels to derive visual spatial tokens from ViT patch features, simulating two visual spatial ordering approaches: "from central region to global" and "from abstract to specific". Then, a cross-attention mechanism is applied to fuse fine-grained visual information, enriching the overall visual representation. 2) We present two model variants: LLaVA-SP-Cropping, which focuses on detail features through progressive cropping, and LLaVA-SP-Pooling, which captures global semantics through adaptive pooling, enabling the model to handle diverse visual understanding tasks. 3) Extensive experiments show that LLaVA-SP, fine-tuned with LoRA, achieves significant performance improvements across various multimodal benchmarks, outperforming the state-of-the-art LLaVA-1.5 model in multiple tasks with nearly identical inference latency. The code and models are available at https://github.com/CnFaker/LLaVA-SP.