标题： 视觉与语言参考提示用于少样本分割 显示英文标题

标题： Vision and Language Reference Prompt into SAM for Few-shot Segmentation

摘要： 分割一切模型（SAM）代表一个大规模的分割模型，能够通过灵活的提示实现强大的零样本能力。 虽然SAM可以在零样本情况下分割任何对象，但它需要用户为每个目标图像提供提示，并且不会将任何标签信息附加到掩码上。 少样本分割模型通过输入带注释的参考图像作为SAM的提示来解决这些问题，并且可以在没有用户提供提示的情况下分割目标图像中的特定对象。 以前基于SAM的少样本分割模型仅使用带注释的参考图像作为提示，由于缺乏参考信息导致准确性有限。 在本文中，我们提出了一种新的少样本分割模型，视觉和语言参考提示进入SAM（VLP-SAM），该模型通过不仅输入图像还输入语言作为参考信息，利用参考图像的视觉信息和文本标签的语义信息。 特别是，VLP-SAM是一个简单且可扩展的结构，具有最小的学习参数，它使用多模态视觉语言模型将包含视觉语言信息的提示嵌入输入到SAM中。 为了证明VLP-SAM的有效性，我们在PASCAL-5i和COCO-20i数据集上进行了实验，并在少样本分割任务中取得了高性能，分别以6.3%和9.5%的mIoU大幅超越了之前的最先进模型。 此外，VLP-SAM展示了其在未见对象上的泛化能力，这些对象未包含在训练数据中。 我们的代码可在https://github.com/kosukesakurai1/VLP-SAM获取。 显示英文摘要

摘要： Segment Anything Model (SAM) represents a large-scale segmentation model that enables powerful zero-shot capabilities with flexible prompts. While SAM can segment any object in zero-shot, it requires user-provided prompts for each target image and does not attach any label information to masks. Few-shot segmentation models addressed these issues by inputting annotated reference images as prompts to SAM and can segment specific objects in target images without user-provided prompts. Previous SAM-based few-shot segmentation models only use annotated reference images as prompts, resulting in limited accuracy due to a lack of reference information. In this paper, we propose a novel few-shot segmentation model, Vision and Language reference Prompt into SAM (VLP-SAM), that utilizes the visual information of the reference images and the semantic information of the text labels by inputting not only images but also language as reference information. In particular, VLP-SAM is a simple and scalable structure with minimal learnable parameters, which inputs prompt embeddings with vision-language information into SAM using a multimodal vision-language model. To demonstrate the effectiveness of VLP-SAM, we conducted experiments on the PASCAL-5i and COCO-20i datasets, and achieved high performance in the few-shot segmentation task, outperforming the previous state-of-the-art model by a large margin (6.3% and 9.5% in mIoU, respectively). Furthermore, VLP-SAM demonstrates its generality in unseen objects that are not included in the training data. Our code is available at https://github.com/kosukesakurai1/VLP-SAM.