标题： MedFocusCLIP：使用像素级注意力在医学数据集中改进少样本分类 显示英文标题

标题： MedFocusCLIP : Improving few shot classification in medical datasets using pixel wise attention

摘要： 随着基础模型的普及，参数高效的微调已成为利用预训练模型执行下游任务的默认方法。 受到大型语言模型、视觉提示微调和类似技术的最新进展的启发，学习一个额外的提示，以高效地微调预训练的视觉基础模型。 然而，我们观察到，这种提示对于细粒度的视觉分类任务（如医学图像分类）来说是不够的，因为在这些任务中存在较大的类间差异和较小的类内差异。 因此，在本文中，我们提出利用Segment Anything Model 2 (SAM2)先进的分割能力作为视觉提示线索，通过引导CLIP（对比语言-图像预训练）视觉编码器中的注意力到图像中的相关区域，来帮助CLIP视觉编码器。 这有助于模型专注于高度区分性的区域，而不会被视觉相似的背景特征分散注意力，这是在少样本、细粒度分类设置中的基本要求。 我们在包括X射线、CT扫描和MRI图像在内的多种医学数据集上评估了我们的方法，并报告了在（COVID、肺部疾病、脑肿瘤、乳腺癌）数据集上，所提出方法的准确率分别为（71%、81%、86%、58%），而经过少样本训练的预训练CLIP模型的准确率分别为（66%、70%、68%、29%）。 所提出的方法还允许通过使用分割获得的定位信息，得到可解释的分类性能解释。 显示英文摘要

摘要： With the popularity of foundational models, parameter efficient fine tuning has become the defacto approach to leverage pretrained models to perform downstream tasks. Taking inspiration from recent advances in large language models, Visual Prompt Tuning, and similar techniques, learn an additional prompt to efficiently finetune a pretrained vision foundational model. However, we observe that such prompting is insufficient for fine-grained visual classification tasks such as medical image classification, where there is large inter-class variance, and small intra-class variance. Hence, in this paper we propose to leverage advanced segmentation capabilities of Segment Anything Model 2 (SAM2) as a visual prompting cue to help visual encoder in the CLIP (Contrastive Language-Image Pretraining) by guiding the attention in CLIP visual encoder to relevant regions in the image. This helps the model to focus on highly discriminative regions, without getting distracted from visually similar background features, an essential requirement in a fewshot, finegrained classification setting. We evaluate our method on diverse medical datasets including X-rays, CT scans, and MRI images, and report an accuracy of (71%, 81%, 86%, 58%) from the proposed approach on (COVID, lung-disease, brain-tumor, breast-cancer) datasets against (66%, 70%, 68%, 29%) from a pretrained CLIP model after fewshot training. The proposed approach also allows to obtain interpretable explanation for the classification performance through the localization obtained using segmentation.