标题： 通过视觉-语言模型适应开放数据的主动学习 显示英文标题

标题： Active Learning via Vision-Language Model Adaptation with Open Data

摘要： 在大规模开放数据上预训练的视觉语言模型（VLM），在经过任务特定标注数据调整后，能够提供强大的下游任务解决能力。然而，数据标注可能昂贵且需要领域专业知识。 主动学习（AL）旨在通过战略性地选择最具信息量的数据进行标注和模型训练来减少这一开销。 最近的主动学习方法已经探索了VLM，但尚未利用公开可用的开放数据，例如VLM的预训练数据。 在这项工作中，我们通过检索与任务相关的示例来扩充特定任务的示例，从而利用此类数据。 正如预期的那样，将它们纳入显著改善了主动学习。 鉴于我们的方法利用了开源VLM和开放数据，我们将其称为基于开放资源的主动学习（ALOR）。 此外，大多数基于VLM的主动学习方法使用提示调优（PT）进行模型适配，这可能是由于其能够直接利用预训练参数，并假设这样做可以降低过度拟合有限标注数据的风险。 我们严格比较了流行适配方法，包括线性探测（LP）、微调（FT）和对比调优（CT）。 我们揭示了两个关键发现：（1）所有适配方法都从纳入检索数据中受益，（2）在各种主动学习方法中，对比调优（CT）明显优于其他方法。 对检索数据的进一步分析揭示了任务相关类别分布的自然不平衡，暴露了VLM中的固有偏见。 这促使我们提出了新颖的尾部优先采样（TFS）策略，这是一种令人尴尬地简单但有效的主动学习方法，优先采样代表性不足类别的数据进行标注。 广泛的实验表明，我们最终的方法——对比微调检索到的和TFS选择的标注数据上的VLM——显著优于现有方法。 显示英文摘要

摘要： Pretrained on web-scale open data, VLMs offer powerful capabilities for solving downstream tasks after being adapted to task-specific labeled data. Yet, data labeling can be expensive and may demand domain expertise. Active Learning (AL) aims to reduce this expense by strategically selecting the most informative data for labeling and model training. Recent AL methods have explored VLMs but have not leveraged publicly available open data, such as VLM's pretraining data. In this work, we leverage such data by retrieving task-relevant examples to augment the task-specific examples. As expected, incorporating them significantly improves AL. Given that our method exploits open-source VLM and open data, we refer to it as Active Learning with Open Resources (ALOR). Additionally, most VLM-based AL methods use prompt tuning (PT) for model adaptation, likely due to its ability to directly utilize pretrained parameters and the assumption that doing so reduces the risk of overfitting to limited labeled data. We rigorously compare popular adaptation approaches, including linear probing (LP), finetuning (FT), and contrastive tuning (CT). We reveal two key findings: (1) All adaptation approaches benefit from incorporating retrieved data, and (2) CT resoundingly outperforms other approaches across AL methods. Further analysis of retrieved data reveals a naturally imbalanced distribution of task-relevant classes, exposing inherent biases within the VLM. This motivates our novel Tail First Sampling (TFS) strategy for AL, an embarrassingly simple yet effective method that prioritizes sampling data from underrepresented classes to label. Extensive experiments demonstrate that our final method, contrastively finetuning VLM on both retrieved and TFS-selected labeled data, significantly outperforms existing methods.