标题： 进步的幻觉？ 对视觉-语言模型测试时适应的批判性审视 显示英文标题

标题： The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models

摘要： 测试时适应（TTA）方法在增强视觉-语言模型（VLMs）如CLIP在推理期间的性能方面引起了广泛关注，而无需额外的标记数据。 然而，当前的TTA研究通常存在重大限制，例如基准结果的重复、评估指标有限、实验设置不一致和分析不足。 这些问题阻碍了TTA方法之间的公平比较，并模糊了它们的实际优势和劣势。 为了解决这些挑战，我们引入了TTA-VLM，这是一个全面的基准，用于评估VLM上的TTA方法。 我们的基准在一个统一且可复现的框架内实现了8种周期性TTA和7种在线TTA方法，并在15个广泛使用的数据集上对其进行评估。 与之前仅专注于CLIP的研究不同，我们将评估扩展到SigLIP——一个使用Sigmoid损失训练的模型，并包括训练时微调方法如CoOp、MaPLe和TeCoA以评估通用性。 除了分类准确率外，TTA-VLM还结合了各种评估指标，包括鲁棒性、校准、分布外检测和稳定性，从而能够更全面地评估TTA方法。 通过广泛的实验，我们发现1）现有的TTA方法与之前的开创性工作相比产生的增益有限；2）当前的TTA方法与训练时微调方法表现出较差的协作性；3）准确率的提升往往以降低模型可信度为代价。 我们发布了TTA-VLM，以提供VLM上TTA方法的公平比较和全面评估，并希望它能鼓励社区开发更多可靠和通用的TTA策略。 显示英文摘要

摘要： Test-time adaptation (TTA) methods have gained significant attention for enhancing the performance of vision-language models (VLMs) such as CLIP during inference, without requiring additional labeled data. However, current TTA researches generally suffer from major limitations such as duplication of baseline results, limited evaluation metrics, inconsistent experimental settings, and insufficient analysis. These problems hinder fair comparisons between TTA methods and obscure their practical strengths and weaknesses. To address these challenges, we introduce TTA-VLM, a comprehensive benchmark for evaluating TTA methods on VLMs. Our benchmark implements 8 episodic TTA and 7 online TTA methods within a unified and reproducible framework, and evaluates them across 15 widely used datasets. Unlike prior studies focused solely on CLIP, we extend the evaluation to SigLIP--a model trained with a Sigmoid loss--and include training-time tuning methods such as CoOp, MaPLe, and TeCoA to assess generality. Beyond classification accuracy, TTA-VLM incorporates various evaluation metrics, including robustness, calibration, out-of-distribution detection, and stability, enabling a more holistic assessment of TTA methods. Through extensive experiments, we find that 1) existing TTA methods produce limited gains compared to the previous pioneering work; 2) current TTA methods exhibit poor collaboration with training-time fine-tuning methods; 3) accuracy gains frequently come at the cost of reduced model trustworthiness. We release TTA-VLM to provide fair comparison and comprehensive evaluation of TTA methods for VLMs, and we hope it encourages the community to develop more reliable and generalizable TTA strategies.