标题： 并非所有注意力头都是你需要的：通过注意力消融精炼CLIP的图像表示 显示英文标题

标题： Not All Attention Heads Are What You Need: Refining CLIP's Image Representation with Attention Ablation

摘要： 本文研究了CLIP图像编码器中注意力头的作用。 虽然CLIP在各种应用中表现出强大的性能，但我们假设某些注意力头会负面影响最终表示，并且消除它们可以提高下游任务的性能。 为了利用这一见解，我们提出了一种简单而有效的方法，称为注意力消融技术（AAT），通过操作注意力权重来抑制特定头的贡献。 通过整合两种针对不同应用场景的替代策略，AAT系统地识别并消融有害的注意力头，以提高表示质量。 实验表明，AAT在多个领域中始终能够提升下游任务的性能，在跨模态检索中，CLIP系列模型的召回率最高提升了11.1%。 结果突显了AAT在几乎不增加推理成本的情况下有效优化大规模视觉-语言模型的潜力。 显示英文摘要

摘要： This paper studies the role of attention heads in CLIP's image encoder. While CLIP has exhibited robust performance across diverse applications, we hypothesize that certain attention heads negatively affect final representations and that ablating them can improve performance in downstream tasks. To capitalize on this insight, we propose a simple yet effective method, called Attention Ablation Technique (AAT), to suppress the contribution of specific heads by manipulating attention weights. By integrating two alternative strategies tailored for different application scenarios, AAT systematically identifies and ablates detrimental attention heads to enhance representation quality. Experiments demonstrate that AAT consistently improves downstream task performance across various domains, boosting recall rate by up to 11.1% on CLIP-family models for cross-modal retrieval. The results highlight the potential of AAT to effectively refine large-scale vision-language models with virtually no increase in inference cost.