Title: Localization vs. Semantics: How Can Language Benefit Visual Representation Learning? Show Chinese title

Title: 定位与语义：语言如何促进视觉表示学习？

Abstract: Despite the superior performance brought by vision-and-language pretraining, it remains unclear whether learning with multi-modal data can help understand each individual modality. In this work, we investigate how language can help with visual representation learning from a probing perspective. Specifically, we compare vision-and-language and vision-only models by probing their visual representations on a broad range of tasks, in order to assess the quality of the learned representations in a fine-grained manner. Interestingly, our probing results suggest that vision-and-language models are better at label prediction tasks like object and attribute prediction, while vision-only models are stronger at dense prediction tasks that require more localized information. With further analysis using detailed metrics, our study suggests that language helps vision models learn better semantics, but not localization. Code is released at https://github.com/Lizw14/visual_probing. Show Chinese abstract

Abstract: 尽管视觉和语言预训练带来了优越的性能，但尚不清楚学习多模态数据是否有助于理解每个单独的模态。 在本工作中，我们从探测的角度研究语言如何帮助视觉表示学习。 具体来说，我们通过在广泛的任务上探测视觉表示，比较视觉和语言模型以及仅视觉模型，以更细致的方式评估所学表示的质量。 有趣的是，我们的探测结果表明，视觉和语言模型在标签预测任务（如物体和属性预测）方面表现更好，而仅视觉模型在需要更多局部信息的密集预测任务中表现更强。 通过使用详细指标进行进一步分析，我们的研究表明，语言有助于视觉模型学习更好的语义，但无助于定位。 代码已发布在 https://github.com/Lizw14/visual_probing.