标题： 视觉与语言训练有助于部署分类学知识，但不会根本改变它 显示英文标题

标题： Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It

摘要： 视觉和语言(VL)训练是否以有意义的方式改变了语言模型的语言表示? 文献中的大多数结果表明，在行为和表征方面都存在不一致或微小的差异。 在本工作中，我们从一个假设出发，即VL训练可能产生显著影响的领域是词汇-概念知识，特别是其分类组织。 通过比较纯文本LM和其VL训练后的对应模型的最小对，我们首先表明，在需要对问题中提到的概念进行分类理解的纯文本问答任务中，VL模型通常比纯文本模型表现更好。 通过一系列有针对性的行为和表征分析，我们表明LM和VLM在分类知识本身方面没有显著差异，但在如何表示包含分类关系概念的问题与非分类关系概念的问题方面存在差异。 这表明，通过额外的VL训练，分类知识本身并没有发生显著变化，但VL训练确实提高了这种知识在特定任务上下文中的应用能力，即使任务的呈现方式完全是语言性的。 显示英文摘要

摘要： Does vision-and-language (VL) training change the linguistic representations of language models in meaningful ways? Most results in the literature have shown inconsistent or marginal differences, both behaviorally and representationally. In this work, we start from the hypothesis that the domain in which VL training could have a significant effect is lexical-conceptual knowledge, in particular its taxonomic organization. Through comparing minimal pairs of text-only LMs and their VL-trained counterparts, we first show that the VL models often outperform their text-only counterparts on a text-only question-answering task that requires taxonomic understanding of concepts mentioned in the questions. Using an array of targeted behavioral and representational analyses, we show that the LMs and VLMs do not differ significantly in terms of their taxonomic knowledge itself, but they differ in how they represent questions that contain concepts in a taxonomic relation vs. a non-taxonomic relation. This implies that the taxonomic knowledge itself does not change substantially through additional VL training, but VL training does improve the deployment of this knowledge in the context of a specific task, even when the presentation of the task is purely linguistic.