标题： 何时分布的接近性意味着表征相似性？ 从可识别性视角出发 显示英文标题

标题： When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective

摘要： 当不同深度神经网络学到的表示何时以及为何相似，是一个活跃的研究课题。 我们选择从可辨识性理论的角度来解答这些问题，该理论表明，表征相似性的度量应该对使模型分布保持不变的变换具有不变性。 聚焦于一个包含几种流行预训练方法的模型族，例如自回归语言模型，我们探讨了生成分布接近的模型何时会有相似的表示。 我们证明了模型分布之间的小的 Kullback-Leibler 散度并不能保证相应的表示是相似的。 这有一个重要的推论，即任意接近最大化似然的模型仍可能学习到不相似的表示，这一现象在我们在 CIFAR-10 数据集上训练的模型的经验观察中得到了反映。 然后我们定义了一种分布距离，其中接近意味着表征相似，并在合成实验中发现更宽的网络学习到的距离更近且表示更相似的分布。 我们的结果建立了分布接近与表征相似性之间的联系。 显示英文摘要

摘要： When and why representations learned by different deep neural networks are similar is an active research topic. We choose to address these questions from the perspective of identifiability theory, which suggests that a measure of representational similarity should be invariant to transformations that leave the model distribution unchanged. Focusing on a model family which includes several popular pre-training approaches, e.g., autoregressive language models, we explore when models which generate distributions that are close have similar representations. We prove that a small Kullback-Leibler divergence between the model distributions does not guarantee that the corresponding representations are similar. This has the important corollary that models arbitrarily close to maximizing the likelihood can still learn dissimilar representations, a phenomenon mirrored in our empirical observations on models trained on CIFAR-10. We then define a distributional distance for which closeness implies representational similarity, and in synthetic experiments, we find that wider networks learn distributions which are closer with respect to our distance and have more similar representations. Our results establish a link between closeness in distribution and representational similarity.