Title: LaVCa: LLM-assisted Visual Cortex Captioning Show Chinese title

Title: LaVCa：基于大语言模型的视觉皮层描述

Abstract: Understanding the property of neural populations (or voxels) in the human brain can advance our comprehension of human perceptual and cognitive processing capabilities and contribute to developing brain-inspired computer models. Recent encoding models using deep neural networks (DNNs) have successfully predicted voxel-wise activity. However, interpreting the properties that explain voxel responses remains challenging because of the black-box nature of DNNs. As a solution, we propose LLM-assisted Visual Cortex Captioning (LaVCa), a data-driven approach that uses large language models (LLMs) to generate natural-language captions for images to which voxels are selective. By applying LaVCa for image-evoked brain activity, we demonstrate that LaVCa generates captions that describe voxel selectivity more accurately than the previously proposed method. Furthermore, the captions generated by LaVCa quantitatively capture more detailed properties than the existing method at both the inter-voxel and intra-voxel levels. Furthermore, a more detailed analysis of the voxel-specific properties generated by LaVCa reveals fine-grained functional differentiation within regions of interest (ROIs) in the visual cortex and voxels that simultaneously represent multiple distinct concepts. These findings offer profound insights into human visual representations by assigning detailed captions throughout the visual cortex while highlighting the potential of LLM-based methods in understanding brain representations. Please check out our webpage at https://sites.google.com/view/lavca-llm/ Show Chinese abstract

Abstract: 理解人脑中神经种群（或体素）的特性可以促进我们对人类感知和认知处理能力的理解，并有助于开发类脑计算机模型。 最近使用深度神经网络（DNNs）的编码模型成功预测了体素级别的活动。 然而，由于DNNs的黑箱特性，解释解释体素响应的特性仍然具有挑战性。 作为解决方案，我们提出了LLM辅助的视觉皮层描述（LaVCa），这是一种数据驱动的方法，使用大型语言模型（LLMs）为体素选择的图像生成自然语言描述。 通过将LaVCa应用于图像诱发的脑活动，我们证明LaVCa生成的描述比之前提出的方法更准确地描述了体素选择性。 此外，LaVCa生成的描述在体素间和体素内层面都能定量地捕捉到比现有方法更详细的特性。 此外，对LaVCa生成的体素特性的更详细分析揭示了视觉皮层感兴趣区域（ROIs）内的细粒度功能分化以及同时表示多个不同概念的体素。 这些发现通过在整个视觉皮层分配详细描述，为人类视觉表征提供了深刻的见解，并突显了基于LLM的方法在理解脑表征中的潜力。 请访问我们的网页 https://sites.google.com/view/lavca-llm/