Title: WaveMind: Towards a Conversational EEG Foundation Model Aligned to Textual and Visual Modalities Show Chinese title

Title: WaveMind：面向文本和视觉模态的对话脑电图基础模型

Abstract: Electroencephalography (EEG) interpretation using multimodal large language models (MLLMs) offers a novel approach for analyzing brain signals. However, the complex nature of brain activity introduces critical challenges: EEG signals simultaneously encode both cognitive processes and intrinsic neural states, creating a mismatch in EEG paired-data modality that hinders effective cross-modal representation learning. Through a pivot investigation, we uncover complementary relationships between these modalities. Leveraging this insight, we propose mapping EEG signals and their corresponding modalities into a unified semantic space to achieve generalized interpretation. To fully enable conversational capabilities, we further introduce WaveMind-Instruct-338k, the first cross-task EEG dataset for instruction tuning. The resulting model demonstrates robust classification accuracy while supporting flexible, open-ended conversations across four downstream tasks, thereby offering valuable insights for both neuroscience research and the development of general-purpose EEG models. Show Chinese abstract

Abstract: 脑电图（EEG）的解释使用多模态大语言模型（MLLMs）提供了一种分析脑信号的新方法。 然而，脑活动的复杂性带来了关键挑战：EEG信号同时编码认知过程和内在神经状态，导致EEG配对数据模态不匹配，阻碍了有效的跨模态表示学习。 通过一项pivot调查，我们揭示了这些模态之间的互补关系。 利用这一见解，我们提出将EEG信号及其对应模态映射到一个统一的语义空间，以实现通用解释。 为了充分实现对话能力，我们进一步引入WaveMind-Instruct-338k，这是首个用于指令调优的跨任务EEG数据集。 该模型在支持四个下游任务中灵活、开放式的对话的同时，表现出强大的分类准确性，从而为神经科学研究和通用EEG模型的发展提供了有价值的见解。