Title: Milmer: a Framework for Multiple Instance Learning based Multimodal Emotion Recognition Show Chinese title

Title: Milmer：基于多实例学习的多模态情感识别框架

Abstract: Emotions play a crucial role in human behavior and decision-making, making emotion recognition a key area of interest in human-computer interaction (HCI). This study addresses the challenges of emotion recognition by integrating facial expression analysis with electroencephalogram (EEG) signals, introducing a novel multimodal framework-Milmer. The proposed framework employs a transformer-based fusion approach to effectively integrate visual and physiological modalities. It consists of an EEG preprocessing module, a facial feature extraction and balancing module, and a cross-modal fusion module. To enhance visual feature extraction, we fine-tune a pre-trained Swin Transformer on emotion-related datasets. Additionally, a cross-attention mechanism is introduced to balance token representation across modalities, ensuring effective feature integration. A key innovation of this work is the adoption of a multiple instance learning (MIL) approach, which extracts meaningful information from multiple facial expression images over time, capturing critical temporal dynamics often overlooked in previous studies. Extensive experiments conducted on the DEAP dataset demonstrate the superiority of the proposed framework, achieving a classification accuracy of 96.72% in the four-class emotion recognition task. Ablation studies further validate the contributions of each module, highlighting the significance of advanced feature extraction and fusion strategies in enhancing emotion recognition performance. Our code are available at https://github.com/liangyubuaa/Milmer. Show Chinese abstract

Abstract: 情绪在人类行为和决策中起着关键作用，使情绪识别成为人机交互（HCI）中的一个关键研究领域。 本研究通过将面部表情分析与脑电图（EEG）信号相结合，解决了情绪识别的挑战，引入了一种新颖的多模态框架-Milmer。 所提出的框架采用基于Transformer的融合方法，以有效整合视觉和生理模态。 它包括一个EEG预处理模块、一个面部特征提取和平衡模块以及一个跨模态融合模块。 为了增强视觉特征提取，我们在与情绪相关的数据集上微调了一个预训练的Swin Transformer。 此外，引入了一种交叉注意力机制，以平衡不同模态之间的标记表示，确保有效的特征集成。 本工作的关键创新是采用了多实例学习（MIL）方法，该方法从随时间变化的多个面部表情图像中提取有意义的信息，捕捉到之前研究中常被忽视的关键时间动态。 在DEAP数据集上进行的大量实验证明了所提出框架的优势，在四类情绪识别任务中达到了96.72%的分类准确率。 消融研究进一步验证了每个模块的贡献，突显了先进特征提取和融合策略在提升情绪识别性能中的重要性。 我们的代码可在https://github.com/liangyubuaa/Milmer获取。