Title: Multimodal Emotion Coupling via Speech-to-Facial and Bodily Gestures in Dyadic Interaction Show Chinese title

Title: 双人互动中通过语音到面部及身体手势的多模态情感耦合

Abstract: Human emotional expression emerges through coordinated vocal, facial, and gestural signals. While speech face alignment is well established, the broader dynamics linking emotionally expressive speech to regional facial and hand motion remains critical for gaining a deeper insight into how emotional and behavior cues are communicated in real interactions. Further modulating the coordination is the structure of conversational exchange like sequential turn taking, which creates stable temporal windows for multimodal synchrony, and simultaneous speech, often indicative of high arousal moments, disrupts this alignment and impacts emotional clarity. Understanding these dynamics enhances realtime emotion detection by improving the accuracy of timing and synchrony across modalities in both human interactions and AI systems. This study examines multimodal emotion coupling using region specific motion capture from dyadic interactions in the IEMOCAP corpus. Speech features included low level prosody, MFCCs, and model derived arousal, valence, and categorical emotions (Happy, Sad, Angry, Neutral), aligned with 3D facial and hand marker displacements. Expressive activeness was quantified through framewise displacement magnitudes, and speech to gesture prediction mapped speech features to facial and hand movements. Nonoverlapping speech consistently elicited greater activeness particularly in the lower face and mouth. Sadness showed increased expressivity during nonoverlap, while anger suppressed gestures during overlaps. Predictive mapping revealed highest accuracy for prosody and MFCCs in articulatory regions while arousal and valence had lower and more context sensitive correlations. Notably, hand speech synchrony was enhanced under low arousal and overlapping speech, but not for valence. Show Chinese abstract

Abstract: 人类情感表达通过协调的语音、面部和手势信号得以展现。 虽然语音与面部对齐已经得到充分研究，但将情感丰富的语音与局部面部及手部运动动态联系起来仍然对于深入理解真实互动中情感和行为线索如何传递至关重要。 进一步调节这种协调的是对话交流的结构，例如顺序轮流发言，它为多模态同步创造了稳定的时序窗口；而同时说话，通常表明高唤醒时刻，会破坏这种对齐并影响情感清晰度。 理解这些动态可以提高实时情感检测的准确性，特别是在人类互动和人工智能系统中跨模态的时间和同步精度。 本研究使用IEMOCAP语料库中的二元交互数据，考察多模态情感耦合现象，利用区域特定的动作捕捉技术。 语音特征包括低级韵律、MFCCs（梅尔频率倒谱系数），以及模型推导出的唤醒、效价和情感类别（快乐、悲伤、愤怒、中性），并与三维面部和手部标记位移对齐。 通过帧间位移幅度量化表现活跃度，并通过语音到手势预测将语音特征映射到面部和手部运动。 非重叠语音始终引发更大的活跃度，尤其是在下脸部和嘴巴部位。 悲伤在非重叠期间表现出更高的表现力，而愤怒则在重叠期间抑制了手势。 预测映射显示，在发音区域，韵律和MFCCs具有最高的准确率，而唤醒和效价的相关性较低且更依赖于上下文。 值得注意的是，在低唤醒和重叠语音情况下，手部与语音的同步性增强，但效价并非如此。