Title: Temporally Heterogeneous Graph Contrastive Learning for Multimodal Acoustic event Classification Show Chinese title

Title: 时间异构图对比学习用于多模态声音事件分类

Abstract: Multimodal acoustic event classification plays a key role in audio-visual systems. Although combining audio and visual signals improves recognition, it is still difficult to align them over time and to reduce the effect of noise across modalities. Existing methods often treat audio and visual streams separately, fusing features later with contrastive or mutual information objectives. Recent advances explore multimodal graph learning, but most fail to distinguish between intra- and inter-modal temporal dependencies. To address this, we propose Temporally Heterogeneous Graph-based Contrastive Learning (THGCL). Our framework constructs a temporal graph for each event, where audio and video segments form nodes and their temporal links form edges. We introduce Gaussian processes for intra-modal smoothness, Hawkes processes for inter-modal decay, and contrastive learning to capture fine-grained relationships. Experiments on AudioSet show that THGCL achieves state-of-the-art performance. Show Chinese abstract

Abstract: 多模态声学事件分类在音视频系统中起着关键作用。 尽管结合音频和视觉信号可以提高识别效果，但仍然难以在时间上对齐它们，并减少跨模态的噪声影响。 现有方法通常分别处理音频和视觉流，之后通过对比或互信息目标融合特征。 最近的进展探索了多模态图学习，但大多数方法无法区分模态内和模态间的时序依赖关系。 为了解决这个问题，我们提出了基于时序异构图的对比学习（THGCL）。 我们的框架为每个事件构建一个时序图，其中音频和视频片段形成节点，它们的时序链接形成边。 我们引入高斯过程用于模态内平滑性，霍克斯过程用于模态间衰减，并通过对比学习捕捉细粒度的关系。 在AudioSet上的实验表明，THGCL实现了最先进的性能。