标题： 多尺度多实例视觉声音定位与分割 显示英文标题

标题： Multi-scale Multi-instance Visual Sound Localization and Segmentation

摘要： 视觉声源定位是一个典型且具有挑战性的问题，旨在预测视频中与声源相对应的物体位置。 先前的方法主要利用全局音频与单尺度视觉特征之间的视听关联来定位每张图像中的发声物体。 尽管这些方法表现良好，但它们忽略了相应图像的多尺度视觉特征，并且无法像真实标签那样学习判别区域。 为了解决这个问题，我们提出了一种新颖的多尺度多实例视觉声源定位框架，即M2VSL，该框架可以直接从输入图像中学习与声源相关的多尺度语义特征以定位发声物体。 具体而言，我们的M2VSL利用可学习的多尺度视觉特征，在相应图像的多级位置上对视听表示进行对齐。 我们还引入了一种新颖的多尺度多实例变换器，用于动态聚合多尺度跨模态表示以实现视觉声源定位。 我们在VGGSound-Instruments、VGG-Sound Sources和AVSBench基准数据集上进行了广泛的实验。 结果表明，所提出的M2VSL在发声物体定位和分割方面可以达到最先进的性能。 显示英文摘要

摘要： Visual sound localization is a typical and challenging problem that predicts the location of objects corresponding to the sound source in a video. Previous methods mainly used the audio-visual association between global audio and one-scale visual features to localize sounding objects in each image. Despite their promising performance, they omitted multi-scale visual features of the corresponding image, and they cannot learn discriminative regions compared to ground truths. To address this issue, we propose a novel multi-scale multi-instance visual sound localization framework, namely M2VSL, that can directly learn multi-scale semantic features associated with sound sources from the input image to localize sounding objects. Specifically, our M2VSL leverages learnable multi-scale visual features to align audio-visual representations at multi-level locations of the corresponding image. We also introduce a novel multi-scale multi-instance transformer to dynamically aggregate multi-scale cross-modal representations for visual sound localization. We conduct extensive experiments on VGGSound-Instruments, VGG-Sound Sources, and AVSBench benchmarks. The results demonstrate that the proposed M2VSL can achieve state-of-the-art performance on sounding object localization and segmentation.