Title: Spotlighting Partially Visible Cinematic Language for Video-to-Audio Generation via Self-distillation Show Chinese title

Title: 通过自蒸馏突出部分可见的影视语言用于视频到音频生成

Abstract: Video-to-Audio (V2A) Generation achieves significant progress and plays a crucial role in film and video post-production. However, current methods overlook the cinematic language, a critical component of artistic expression in filmmaking. As a result, their performance deteriorates in scenarios where Foley targets are only partially visible. To address this challenge, we propose a simple self-distillation approach to extend V2A models to cinematic language scenarios. By simulating the cinematic language variations, the student model learns to align the video features of training pairs with the same audio-visual correspondences, enabling it to effectively capture the associations between sounds and partial visual information. Our method not only achieves impressive improvements under partial visibility across all evaluation metrics, but also enhances performance on the large-scale V2A dataset, VGGSound. Show Chinese abstract

Abstract: 视频到音频（V2A）生成取得了显著进展，并在电影和视频后期制作中发挥着关键作用。 然而，当前的方法忽视了电影语言，这是电影制作中艺术表达的关键组成部分。 因此，在Foley目标仅部分可见的场景中，它们的性能会下降。 为了解决这个挑战，我们提出了一种简单的自我蒸馏方法，以将V2A模型扩展到电影语言场景。 通过模拟电影语言的变化，学生模型学习将训练对的视频特征与相同的音画对应关系对齐，使其能够有效地捕捉声音与部分视觉信息之间的关联。 我们的方法不仅在所有评估指标下在部分可见性情况下实现了令人印象深刻的改进，而且还在大规模的V2A数据集VGGSound上提升了性能。