标题： 基于时间与身份互补学习的音频辅助人脸视频修复 显示英文标题

标题： Audio-Assisted Face Video Restoration with Temporal and Identity Complementary Learning

摘要： 面部视频伴有音频已经成为我们日常生活的重要组成部分，但它们常常遭受复杂的退化。 大多数面部视频恢复方法忽略了视觉和音频特征之间的内在关联，尤其是在嘴部区域。 一些基于音频的面部视频恢复方法已被提出，但它们仅关注压缩伪影的去除。 在本文中，我们提出了一种通用的音频辅助面部视频恢复网络（GAVN），通过身份和时间互补学习来解决各种流媒体视频失真问题。 具体来说， GAVN首先在低分辨率空间中捕捉帧间时间特征以粗略恢复帧并节省计算成本。 然后，GAVN在高分辨率空间中借助音频信号和面部关键点提取帧内身份特征以恢复更多面部细节。 最后，重建模块整合时间特征和身份特征以生成高质量的面部视频。 实验结果表明，GAVN在面部视频压缩伪影去除、去模糊和超分辨率方面优于现有的最先进方法。 代码将在发表后发布。 显示英文摘要

摘要： Face videos accompanied by audio have become integral to our daily lives, while they often suffer from complex degradations. Most face video restoration methods neglect the intrinsic correlations between the visual and audio features, especially in mouth regions. A few audio-aided face video restoration methods have been proposed, but they only focus on compression artifact removal. In this paper, we propose a General Audio-assisted face Video restoration Network (GAVN) to address various types of streaming video distortions via identity and temporal complementary learning. Specifically, GAVN first captures inter-frame temporal features in the low-resolution space to restore frames coarsely and save computational cost. Then, GAVN extracts intra-frame identity features in the high-resolution space with the assistance of audio signals and face landmarks to restore more facial details. Finally, the reconstruction module integrates temporal features and identity features to generate high-quality face videos. Experimental results demonstrate that GAVN outperforms the existing state-of-the-art methods on face video compression artifact removal, deblurring, and super-resolution. Codes will be released upon publication.