标题： 基于渐进残差提取的预训练用于语音表示学习 显示英文标题

标题： Progressive Residual Extraction based Pre-training for Speech Representation Learning

摘要： 自监督学习（SSL）在语音处理领域引起了广泛关注，在语音识别等语言任务上表现出色。然而，在各种下游任务上同时提升预训练模型的性能具有很大挑战性，因为每个下游任务都需要不同的语音信息。为此，我们提出了一种基于渐进残差提取的自监督学习方法，名为ProgRE。具体而言，我们在一种编码器风格的SSL主干网络中引入了两个轻量且专门的任务模块，以增强其从语音中提取音高变化和说话人信息的能力。此外，为了防止强化的音高变化和说话人信息对无关内容信息学习的干扰，我们从主分支中残差移除这两个模块提取的信息。然后，主分支通过HuBERT的语音掩蔽预测进行训练，以确保Transformer深层特征在内容任务上的性能。这样，我们可以从输入语音中逐步提取音高变化、说话人和内容表示。最后，我们可以使用不同层权重组合多种表示形式，以获得针对各种下游任务的任务特定表示。实验结果表明，与wav2vec2.0、HuBERT和WavLM等优秀的SSL方法相比，我们提出的方法在说话人识别、语音识别、情感识别、语音增强和语音转换等多种任务上实现了联合性能提升。 显示英文摘要

摘要： Self-supervised learning (SSL) has garnered significant attention in speech processing, excelling in linguistic tasks such as speech recognition. However, jointly improving the performance of pre-trained models on various downstream tasks, each requiring different speech information, poses significant challenges. To this purpose, we propose a progressive residual extraction based self-supervised learning method, named ProgRE. Specifically, we introduce two lightweight and specialized task modules into an encoder-style SSL backbone to enhance its ability to extract pitch variation and speaker information from speech. Furthermore, to prevent the interference of reinforced pitch variation and speaker information with irrelevant content information learning, we residually remove the information extracted by these two modules from the main branch. The main branch is then trained using HuBERT's speech masking prediction to ensure the performance of the Transformer's deep-layer features on content tasks. In this way, we can progressively extract pitch variation, speaker, and content representations from the input speech. Finally, we can combine multiple representations with diverse speech information using different layer weights to obtain task-specific representations for various downstream tasks. Experimental results indicate that our proposed method achieves joint performance improvements on various tasks, such as speaker identification, speech recognition, emotion recognition, speech enhancement, and voice conversion, compared to excellent SSL methods such as wav2vec2.0, HuBERT, and WavLM.