Title: State-Space Models in Efficient Whispered and Multi-dialect Speech Recognition Show Chinese title

Title: 状态空间模型在高效耳语和多方言语音识别中的应用

Abstract: Whispered speech recognition presents significant challenges for conventional automatic speech recognition systems, particularly when combined with dialect variation. However, utilizing an efficient method to solve this problem using a low-range dataset and processing load is beneficial. This paper proposes a solution using a Mamba-based state-space model and four fine-tuned self-supervised models consisting of Wav2Vec2, WavLM, HuBERT, and Whisper to address the dual challenges of whispered speech and dialect diversity. Based on our knowledge, this represents the best performance reported on the wTIMIT and CHAINS datasets for whispered speech recognition. We trained the models using whispered and normal speech data across Singaporean, US, and Irish dialects. The findings demonstrated that utilizing the proposed Mamba-based model could work as a highly efficient model trained with low amounts of whispered data to simultaneously work on whispered and normal speech recognition. The code for this work is freely available. Show Chinese abstract

Abstract: 耳语语音识别对传统自动语音识别系统提出了重大挑战，尤其是在结合方言变化时。 然而，使用一种高效的方法，利用低范围数据集和处理负载来解决这个问题是有益的。 本文提出了一种解决方案，使用基于Mamba的状态空间模型和四个微调的自监督模型，包括Wav2Vec2、WavLM、HuBERT和Whisper，以应对耳语语音和方言多样性的双重挑战。 据我们所知，这是在耳语语音识别的wTIMIT和CHAINS数据集上报告的最佳性能。 我们使用新加坡、美国和爱尔兰方言的耳语和正常语音数据训练了这些模型。 研究结果表明，使用所提出的基于Mamba的模型可以作为一种高效模型，仅用少量耳语数据进行训练，同时处理耳语和正常语音识别。 本工作的代码是免费提供的。