Title: Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis Show Chinese title

Title: 跨语言F5-TTS：迈向语言无关的语音克隆和语音合成

Abstract: Flow-matching-based text-to-speech (TTS) models have shown high-quality speech synthesis. However, most current flow-matching-based TTS models still rely on reference transcripts corresponding to the audio prompt for synthesis. This dependency prevents cross-lingual voice cloning when audio prompt transcripts are unavailable, particularly for unseen languages. The key challenges for flow-matching-based TTS models to remove audio prompt transcripts are identifying word boundaries during training and determining appropriate duration during inference. In this paper, we introduce Cross-Lingual F5-TTS, a framework that enables cross-lingual voice cloning without audio prompt transcripts. Our method preprocesses audio prompts by forced alignment to obtain word boundaries, enabling direct synthesis from audio prompts while excluding transcripts during training. To address the duration modeling challenge, we train speaking rate predictors at different linguistic granularities to derive duration from speaker pace. Experiments show that our approach matches the performance of F5-TTS while enabling cross-lingual voice cloning. Show Chinese abstract

Abstract: 基于流匹配的文本到语音（TTS）模型已经展示了高质量的语音合成效果。 然而，大多数当前基于流匹配的TTS模型仍然依赖于与音频提示对应的参考转录文本进行合成。 这种依赖性在音频提示转录文本不可用时，特别是对于未见过的语言，会阻碍跨语言语音克隆。 基于流匹配的TTS模型去除音频提示转录文本的关键挑战在于训练期间识别单词边界以及在推理期间确定适当的持续时间。 在本文中，我们引入了 跨语言F5-TTS，一种无需音频提示转录文本即可实现跨语言语音克隆的框架。 我们的方法通过强制对齐预处理音频提示以获得单词边界，在训练期间排除转录文本的同时，实现从音频提示直接合成。 为了解决持续时间建模的挑战，我们在不同的语言粒度上训练说话速度预测器，以从说话者节奏中推导出持续时间。 实验表明，我们的方法在性能上与F5-TTS相当，同时实现了跨语言语音克隆。