标题： TaDiCodec：用于语音语言建模的文本感知扩散语音标记器 显示英文标题

标题： TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling

摘要： 语音分词器是语音语言模型的基础组件，但当前的设计存在一些局限性，包括：1）依赖于多层残差向量量化结构或高帧率，2）依赖于辅助预训练模型进行语义蒸馏，3）需要复杂的两阶段训练过程。 在本工作中，我们介绍了文本感知扩散变换器语音编解码器（TaDiCodec），这是一种旨在克服这些挑战的新方法。 TaDiCodec通过扩散自编码器进行量化和重建的端到端优化，同时将文本指导整合到扩散解码器中，以提高重建质量并实现最佳压缩。 TaDiCodec在24 kHz语音上使用单层代码本实现了极低的帧率6.25 Hz和相应的比特率为0.0875 kbps，同时在关键语音生成评估指标如单词错误率（WER）、说话人相似性（SIM）和语音质量（UTMOS）上保持优异性能。 值得注意的是，TaDiCodec采用了一阶段端到端训练范式，消除了对辅助预训练模型的需求。 我们还验证了TaDiCodec在基于语言模型的零样本文本到语音中的兼容性，适用于自回归建模和掩码生成建模，证明了其在语音语言建模中的有效性和效率，以及显著的小的重建-生成差距。 我们将开源我们的代码和模型检查点。 音频样本可在 https:/tadicodec.github.io/ 获取。 我们在 https:/github.com/HeCheng0625/Diffusion-Speech-Tokenizer 发布代码和模型检查点。 显示英文摘要

摘要： Speech tokenizers serve as foundational components for speech language models, yet current designs exhibit several limitations, including: 1) dependence on multi-layer residual vector quantization structures or high frame rates, 2) reliance on auxiliary pre-trained models for semantic distillation, and 3) requirements for complex two-stage training processes. In this work, we introduce the Text-aware Diffusion Transformer Speech Codec (TaDiCodec), a novel approach designed to overcome these challenges. TaDiCodec employs end-to-end optimization for quantization and reconstruction through a diffusion autoencoder, while integrating text guidance into the diffusion decoder to enhance reconstruction quality and achieve optimal compression. TaDiCodec achieves an extremely low frame rate of 6.25 Hz and a corresponding bitrate of 0.0875 kbps with a single-layer codebook for 24 kHz speech, while maintaining superior performance on critical speech generation evaluation metrics such as Word Error Rate (WER), speaker similarity (SIM), and speech quality (UTMOS). Notably, TaDiCodec employs a single-stage, end-to-end training paradigm, and obviating the need for auxiliary pre-trained models. We also validate the compatibility of TaDiCodec in language model based zero-shot text-to-speech with both autoregressive modeling and masked generative modeling, demonstrating its effectiveness and efficiency for speech language modeling, as well as a significantly small reconstruction-generation gap. We will open source our code and model checkpoints. Audio samples are are available at https:/tadicodec.github.io/. We release code and model checkpoints at https:/github.com/HeCheng0625/Diffusion-Speech-Tokenizer.