标题： SecoustiCodec：跨模态对齐的流式单编码本语音编解码器 显示英文标题

标题： SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec

摘要： 语音编解码器在统一语音和文本语言模型中起到关键的桥梁作用。 现有的编解码方法在语义编码方面面临一些挑战，例如残留的副语言信息（如音色、情感）、语义完整性不足、重建能力有限以及缺乏对流式传输的支持。 为了解决这些挑战，我们提出了SecoustiCodec，这是一种跨模态对齐的低比特率流式语音编解码器，在单个代码本空间中分离语义和副语言信息。 为了确保语义完整性和重建保真度，引入了副语言编码，以弥合语义编码和声学编码之间的信息差距。 提出了一种基于VAE（变分自编码器）和FSQ（有限标量量化）的仅语义高效量化方法。 这种方法缓解了标记的长尾分布问题，同时保持了高代码本利用率。 提出了一种基于对比学习的语义解缠方法，该方法在联合多模态帧级空间中对齐文本和语音，有效从语义编码中去除副语言信息。 提出了一种声学约束的多阶段优化策略，以确保鲁棒和稳定的收敛。 图~\ref{fig:pesq_kbps_below_2kbps}显示SecoustiCodec在0.27/1 kbps时实现了最先进的重建质量（PESQ）1.77/2.58。 SecoustiCodec的代码和模型权重将在同行评审过程完成后开源。 我们已经开源了SecoustiCodec的演示、代码和模型权重。 显示英文摘要

摘要： Speech codecs serve as a crucial bridge in unifying speech and text language models. Existing codec methods face several challenges in semantic encoding, such as residual paralinguistic information (e.g., timbre, emotion), insufficient semantic completeness, limited reconstruction capability, and lack of support for streaming. To address these challenges, we propose SecoustiCodec, a cross-modal aligned low-bitrate streaming speech codec that disentangles semantic and paralinguistic information in a single-codebook space. To ensure semantic completeness and reconstruction fidelity, paralinguistic encoding is introduced to bridge the information gap between semantic and acoustic encoding. A semantic-only efficient quantization method based on VAE (Variational Autoencoder) and FSQ (Finite Scalar Quantization) is proposed. This approach alleviates the long-tail distribution problem of tokens while maintaining high codebook utilization. A semantic disentanglement method based on contrastive learning is proposed, which aligns text and speech in a joint multimodal frame-level space, effectively removing paralinguistic information from semantic encoding. An acoustic-constrained multi-stage optimization strategy is proposed to ensure robust and stable convergence. Figure~\ref{fig:pesq_kbps_below_2kbps} shows SecoustiCodec achieves SOTA (state-of-the-art) reconstruction quality (PESQ) of 1.77/2.58 at 0.27/1 kbps. The code and model weights for SecoustiCodec will be open-sourced upon the completion of the peer-review process. We've open-sourced SecoustiCodec's demo, code, and model weights.