Title: SECodec: Structural Entropy-based Compressive Speech Representation Codec for Speech Language Models Show Chinese title

Title: 基于结构熵的压缩语音表示编解码器 SECodec：语音语言模型的结构性熵基压缩语音表示编解码器

Abstract: With the rapid advancement of large language models (LLMs), discrete speech representations have become crucial for integrating speech into LLMs. Existing methods for speech representation discretization rely on a predefined codebook size and Euclidean distance-based quantization. However, 1) the size of codebook is a critical parameter that affects both codec performance and downstream task training efficiency. 2) The Euclidean distance-based quantization may lead to audio distortion when the size of the codebook is controlled within a reasonable range. In fact, in the field of information compression, structural information and entropy guidance are crucial, but previous methods have largely overlooked these factors. Therefore, we address the above issues from an information-theoretic perspective, we present SECodec, a novel speech representation codec based on structural entropy (SE) for building speech language models. Specifically, we first model speech as a graph, clustering the speech features nodes within the graph and extracting the corresponding codebook by hierarchically and disentangledly minimizing 2D SE. Then, to address the issue of audio distortion, we propose a new quantization method. This method still adheres to the 2D SE minimization principle, adaptively selecting the most suitable token corresponding to the cluster for each incoming original speech node. Furthermore, we develop a Structural Entropy-based Speech Language Model (SESLM) that leverages SECodec. Experimental results demonstrate that SECodec performs comparably to EnCodec in speech reconstruction, and SESLM surpasses VALL-E in zero-shot text-to-speech tasks. Code, demo speeches, speech feature graph, SE codebook, and models are available at https://github.com/wlq2019/SECodec. Show Chinese abstract

Abstract: 随着大型语言模型（LLMs）的快速发展，离散语音表示对于将语音集成到LLMs中变得至关重要。现有的语音表示离散化方法依赖于预定义的码本大小和基于欧几里得距离的量化方法。然而，1）码本的大小是一个影响编解码器性能和下游任务训练效率的关键参数；2）当码本大小被控制在合理范围内时，基于欧几里得距离的量化可能导致音频失真。事实上，在信息压缩领域，结构信息和熵引导至关重要，但以前的方法大多忽略了这些因素。因此，我们从信息论的角度解决了上述问题，提出了一种基于结构熵（SE）的新颖语音表示编解码器SECodec，用于构建语音语言模型。具体来说，我们首先将语音建模为图，对图中的语音特征节点进行聚类，并通过分层且解耦地最小化二维SE来提取相应的码本。然后，为了应对音频失真的问题，我们提出了一个新的量化方法。该方法仍然遵循二维SE最小化原则，根据每个输入原始语音节点对应的聚类，自适应地选择最合适的标记。此外，我们开发了一种基于结构熵的语音语言模型（SESLM），利用SECodec。实验结果表明，SECodec在语音重建方面与EnCodec表现相当，而SESLM在零样本文本转语音任务中优于VALL-E。代码、演示语音、语音特征图、SE码本以及模型可在https://github.com/wlq2019/SECodec获取。