标题： 基于熵的粗粒度和压缩语义语音表示学习 显示英文标题

标题： Entropy-based Coarse and Compressed Semantic Speech Representation Learning

摘要： 离散语音表示学习最近在声学和语义建模方面引起了越来越多的关注。 现有的方法通常将16 kHz的波形以每秒25或50个标记的速率编码为离散标记。 然而，考虑到语音通常每秒只传达2到5个词，这种细粒度的分词引入了冗余，并阻碍了下游训练和推理的效率。 此外，这种频率下的语义语音表示主要捕捉语音级别的信息，而语义理解可能不需要如此详细的标记级分辨率。 为了解决这些限制，我们提出了一种基于熵的动态聚合框架，用于学习压缩的语义语音表示。 首先，通过在大规模未标记数据上进行下一个标记预测来预训练一个语音语言模型，以捕捉常见的标记模式。 然后使用预测熵自适应地确定聚合边界，接着通过一个交叉注意力模块融合每个片段内的信息。 通过调整熵阈值，可以灵活控制表示的粒度和压缩比。 在自动语音识别、语音到文本翻译和语音转换任务上的实验表明，压缩表示的表现与密集标记序列相当或更好，证明了所提出方法的有效性。 显示英文摘要

摘要： Discrete speech representation learning has recently attracted increasing interest in both acoustic and semantic modeling. Existing approaches typically encode 16 kHz waveforms into discrete tokens at a rate of 25 or 50 tokens per second. However, given that speech generally conveys only 2 to 5 words per second, such fine-grained tokenization introduces redundancy and hinders efficiency in downstream training and inference. Moreover, semantic speech representations at this frequency primarily capture phonetic-level information, while semantic understanding may not require such detailed token-level resolution. To address these limitations, we propose an entropy-based dynamic aggregation framework for learning compressed semantic speech representations. A speech language model is first pre-trained via next-token prediction on large-scale unlabeled data to capture frequent token patterns. Predictive entropy is then used to adaptively determine aggregation boundaries, followed by a cross-attention module that fuses information within each segment. By adjusting the entropy threshold, the granularity and compression ratio of the representations can be flexibly controlled. Experiments on ASR, speech-to-text translation, and voice conversion tasks demonstrate that the compressed representations perform on par with or better than dense token sequences, demonstrating the effectiveness of the proposed approach.