标题： 局部归一化失真与大语言模型解码策略的遍历理论 显示英文标题

标题： Local Normalization Distortion and the Thermodynamic Formalism of Decoding Strategies for Large Language Models

摘要： 硬件和语言模型架构的进步推动了自然语言生成的革命。 然而，自回归模型计算下一个标记选择的概率分布，从这些分布中进行采样，称为解码，与其它设计选择相比受到的关注要少得多。 现有的解码策略主要基于启发式方法，导致难以以系统化的方式应用或改进这些方法。 我们通过将流行的解码算法表示为遍历理论语言中的平衡状态，并说明它们优化的功能，来发展语言模型解码策略的理论。 利用这一点，我们分析了局部归一化步骤对top-k、核采样和温度采样的影响，这些步骤用于使概率总和为一。 我们认为局部归一化失真是解码策略的根本缺陷，并量化了这种失真的大小及其对生成文本质量与多样性的数学代理指标的影响。 与普遍的解释相反，我们认为top-k采样相对于核采样的性能不佳的主要原因是局部归一化失真。 这为未来解码算法的设计和机器生成文本的检测提供了结论。 显示英文摘要

摘要： Advances in hardware and language model architecture have spurred a revolution in natural language generation. However, autoregressive models compute probability distributions over next-token choices, and sampling from these distributions, known as decoding, has received significantly less attention than other design choices. Existing decoding strategies are largely based on heuristics, resulting in methods that are hard to apply or improve in a principled manner. We develop the theory of decoding strategies for language models by expressing popular decoding algorithms as equilibrium states in the language of ergodic theory and stating the functions they optimize. Using this, we analyze the effect of the local normalization step of top-k, nucleus, and temperature sampling, used to make probabilities sum to one. We argue that local normalization distortion is a fundamental defect of decoding strategies and quantify the size of this distortion and its effect on mathematical proxies for the quality and diversity of generated text. Contrary to the prevailing explanation, we argue that the major cause of the under-performance of top-k sampling relative to nucleus sampling is local normalization distortion. This yields conclusions for the future design of decoding algorithms and the detection of machine-generated text.