标题： 语言模型中 Berezinskii-Kosterlitz-Thouless 相变的首次数值观测 显示英文标题

标题： First numerical observation of the Berezinskii-Kosterlitz-Thouless transition in language models

摘要： 数十年来，自然语言中涉及不同统计性质的若干幂律临界特性——类似于物理系统在相变附近所表现出的标度性质——已被记录在案。 大型语言模型（LLMs）的近期兴起通过与物理学中的标度定律和涌现能力等概念的引人入胜的相似性，提供了进一步的证据和兴奋点。 然而，从统计物理学的角度理解的表现出相变的具体生成语言模型类别实例却尚未出现。 在这项工作中，受统计物理学中一维Potts模型的启发，我们构建了一个简单的概率语言模型，该模型属于上下文敏感语法（CSG）类，并在自然语言模型框架下数值展示了明确的相变现象。 我们明确表明，一个精确定义的序参量——该参量捕获了由语言模型生成的句子中的符号频率偏差——从严格为零变为严格非零值（在句子无限长度极限下），这表明在调整我们考虑的随机语言模型参数时出现的数学奇异性。 此外，我们将这种相变识别为Berezinskii-Kosterlitz-Thouless（BKT）相变的一种变体，这种相变已知不仅在相变点而且在整个相中表现出临界性质。 这一发现导致了一种可能性，即自然语言中的临界特性可能不需要仔细的精细调节或自组织临界性，而是可以通过语言结构与BKT相之间的潜在联系得到一般性的解释。 显示英文摘要

摘要： Several power-law critical properties involving different statistics in natural languages -- reminiscent of scaling properties of physical systems at or near phase transitions -- have been documented for decades. The recent rise of large language models (LLMs) has added further evidence and excitement by providing intriguing similarities with notions in physics such as scaling laws and emergent abilities. However, specific instances of classes of generative language models that exhibit phase transitions, as understood by the statistical physics community, are lacking. In this work, inspired by the one-dimensional Potts model in statistical physics we construct a simple probabilistic language model that falls under the class of context sensitive grammars (CSG), and numerically demonstrate an unambiguous phase transition in the framework of a natural language model. We explicitly show that a precisely defined order parameter -- that captures symbol frequency biases in the sentences generated by the language model -- changes from strictly 0 to a strictly nonzero value (in the infinite-length limit of sentences), implying a mathematical singularity arising when tuning the parameter of the stochastic language model we consider. Furthermore, we identify the phase transition as a variant of the Berezinskii-Kosterlitz-Thouless (BKT) transition, which is known to exhibit critical properties not only at the transition point but also in the entire phase. This finding leads to the possibility that critical properties in natural languages may not require careful fine-tuning nor self-organized criticality, but is generically explained by the underlying connection between language structures and the BKT phases.