Title: Zero-Shot Statistical Tests for LLM-Generated Text Detection using Finite Sample Concentration Inequalities Show Chinese title

Title: 基于有限样本浓度不等式的零样本统计检验用于检测LLM生成的文本

Abstract: Verifying the provenance of content is crucial to the function of many organizations, e.g., educational institutions, social media platforms, firms, etc. This problem is becoming increasingly difficult as text generated by Large Language Models (LLMs) becomes almost indistinguishable from human-generated content. In addition, many institutions utilize in-house LLMs and want to ensure that external, non-sanctioned LLMs do not produce content within the institution. In this paper, we answer the following question: Given a piece of text, can we identify whether it was produced by LLM $A$ or $B$ (where $B$ can be a human)? We model LLM-generated text as a sequential stochastic process with complete dependence on history and design zero-shot statistical tests to distinguish between (i) the text generated by two different sets of LLMs $A$ (in-house) and $B$ (non-sanctioned) and also (ii) LLM-generated and human-generated texts. We prove that the type I and type II errors for our tests decrease exponentially in the text length. In designing our tests, we derive concentration inequalities on the difference between log-perplexity and the average entropy of the string under $A$. Specifically, for a given string, we demonstrate that if the string is generated by $A$, the log-perplexity of the string under $A$ converges to the average entropy of the string under $A$, except with an exponentially small probability in string length. We also show that if $B$ generates the text, except with an exponentially small probability in string length, the log-perplexity of the string under $A$ converges to the average cross-entropy of $B$ and $A$. Lastly, we present preliminary experimental results to support our theoretical results. By enabling guaranteed (with high probability) finding of the origin of harmful LLM-generated text with arbitrary size, we can help combat misinformation. Show Chinese abstract

Abstract: 验证内容的来源对于许多组织的功能至关重要，例如教育机构、社交媒体平台、公司等。 随着由大型语言模型（LLMs）生成的文本几乎与人工生成的内容无法区分，这个问题变得越来越困难。 此外，许多机构使用内部的LLMs，并希望确保外部未经批准的LLMs不会在机构内部生成内容。 在本文中，我们回答以下问题：给定一段文本，我们能否确定它是由LLM$A$还是$B$生成的（其中$B$可以是人类）？ 我们将LLM生成的文本建模为一个完全依赖于历史的顺序随机过程，并设计零样本统计检验来区分（i）由两组不同的LLMs$A$（内部）和$B$（未经批准）生成的文本，以及（ii）LLM生成的文本和人工生成的文本。 我们证明了我们的测试的一类错误和二类错误随着文本长度呈指数级减少。 在设计我们的测试时，我们推导了log-perplexity与字符串在$A$下的平均熵之间的差值的集中不等式。 具体来说，对于给定的字符串，我们证明如果该字符串由$A$生成，则在$A$下的对数困惑度会收敛于在$A$下的字符串平均熵，除了字符串长度的指数小概率情况外。 我们还表明，如果$B$生成文本，则在$A$下的对数困惑度会收敛于$B$和$A$的平均交叉熵，除了字符串长度的指数小概率情况外。 最后，我们提供了初步的实验结果来支持我们的理论结果。 通过实现保证（以高概率）找到任意大小的有害 LLM 生成文本的来源，我们可以帮助对抗虚假信息。