Title: Zero-Shot Statistical Tests for LLM-Generated Text Detection using Finite Sample Concentration Inequalities Show Chinese title

Title: 基于有限样本集中不等式的零样本统计检验用于检测LLM生成文本

Abstract: Verifying the provenance of content is crucial to the function of many organizations, e.g., educational institutions, social media platforms, firms, etc. This problem is becoming increasingly challenging as text generated by Large Language Models (LLMs) becomes almost indistinguishable from human-generated content. In addition, many institutions utilize in-house LLMs and want to ensure that external, non-sanctioned LLMs do not produce content within the institution. We answer the following question: Given a piece of text, can we identify whether it was produced by LLM $A$ or $B$ (where $B$ can be a human)? We model LLM-generated text as a sequential stochastic process with complete dependence on history and design zero-shot statistical tests to distinguish between (i) the text generated by two different sets of LLMs $A$ (in-house) and $B$ (non-sanctioned) and also (ii) LLM-generated and human-generated texts. We prove that our tests' type I and type II errors decrease exponentially as text length increases. For designing our tests for a given string, we demonstrate that if the string is generated by the evaluator model $A$, the log-perplexity of the string under $A$ converges to the average entropy of the string under $A$, except with an exponentially small probability in the string length. We also show that if $B$ generates the text, except with an exponentially small probability in string length, the log-perplexity of the string under $A$ converges to the average cross-entropy of $B$ and $A$. For our experiments: First, we present experiments using open-source LLMs to support our theoretical results, and then we provide experiments in a black-box setting with adversarial attacks. Practically, our work enables guaranteed finding of the origin of harmful or false LLM-generated text, which can be useful for combating misinformation and compliance with emerging AI regulations. Show Chinese abstract

Abstract: 验证内容的来源对于许多组织的功能至关重要，例如教育机构、社交媒体平台、公司等。 这个问题变得越来越具有挑战性，因为由大型语言模型（LLMs）生成的文本几乎与人工生成的内容无法区分。 此外，许多机构使用内部的LLMs，并希望确保外部未经批准的LLMs不会在机构内部生成内容。 我们回答以下问题：给定一段文本，我们能否识别它是否是由LLM $A$ 或 $B$ 生成的（其中 $B$ 可以是人类）？ 我们将LLM生成的文本建模为一个完全依赖于历史的顺序随机过程，并设计零样本统计检验来区分（i）由两组不同的LLMs $A$ （内部）和 $B$ （未经批准）生成的文本，以及（ii）LLM生成的文本和人工生成的文本。 我们证明了随着文本长度的增加，我们测试的第一类错误和第二类错误呈指数级减少。 对于设计给定字符串的测试，我们证明，如果该字符串是由评估器模型$A$生成的，则在$A$下该字符串的对数困惑度会收敛于在$A$下该字符串的平均熵，除非字符串长度的指数小概率事件。 我们还表明，如果$B$生成文本，除非字符串长度的指数小概率事件，那么在$A$下该字符串的对数困惑度会收敛于$B$和$A$的平均交叉熵。 对于我们的实验：首先，我们使用开源大语言模型进行实验以支持我们的理论结果，然后我们在对抗攻击的黑盒设置中提供实验。 实际上，我们的工作能够保证找到有害或虚假的大语言模型生成文本的来源，这有助于应对虚假信息并符合新兴的人工智能法规。