Title: PLSemanticsBench: Large Language Models As Programming Language Interpreters Show Chinese title

Title: PLSemanticsBench：大型语言模型作为编程语言解释器

Abstract: As large language models (LLMs) excel at code reasoning, a natural question arises: can an LLM execute programs (i.e., act as an interpreter) purely based on a programming language's formal semantics? If so, it will enable rapid prototyping of new programming languages and language features. We study this question using the imperative language IMP (a subset of C), formalized via small-step operational semantics (SOS) and rewriting-based operational semantics (K-semantics). We introduce three evaluation sets-Human-Written, LLM-Translated, and Fuzzer- Generated-whose difficulty is controlled by code-complexity metrics spanning the size, control-flow, and data-flow axes. Given a program and its semantics formalized with SOS/K-semantics, models are evaluated on three tasks ranging from coarse to fine: (1) final-state prediction, (2) semantic rule prediction, and (3) execution trace prediction. To distinguish pretraining memorization from semantic competence, we define two nonstandard semantics obtained through systematic mutations of the standard rules. Across strong code/reasoning LLMs, performance drops under nonstandard semantics despite high performance under the standard one. We further find that (i) there are patterns to different model failures, (ii) most reasoning models perform exceptionally well on coarse grained tasks involving reasoning about highly complex programs often containing nested loop depths beyond five, and surprisingly, (iii) providing formal semantics helps on simple programs but often hurts on more complex ones. Overall, the results show a promise that LLMs could serve as programming language interpreters, but points to the lack of their robust semantics understanding. We release the benchmark and the supporting code at https://github.com/EngineeringSoftware/PLSemanticsBench. Show Chinese abstract

Abstract: 随着大型语言模型（LLMs）在代码推理方面的出色表现，一个自然的问题浮现：基于编程语言的形式语义，LLM能否纯粹地执行程序（即充当解释器）？ 如果是这样，它将能够快速原型化新的编程语言和语言特性。 我们使用命令式语言IMP（C的一个子集）来研究这个问题，该语言通过小步操作语义（SOS）和基于重写的操作语义（K语义）进行形式化。 我们引入了三个评估集——人工编写的、LLM翻译的和模糊测试生成的——其难度由跨越大小、控制流和数据流轴的代码复杂度指标控制。 给定一个程序及其用SOS/K语义形式化的语义，模型在三个从粗到细的任务上进行评估：（1）最终状态预测，（2）语义规则预测，以及（3）执行轨迹预测。 为了区分预训练记忆与语义能力，我们通过标准规则的系统变异定义了两种非标准语义。 在强大的代码/推理LLM中，尽管在标准语义下表现优异，但在非标准语义下性能下降。 我们进一步发现，（i）不同模型的失败存在模式，（ii）大多数推理模型在涉及高度复杂程序的粗粒度任务中表现非常出色，这些程序通常包含超过五层的嵌套循环，而且（iii）提供形式语义有助于简单程序，但往往对更复杂的程序有害。 总体而言，结果表明LLM可能作为编程语言解释器发挥作用，但也指出了它们在稳健语义理解方面的不足。 我们在https://github.com/EngineeringSoftware/PLSemanticsBench发布基准测试和支持代码。