标题： 通过蒙特卡洛树搜索进行过程监督以增强推理 显示英文标题

标题： Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search

摘要： 大型语言模型（LLMs）已经在各种任务中展示了其显著的能力。 然而，推理仍然是LLMs的挑战。 为了提高LLMs的推理能力，过程监督已被证明优于结果监督。 在本工作中，我们研究使用蒙特卡洛树搜索（MCTS）让LLMs自身生成过程监督数据以训练它们。 我们用LLM采样推理步骤，并为每个步骤分配一个分数，该分数捕捉其“相对正确性”，然后通过最小化生成推理步骤的加权对数似然来训练LLM。 这个先生成后训练的过程反复进行直到收敛。我们的实验结果表明，所提出的方法显著提高了LLMs在两个数学推理数据集上的性能。 此外，在一个数据集上训练的模型在另一个数据集上也表现出改进的性能，显示出增强推理能力的可迁移性。 显示英文摘要

摘要： Large language models (LLMs) have demonstrated their remarkable capacity across a variety of tasks. However, reasoning remains a challenge for LLMs. To improve LLMs' reasoning ability, process supervision has proven to be better than outcome supervision. In this work, we study using Monte Carlo Tree Search (MCTS) to generate process supervision data with LLMs themselves for training them. We sample reasoning steps with an LLM and assign each step a score that captures its "relative correctness," and the LLM is then trained by minimizing weighted log-likelihood of generating the reasoning steps. This generate-then-train process is repeated iteratively until convergence.Our experimental results demonstrate that the proposed methods considerably improve the performance of LLMs on two mathematical reasoning datasets. Furthermore, models trained on one dataset also exhibit improved performance on the other, showing the transferability of the enhanced reasoning ability.