标题： 推理流程：使用最少示例训练大语言模型进行发散问题解决 显示英文标题

标题： Flow of Reasoning:Training LLMs for Divergent Problem Solving with Minimal Examples

摘要： 生成给定问题的多种解决方案的能力是人类创造力的显著特征。 这种发散性推理对机器也很重要，可以增强其鲁棒性，并使其在许多应用中协助人类，例如科学发现。 然而，现有的基于大型语言模型（LLMs）的多步骤推理方法大多只关注推理准确性，而没有进一步发现更多多样化的有效解决方案。 例如，监督微调可以提高LLM的推理质量，但需要大量监督数据来捕捉所有可能的解决方案范围。 强化学习旨在寻找有限的最高奖励解决方案，而忽略了解决方案的多样性。 为了填补这一空白，我们提出了 推理流（FoR），一种高效的多样性探索LLM微调方法，旨在用最少的数据提高推理质量和多样性。 FoR将多步骤LLM推理形式化为DAG结构的推理图上的马尔可夫流。 这种形式化使我们能够纳入并适应有原则的 GFlowNet方法，用于微调LLM以按目标问题的（未归一化）奖励概率采样多样化的推理路径。 大量实验表明，使用有限的训练示例（例如，15个示例），FoR能够发现多样化的、富有创意的高质量解决方案，在五个具有挑战性的谜题解决任务中，包括 BlocksWorld（具身推理）、Game24（数学谜题解决）、魔方 （空间推理）、1D-ARC（抽象推理）和PrOntoQA（逻辑推理），显著优于一系列现有的推理和训练方法。 代码可在 https://github.com/Yu-Fangxu/FoR 获取。 显示英文摘要

摘要： The ability to generate diverse solutions to a given problem is a hallmark of human creativity. This divergent reasoning is also crucial for machines, enhancing their robustness and enabling them to assist humans in many applications such as scientific discovery. However, existing approaches to multi-step reasoning with large language models (LLMs) have mostly focused only on reasoning accuracy, without further discovering more diverse valid solutions. For example, supervised fine-tuning can improve LLM reasoning quality, but requires extensive supervised data to capture the full range of possible solutions. Reinforcement learning aims to find limited highest-reward solutions while neglecting the solution diversity. To fill this gap, we propose Flow of Reasoning (FoR), an efficient diversity-seeking LLM finetuning method aimed at improving reasoning quality and diversity with minimal data. FoR formulates multi-step LLM reasoning as a Markovian flow on a DAG-structured reasoning graph. This formulation allows us to incorporate and adapt principled GFlowNet approaches, for finetuning LLMs to sample diverse reasoning paths with probabilities proportional to the (unnormalized) reward of target problems. Extensive experiments show that, with limited training examples (e.g., 15 examples), FoR enables the discovery of diverse, creative, high-quality solutions, greatly outperforming a wide range of existing inference and training methods across five challenging puzzle-solving tasks, including BlocksWorld (embodied reasoning), Game24 (math puzzle solving), Rubik's Cube (spatial reasoning), 1D-ARC (abstraction reasoning), and PrOntoQA (logical reasoning). Code is available at https://github.com/Yu-Fangxu/FoR.