标题： 分步策略用于稀有工具知识（SPaRK）：在大型语言模型中驱动多样化工具使用的离线强化学习 显示英文标题

标题： Step-wise Policy for Rare-tool Knowledge (SPaRK): Offline RL that Drives Diverse Tool Use in LLMs

摘要： 我们提出了一种分步策略，用于稀有工具知识（SPaRK），这是一种新颖的强化学习框架，它教会大型语言模型在传统高温采样之外探索多种工具使用模式。 基于分步强化学习的最新进展，我们引入了一个双目标奖励系统，同时优化答案质量和工具多样性，在MMLU-Pro数据集上合成生成的轨迹上通过离线PPO训练Llama-3.1 8B模型。 我们的方法独特地采用了一种以稀有性优先的利用策略，其中GPT-4o评判者对八个不同工具加上思维链推理的候选动作进行评分，策略倾向于较少使用但仍然可行的工具，以鼓励系统性探索。 实证结果表明，SPaRK在14个MMLU-Pro类别中表现出具有竞争力的性能，同时在工具选择上的熵显著高于基线和监督微调方法，这表明通过显式工具多样性进行算法探索可以在不牺牲准确性的情况下增强推理能力。 显示英文摘要

摘要： We present Step-wise Policy for Rare-tool Knowledge (SPaRK), a novel reinforcement learning framework that teaches large language models to explore diverse tool usage patterns beyond conventional high-temperature sampling. Building on recent advances in step-wise reinforcement learning, we introduce a dual-objective reward system that simultaneously optimizes for answer quality and tool diversity, training a Llama-3.1 8B model through offline PPO on synthetically generated trajectories from the MMLU-Pro dataset. Our approach uniquely employs a rarity-first exploitation strategy where a GPT-4o judge scores candidate actions across eight distinct tools plus chain-of-thought reasoning, with the policy favoring less-frequently used but still viable tools to encourage systematic exploration. Empirical results demonstrate that SPaRK achieves competitive performance across 14 MMLU-Pro categories while exhibiting significantly higher entropy in tool selection compared to both baseline and supervised fine-tuning approaches, suggesting that algorithmic exploration through explicit tool diversity can enhance reasoning capabilities without sacrificing accuracy.