PORTool: 基于奖励树和重要性感知的策略优化方法，用于多工具集成推理

PORTool: Importance-Aware Policy Optimization with Rewarded Tree for Multi-Tool-Integrated Reasoning

AI 导读

研究团队提出PORTool算法，以解决多工具集成推理中仅依靠结果奖励导致的信用分配模糊问题。该方法通过重要性感知策略优化，在结果级监督下强化智能体的工具使用能力，同时实现步骤级奖励分配。PORTool生成奖励树来明确关键决策步骤，从而更精确地引导模型学习有效的工具调用序列，提升复杂任务解决的效率和可靠性。

推荐理由

不少 Agent 团队训练时都遇到过奖励信号太稀疏的问题，PORTool 试着把奖励细粒度化，给了个可实操的解法，做工具调用智能体的值得深读。

正文

Multi-tool-integrated reasoning enables LLM-empowered tool-use agents to solve complex tasks by interleaving natural-language reasoning with calls to external tools. However, training such agents using outcome-only rewards suffers from credit-assignment ambiguity, obscuring which intermediate steps (or tool-use decisions) lead to success or failure. In this paper, we propose PORTool, an importance-aware policy-optimization algorithm that reinforces agents’ tool-use competence from outcome-level supervision while assigning reward at the step level. Specifically, PORTool generates a rewarded rollout tree in which trajectories share prefixes before branching, enabling direct comparisons among alternative tool-use decisions within the same context. It then estimates each step’s importance by a correctness-dominant signal, i.e., whether descendants of that step can ultimately produce a correct final answer, plus an auxiliary term indicating whether the step’s tool calls execute successfully. Using these step-wise importance estimates, PORTool updates the policy to generate efficient tool-call steps, guided by both local comparisons within each branching decision and the overall quality of entire trajectories. Experiments show that PORTool improves final-answer accuracy while reducing tool-call steps compared with state-of-the-art baselines, and ablation studies confirm the robustness of the proposed step-wise importance estimates.

† Purdue University
** Work done while at Apple