Improving Long-Horizon RL with Process-Level Rewards

We learn when we make mistakes. We commit an error, observe our failure, reflect, and improve. However, we learn more from our mistakes when we receive detailed feedback on our failures — which provides valuable guidance to steer our actions in a stronger direction. In difficult tasks, simply knowing that we were right or wrong is not nearly informative enough to draw meaningful conclusions and improve. In these cases, we might struggle to understand exactly where we went wrong, or just how badly we actually failed.

Language models learn through a similar trial-and-error paradigm. In reinforcement learning, models generate numerous attempts, or rollouts, for a given task and receive a reward, which scores the attempt; the training algorithm uses these rewards to update the model’s weights to improve future outputs. Like humans, models often require more detailed feedback to learn most effectively. As such, if the reward received by a model for a task is noisy or vague, the model’s ability to learn may be adversely affected.

In this blog post, we demonstrate that reward granularity matters, and that increasing reward granularity can make RL training more effective, especially for the longer-horizon, multi-turn tasks that many AI agents are used for today.

Outcome-level rewards make credit assignment difficult

Reinforcement learning setups commonly use a binary reward: the model completes an entire rollout and receives a single pass/fail score at the end. This reward is considered outcome-level, because it only evaluates the final result rather than the individual steps that produced it.

Outcome-level rewards introduce the issue of credit assignment. When a trajectory passes or fails, the model receives feedback on whether the rollout succeeded, but not which steps were responsible. A successful trajectory may contain unnecessary or incorrect reasoning, while a failed trajectory may be almost entirely correct except for one late mistake. In either case, the same outcome-level binary reward is applied to every action in the trajectory.

In GRPO, weak credit assignment from terminal binary rewards creates two particularly clear problems:

1. Sample inefficiency: Binary rewards do not capture partial progress. GRPO calculates advantage from reward differences within a group of sampled rollouts. If every rollout in a group passes or every rollout fails, their normalized advantages are zero, and the group contributes no policy-gradient update. The rollouts may still differ in meaningful ways, but the reward does not expose those differences, so potentially useful training data is wasted.

2. Noisy and conflicting updates: A mostly correct rollout with one late error receives the same failing reward as a rollout that was wrong from the start. Near-identical trajectories can also receive opposite advantages because one succeeds at the final step and the other fails. The gradient may therefore reinforce and penalize many of the same actions even when the meaningful difference between the trajectories occurs much later.

The problem of credit assignment becomes more severe as the task horizon grows. A longer trajectory contains more actions that could have influenced the outcome, while the amount of feedback remains fixed at a single binary reward. Each action is therefore updated using a signal that depends on many other actions in the rollout. As the number of steps grows, it becomes harder to isolate the contribution of any one step, and the resulting updates to the model can become noisier.

Increasing reward granularity with process-level rewards

One way to improve credit assignment is to use a process-level reward that evaluates the intermediate steps of a trajectory, rather than only whether the final task was completed. A simple version might reward the fraction of steps completed correctly:

This rewards partially correct trajectories with partial credit, giving the model a more accurate signal on its performance as compared to a pass/fail. The reward must still be designed so that maximizing it remains aligned with completing the overall task. Depending on the task, this may require accounting for the order of steps, dependencies between them, or their relative importance.

For short or simple tasks, a terminal binary reward may already provide enough signal for effective training. As the task horizon grows, however, process-level feedback becomes more valuable because it provides a clearer signal about the quality of the intermediate actions that led to the final outcome. Although process-level rewards can be difficult or time-consuming to construct at the scale required for training, they offer a direct way to mitigate the credit-assignment problem in long-horizon RL.

Example: Increasing reward granularity in τ²-bench retail

To demonstrate the effect of reward granularity, we perform RL training on the retail domain of τ²-bench, a benchmark of multi-turn customer service tasks. The domain consists of longer-horizon tasks where an agent converses with an LLM-simulated user, invokes tools against a product and order database, and works to resolve the customer’s issue.

Experimental setup:

We train Qwen3-4B with GRPO on 74 tasks from the retail domain and evaluate on a held-out set of 40 tasks across 16 trials. We train with two different reward functions:

1. Outcome-level reward: The native reward given by the environment. At the end of each trajectory, a deterministic evaluator compares the final database state with the expected state, rewarding binary pass/fail.

2. Hybrid reward: Weighted sum of binary pass/fail and process-level partial-credit signals evaluated directly from the domain environment. The partial credit consists of:

   • Golden actions — whether the agent made each of the exact tool calls a correct solution must contain.
   • Communicated information coverage — whether the agent communicated a required policy to the user, judged deterministically.

The addition of these rewards increases the reward granularity, making a complete failure distinguishable from a strong attempt that breaks down at later steps.

We compare evaluation performance between an untrained baseline and two models trained on each reward (holding all other hyperparameters constant).

Results:

In training, we notice considerable differences in sample efficiency. In training with the binary reward, 43% of rollout groups are discarded due to identical rewards. In these cases, all trajectories within a group pass or fail, and there is no group-relative advantage. By introducing the hybrid reward, the frequency of these groups drops to 7% as granular rewards create comparability.

We observe that task success rate climbs comparably between the outcome-level and hybrid rewards, with the hybrid reward achieving slightly higher growth given both models begin from the same untrained baseline.

In our evaluations, both hybrid and outcome-level rewards improve pass@k at lower values of k, but converge to the same pass@16 across the full set of trials. We interpret this as evidence that RL training shifts probability mass toward behaviors the model can already produce, rather than teaching entirely new behaviors — a result consistent with the findings of DeepSeek’s original GRPO paper.

Across both training and evaluation, the difference between hybrid and outcome-level rewards is noticeable but marginal. We suspect that this gap may become more pronounced when training across a larger and more diverse set of tasks — which we may explore in further iterations of this experiment.

Conclusion

Credit assignment remains a central challenge in long-horizon RL. As trajectories grow longer, a single outcome-level reward provides less information about how well the intermediate steps were completed, making it harder to identify the behavior responsible for success or failure.

Beyond process-level rewards, process-level supervision offers a promising direction. Rather than assigning a single score to the trajectory as a whole, it applies distinct learning signals to different parts of the trajectory, directly upweighting actions that contribute to successful outcomes and downweighting those that lead the model astray.

Nonetheless, process-level rewards move toward the broader goal of constructing a more informative learning signal. By capturing partial progress and distinguishing between trajectories that receive the same final outcome, they provide a practical improvement in training signal over the terminal binary rewards common in RL today and a foundation for more precise credit assignment.