AgentOPSD turns a per-turn teacher-student log-prob gap into sequential credit by folding it into a running Bayesian success-belief and rewarding turns that most revise that belief, losing only 0.54 success points per extra task turn versus 2.9 for Group Relative Policy Optimization (GRPO) on long-horizon agent tasks.
You’ve shipped a multi-turn agent, say a shopping bot or a search-and-answer system, and you train it with RL from a single end-of-episode success signal. Today, if the episode succeeds, every one of its 20 turns gets the same positive gradient, including the three redundant clicks and the one unlucky detour that actually broke reasoning. If it fails, every turn is punished equally, even the correct plan that preceded a bad final action. This is the standard behavior of Group Relative Policy Optimization (GRPO) and its agent variants: one trajectory-level advantage, broadcast uniformly. The longer the horizon, the more this smearing hurts, because a single scalar has to explain more decisions.
The intuition: a turn is important not because it looks good locally, but because it changed your estimate of whether the whole trajectory will succeed. AgentOPSD makes that estimate explicit and updates it turn by turn.
The local signal comes from On-Policy Self-Distillation: the same policy is scored twice on the action it actually took, once with a retrieved “skill hint” prepended (a training-only privileged context describing useful subgoals) and once without. Sum the per-token log-prob differences over the turn, and you get one number e_k. Positive means the skill-conditioned branch, treated as a proxy for successful behavior, likes this action more than the plain policy does. The paper shows this contrast approximates the Bayes factor between success-conditional and failure-conditional action likelihoods.
Now the recursion. Start the trajectory’s success belief B_0 at the group’s empirical success rate (the same \bar R Group Relative Policy Optimization (GRPO) already computes). Accumulate evidence in log-odds space space with a geometric decay, so recent turns weigh more. The credit for turn k is the marginal belief revision \Delta B_k = B_k - B_{k-1}, signed by the trajectory’s final outcome. Standardize these revisions within the trajectory, clip into a bounded band around 1, and use them as a multiplier on GRPO’s existing advantage. So sign is always set by the verifier; belief revision only reshapes magnitude across turns.
B_prev = clip(group_success_rate, eps, 1-eps)
l_prev, c_prev = logit(B_prev), 0.0
for k, turn in enumerate(trajectory):
e_k = sum(logp_with_skill[t] - logp_plain[t] for t in turn) # detached
c_k = gamma * c_prev + e_k
l_k = logit(B_0) + c_k
B_k = sigmoid(l_k)
q_k = sign(A_seq) * (B_k - B_prev) # outcome-aligned credit
B_prev, c_prev = B_k, c_k
w = clip(1 + b * zscore(q), 1-b, 1+b)
A_reshaped = A_seq * ((1 - lam) + lam * w) # feed into GRPO clipped loss
One extra teacher forward pass per turn, no critic network, no extra rollouts.
The prevailing move in self-distillation RL is to take the teacher-student gap and inject it locally: as a per-token loss, a magnitude gate, or a step-wise reweighting. This paper’s claim is the opposite. A local teacher-student gap is not sequential credit. What matters is how much that gap moves a running estimate of eventual success given everything that happened before. The cleanest evidence is the ablation that swaps the recursive belief revision \Delta B_k for the raw local gap e_k while keeping everything else fixed, not the headline benchmark number.
The load-bearing ablation on ALFWorld with Qwen2.5-7B: replacing the recursive belief revision with the raw local gap drops success from 89.1% to 82.8%, and dropping the outcome-aligned sign (using only magnitude) drops it to 80.5%. Removing the empirical prior B_0 and starting belief at an arbitrary uncertainty level drops it further to 78.9%. These three isolate the paper’s actual mechanism: recursion, outcome sign, and a verifier-grounded starting point each contribute independently.
Secondary evidence for the mechanism:
•
Horizon robustness on ALFWorld: AgentOPSD loses 0.54 success points per additional turn in a linear fit, versus -2.91 for GRPO and -3.59 for RLSD (Self-Distilled RLVR). Uniform-credit methods degrade fastest exactly where turn-level credit should matter.
•
Headline numbers: AgentOPSD beats GRPO and self-distillation baselines including GRPO+OPSD, Skill-SD, RLSD, SDAR, and StepOPSD on all eight aggregate comparisons across two Qwen2.5 scales on ALFWorld, WebShop, and Search-QA (except SDAR, which it beats on 6 of 8). All baselines see the same retrieved skills, so the gain is attributed to credit construction, not privileged information.
•
Sensitivity: only the reshaping weight \lambda matters much (0.5 is best; smaller values collapse toward vanilla GRPO). Decay \gamma and clip radius barely move results. On short-horizon Search-QA (max 4 turns), the whole method is nearly inert, which is consistent with the framing.
Reach for this when you’re training an agent with sparse end-of-episode rewards and long trajectories, where GRPO’s uniform advantage is drowning out the few decisions that matter. Concretely, imagine a search-and-synthesize agent whose reward is only “did the final answer match.” You already have some form of “good example” context (retrieved skills, a stronger prompt, a curated demo). AgentOPSD says: keep training on your own rollouts as usual, but at every turn score the action twice, once with that privileged context and once without, then let the running success belief tell you which turns to amplify. No critic to train, no extra rollouts, one added teacher forward per turn.
Code is at GitHub. The method reuses standard Group Relative Policy Optimization (GRPO) rollouts and losses; the additions are the per-turn belief recursion and a bounded multiplier on the advantage. The paper reports one shared hyperparameter setting across three environments and two model scales, so the tuning surface is small. Skills at inference are not required; they are training-only privileged context.
Local distillation gaps become credit only when you make them argue with the history that came before. A per-turn log-prob difference is a signal, not an assignment. Running it through a belief state and reading off the marginal revision is what converts local evidence into sequential credit, and it’s the cheapest way to restore per-turn structure without paying for a critic or extra rollouts.
•
The whole scheme depends on having a “privileged” context that genuinely correlates with success. Here it’s retrieved skills from a prior work’s skill bank; if your setting has no comparable hint source, the teacher branch has nothing informative to say and the gaps go noisy.
•
The advantage over uniform-credit baselines shrinks sharply on short-horizon tasks (four-turn Search-QA barely moves). If your agent’s episodes are two or three turns, the ceremony is unlikely to pay for itself.
•
Evaluation is confined to Qwen2.5-3B/7B on three text-only environments. Transfer to larger models, tool-use with real APIs, or code agents with structured verifiers is untested here.