OPID trains an agent with outcome-only RL, then mines its own finished rollouts for natural-language “skills” and uses them as a self-teacher to add dense token-level supervision, lifting WebShop success by +26.5 points on a 1.7B model versus plain Group Relative Policy Optimization (GRPO).
You’ve shipped an LLM agent that takes 15+ steps to complete a task: clicks through a shopping site, runs searches, or files a tool-mediated ticket. You train it with RL, but the only signal you get is “did the whole episode succeed.” If step 4 was the actual mistake, the gradient doesn’t know. The dominant fix is Group Relative Policy Optimization (GRPO), which normalizes trajectory rewards within a group of rollouts. It’s stable but coarse. A whole 30-step episode collapses to one scalar, and every token in that episode shares it. OPID’s pitch: the trajectory you just ran itself contains the credit-assignment information you need. Extract it.
After each rollout, an analyzer LLM (the paper uses GLM-5.2) reads the completed trajectory and writes two kinds of natural-language notes. An episode-level skill summarizes the overall workflow if the task succeeded (“locate object, clean at sink, place in cabinet”) or a failure-avoidance rule if it failed. Step-level skills target a handful of critical timesteps (capped at 5 for embodied, 2 for search), capturing local decisions like “check that the soapbar is clean before placing it in the cart.”
At training time, OPID picks one skill per step (step-level if that step was flagged critical, episode-level otherwise) and injects it into the history. Now the trick: the old policy re-scores the same already-sampled response twice. Once with the original history, once with the skill-augmented history. The per-token log-probability difference becomes a dense token-level advantage, added to the usual Group Relative Policy Optimization (GRPO) group-relative outcome advantage. No regeneration, no separate teacher model.
for traj in rollouts(policy):
ep_skill, step_skills = analyzer(traj) # hindsight notes
for t, (history, response) in enumerate(traj):
skill = step_skills[t] if t in critical else ep_skill
for token in response:
logp_base = old_policy(token | history)
logp_skill = old_policy(token | history + skill)
A_skill[token] = logp_skill - logp_base # dense signal
A_total = A_outcome + lambda_skill * A_skill
update_policy(clipped_ppo_loss(A_total))
At inference, none of this runs. The analyzer, the skills, the second scoring pass: all gone. The policy acts from the plain history. The paper proves the unclipped skill loss is a sampled-token reverse-KL surrogate, which is why it composes cleanly with PPO clipping.
The prevailing approach to skill-conditioned agents is to maintain an external skill library: extract skills offline, store them, retrieve a relevant one at inference, condition on it. This paper shows the opposite. The most useful skills come from the policy’s own just-finished trajectories, and they belong in the gradient, not the prompt. Retrieved skills are stale because they were written under a different state distribution than the one your current policy induces; on-policy hindsight skills are distribution-matched by construction. The cleanest evidence is the Skill-GRPO collapse when its skill prompt is removed at test time, not the headline averages.
The load-bearing finding is the train-test mismatch experiment. Skill-GRPO trains with retrieved skills in context; when those skills are removed at validation, accuracy on ALFWorld drops by -14.8 to -25.0 points versus plain Group Relative Policy Optimization (GRPO) across the three Qwen sizes. OPID is evaluated without any skill input and beats Skill-GRPO by +24 to +38 points on the same setup. That gap is what tells you the skill knowledge actually got internalized into the weights rather than being a prompt-following crutch.
•
Versus outcome-only Group Relative Policy Optimization (GRPO): +9.3 / +8.6 / +10.9 points on ALFWorld / Search-QA / WebShop for Qwen2.5-3B. The smaller Qwen3-1.7B sees +26.5 on WebShop success rate (64.8 vs 58.6).
•
Sample efficiency: OPID at 60% of training data roughly matches GRPO at 100%.
•
Cross-domain: on the ALFWorld unseen split, OPID beats GRPO by +7.7 average.
•
Ablations: dropping episode-level skills costs ~10 points, dropping step-level costs ~5, and replacing critical-first routing with “apply both layers everywhere” costs +6.8 points, showing the gating matters more than just having both skill types.
Search-QA gains are smaller (≤+0.4 average over the best baseline) and on Qwen3-1.7B Search-QA, OPID is roughly tied with GRPO. The method helps most where horizons are long and outcome rewards are sparsest.
Reach for this when you’re training a multi-turn agent with verifier rewards (web tasks, embodied control, search-and-answer) and you suspect a lot of your rollouts are failing for diagnosable reasons that a stronger LLM could articulate post-hoc. Today you probably run Group Relative Policy Optimization (GRPO) and hope. OPID says: after each rollout group, spend extra inference on an analyzer model to write down what went right or wrong, then bake that knowledge into the next gradient step. Your inference stack stays unchanged.
Code is at GitHub. Training was done on 8x A800 80G GPUs for 150 steps on Qwen2.5-3B/7B and Qwen3-1.7B. The analyzer is GLM-5.2 at temperature 0.4. No new dataset is released; the benchmarks (ALFWorld, WebShop, the Search-R1 QA setting QA suite) are existing public ones.
Your agent’s own finished trajectories are a free teacher; use them to shape gradients, not prompts. Retrieved skills at inference create a train-test mismatch that silently bleeds performance the moment your deployment context drifts. Hindsight skills extracted from on-policy rollouts are distribution-matched by construction, and distilling them into the weights gives you the lift without the inference-time dependency.
•
The analyzer is a separate strong LLM (GLM-5.2). Training cost includes those calls on every rollout group. The paper doesn’t report this overhead or test cheaper analyzers, so the actual training-compute ratio versus plain Group Relative Policy Optimization (GRPO) is unclear.
•
Gains concentrate on long-horizon embodied and web tasks. On Search-QA, where horizons are short (max 4 steps) and rewards are less sparse, OPID is roughly tied with Group Relative Policy Optimization (GRPO) on the 1.7B model. If your task is short-horizon, the dense supervision may not buy much.
•
The skill coefficient is tiny (λ=0.001) and the paper doesn’t sweep it; the method’s sensitivity to this knob, and to analyzer quality, is not characterized.