OraRL adds each ground-truth annotation as an extra “oracle” rollout inside the on-policy Group Relative Policy Optimization (GRPO) group, but keeps it out of the baseline used to score the other rollouts. This preserves useful positive advantages that naive mixing destroys, cutting a ~22% rate of sign-flipped advantages down to ~2%.
You’re post-training a vision-language model to output structured things: temporal spans, bounding boxes, segmentation prompts, tracked trajectories. You already have annotations. The dominant recipe is Group Relative Policy Optimization (GRPO): sample 8 answers per prompt, score each with a task reward, push the above-average ones up and the below-average ones down. Problem: for tight structured outputs (an exact time interval, a pixel-accurate box), the model almost never samples anything close to the annotation, so the whole group of 8 is mediocre and there’s no strong positive signal to learn from. Chain-of-thought doesn’t help; it just makes every rollout longer and more expensive. The obvious fix, “just add the annotation itself as a 9th rollout,” quietly breaks the math, and this paper explains why and how to do it right.
The intuition: the annotation is a guaranteed-perfect answer. Drop it into the group of sampled answers as a bonus rollout, and now every prompt has at least one high-quality positive to learn from. The authors call this annotation-as-rollout.
The subtle failure they identify is advantage inversion. Group Relative Policy Optimization (GRPO) scores a rollout by subtracting the group’s mean reward. If you toss in a perfect oracle, the mean jumps up. Sampled rollouts that were genuinely better than the policy’s average now look below-average and get pushed down. The model learns to imitate the oracle and abandons its own decent attempts.
OraRL fixes this by computing the baseline from the sampled rollouts only. The oracle contributes through two separate terms instead:
•
a directional gain that amplifies above-average sampled rollouts more when the oracle-policy reward gap is large (signaling there’s headroom),
•
a detached oracle advantage that decays as the policy closes the gap, and is capped so the oracle never dominates the update.
Then for efficiency they prune: keep the oracle plus the strongest positive and strongest negative sampled rollouts, and re-center the surviving advantages to zero mean.
rollouts = sample_policy(prompt, n=8)
rewards = [reward(r) for r in rollouts]
oracle = serialize(annotation) # 9th rollout, guaranteed positive
mu = mean(rewards) # baseline excludes oracle
adv = [r - mu for r in rewards]
gain = clip((std_with_oracle / std_without) ** 0.25, 1, 4)
adv = [gain * a if a > 0 else a for a in adv]
adv_oracle = min(2 * gap_weight, cap_from_best_positive)
keep = {oracle} | top_positive(adv) | top_negative(adv, k=2)
update_policy(keep, recenter(adv))
The standard playbook when GRPO groups look weak is to sample more, add Chain-of-Thought, or bring in a teacher model with off-policy correction (e.g., LUFFY). This paper takes the opposite tack. The annotation you already have is a free, perfect rollout, but only if you refuse to let it contaminate the baseline the other rollouts are scored against. The load-bearing evidence isn’t the leaderboard sweep; it’s the direct measurement that naive oracle mixing inverts the sign on ~22% of rollouts that GRPO would have rewarded.
The measurement that proves the mechanism: on 92,024 rollouts, naive oracle injection assigns negative advantages to 22.4% of rollouts that vanilla GRPO would score as positive. Whole groups get poisoned: 42.5% contain at least one inverted rollout, 8.3% lose every positive signal entirely. The prior fix from the authors’ own Tempsamp-R1 (task-specific reward shaping) cuts the flip rate to 11.9% but requires per-task hand tuning. OraRL brings it to 1.9% before pruning and 0.3% after, using one update rule for every task.
That mechanism translates into consistent downstream gains:
•
On the same 4B backbone, matched training budget: OraRL beats GRPO, Dr. GRPO, CPPO, and continued SFT by 2.0+ points on a three-task average; the other GRPO variants sit within 0.4 points of each other.
•
Naive oracle injection actually underperforms GRPO (55.4 vs 60.3 average, tracking drops 11.9 points), confirming that just adding the annotation without decoupling the baseline is worse than not adding it.
•
Scales monotonically from 0.8B to 9B and continues improving up to 100k RL prompts, where GRPO plateaus earlier.
•
Trained Video-ORA-9B wins its target benchmarks without CoT, hitting 73.1 on VSI-Bench versus 55.0 for GPT-5 and 55.1 for Gemini-3-Pro. Decoding is ~37× faster than the CoT-enabled backbone (130 ms vs 4,780 ms of generation after first token).
•
Sign-balanced pruning keeps 4 of 9 rollouts, giving a 1.48× step-time speedup for a 0.4-point average drop.
Reach for this when you’re RL-fine-tuning a model on tasks with exact structured targets: spans, boxes, JSON, SQL, function-call arguments, anywhere the model rarely samples the right answer on its own but you have labeled examples. Today you either grind through low-signal GRPO groups or fall back to SFT and lose the discriminative signal. OraRL says: serialize each label into the model’s response format, append it as a 9th rollout, but compute the advantage baseline from the sampled 8 only, and add two extra terms that fade as the policy catches up.
Everything is released: code, model weights from 0.8B to 9B, and the training data covering seven video tasks. The method itself is a drop-in modification to any GRPO loop in verl or similar frameworks; the video-specific pieces are the task adapters that serialize annotations and compute scalar rewards.
When you have ground truth, feed it to the policy as a rollout, but never let it into the baseline that scores the other rollouts. The oracle belongs in the optimization target, not in the yardstick. Mixing the two collapses learning toward imitation and throws away the exploration signal that made group-relative RL worth using in the first place.
•
Requires a scalar task reward and an annotation that serializes cleanly into the model’s output format. Ambiguous, partial, or noisy labels are untested, and the paper explicitly flags this.
•
The gains are strongest on structured perception tasks where sampled rollouts rarely hit the exact target. On tasks where GRPO already produces strong positives (e.g., some multiple-choice QA), the oracle-policy gap is small and OraRL’s advantage shrinks.
•
Spatial-reasoning tasks needing genuine planning (route planning, rotation, backward-facing relative direction) still trail proprietary models. Better rollouts don’t compensate for a backbone that hasn’t learned the underlying capability.